Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Facebook Open-Sources Its Presto SQL Query Engine

Nerval's Lobster (2598977) writes | about 9 months ago

0

Nerval's Lobster (2598977) writes "Facebook stores its warehouse data in a set of enormous Hadoop/HDFS-based clusters. That helps the social network wrestle with the enormous amounts of user information it needs to store and analyze every day; but at a certain point (namely, once the warehouse grew to petabyte scale), its network administrators decided they needed something other than Hadoop MapReduce and Hive to process that data in a fully optimized way. Enter Presto, Facebook’s very own distributed SQL query engine designed with a focus on speed. The platform supports standard ANSI SQL, which means it’s capable of everything from complex queries and aggregations to joins and window functions. Presto also boasts scalability and flexibility; for example, with the addition of key plugins, it can handle Facebook data not stored in HDFS clusters, such as HBase and custom systems. It doesn’t rely on MapReduce for processing, which allows it to process queries at speed. Facebook engineers began developing Presto near the end of 2012 and rolled it out to the entire company the following spring; employees currently use it to process roughly 30,000 queries (totaling around one petabyte of data) per day. And now that the platform’s stable enough, Facebook is open-sourcing it via Github and a dedicated Website. Now all you need is an enormous amount of data that threatens to overwhelm your current setup."
Link to Original Source

cancel ×

0 comments

Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...