Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Twitter Is Moving To the Cassandra Database

kdawson posted more than 4 years ago | from the big-table-doesn't-capture-the-half-of-it dept.

Databases 157

MyNoSQL has up an interview with Ryan King on how Twitter is transitioning to the Cassandra database. Here's some detailed background on Cassandra, which aims to "bring together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model." Before settling on Cassandra, the Twitter team looked into: "...HBase, Voldemort, MongoDB, MemcacheDB, Redis, Cassandra, HyperTable, and probably some others I'm forgetting. ... We're currently moving our largest (and most painful to maintain) table — the statuses table, which contains all tweets and retweets. ... Some side notes here about importing. We were originally trying to use the BinaryMemtable interface, but we actually found it to be too fast — it would saturate the backplane of our network. We've switched back to using the Thrift interface for bulk loading (and we still have to throttle it). The whole process takes about a week now. With infinite network bandwidth we could do it in about 7 hours on our current cluster." Relatedly, an anonymous reader notes that the upcoming NoSQL Live conference, which will take place in Boston March 11th, has announced their lineup of speakers and panelists including Ryan King and folks from LinkedIn, StumbleUpon, and Rackspace.

cancel ×

157 comments

Sorry! There are no comments related to the filter you selected.

Don't believe them! (4, Funny)

smellsofbikes (890263) | more than 4 years ago | (#31248230)

They keep saying that the Cassandra database is better, but somehow I don't believe them. I can't imagine they know what they're talking about. Maybe in the long-term they'll be proven right but I really don't think they are. I don't know why, though...

heh heh heh.

Re:Don't believe them! (0)

Yvan256 (722131) | more than 4 years ago | (#31248260)

Do you have an ex-girlfriend called Cassandra, by any chance?

Re:Don't believe them! (0)

Anonymous Coward | more than 4 years ago | (#31248316)

It's a reference to Greek mythology you idiot

Re:Don't believe them! (0)

Anonymous Coward | more than 4 years ago | (#31249234)

Intelligence is not knowledge. - Einstein

Re:Don't believe them! (0)

Anonymous Coward | more than 4 years ago | (#31249424)

GP still looks dumb.

Re:Don't believe them! (0)

Anonymous Coward | more than 4 years ago | (#31250094)

And you still look like a douche.

Re:Don't believe them! (0)

Anonymous Coward | more than 4 years ago | (#31248798)

Yeah and she told everyone about the GP's micropenis.

Re:Don't believe them! (1)

einhverfr (238914) | more than 4 years ago | (#31250366)

Drink a few beers. Read the Iliad. You'll feel better.

(AJAX: When second-best is good enough. Or maybe AJAX is almost as good as ACHILLES.)

Re:Don't believe them! (1)

Push Latency (930039) | more than 4 years ago | (#31248482)

I took an axe to my last Cassandra cluster and feel quite better now.

Re:Don't believe them! (1)

sconeu (64226) | more than 4 years ago | (#31248810)

Damn... you beat me to it. I was going to say, "Cassandra? I don't believe it!"

Re:Don't believe them! (1)

mariushm (1022195) | more than 4 years ago | (#31249142)

For some reason my mind went to Cassandra Crossing (http://en.wikipedia.org/wiki/The_Cassandra_Crossing)

Cassandra, eh? (4, Funny)

maugle (1369813) | more than 4 years ago | (#31248370)

I hear Cassandra can even predict when disastrous system failures are going to occur! Unfortunately, for some reason nobody ever believes the warnings.

Re:Cassandra, eh? (2, Funny)

einhverfr (238914) | more than 4 years ago | (#31249794)

Especially when trojan horses are the cause of such a disaster....

Re:Cassandra, eh? (1)

idontgno (624372) | more than 4 years ago | (#31249938)

And, of course, when the system failure strikes, Cassandra will be blamed, not the underlying issues Cassandra warned of.

Re:Cassandra, eh? (1)

einhverfr (238914) | more than 4 years ago | (#31250008)

Of course!

Because Cassandra is a Trojan....

Re:Cassandra, eh? (0)

Anonymous Coward | more than 4 years ago | (#31250500)

You mean the Fail Whale Watching component?

Re:Cassandra, eh? (1)

Hurricane78 (562437) | more than 4 years ago | (#31251350)

Disastrous failure? Twitter? There’s at least one joke in there somewhere. ^^

hmmm (0)

Anonymous Coward | more than 4 years ago | (#31248442)

facebook uses casandra, digg uses cassandra, twitter is mocing to cassandra. Maybe in 5 years slashdot will get with it.

Re:hmmm (1, Insightful)

Anonymous Coward | more than 4 years ago | (#31248666)

Maybe in 5 years slashdot will get with it.

Do you realize how many years it took Slashdot to just remove their HTML table layout from Slashcode? I wouldn't bet on a major backend change for Slashdot, ever.

Re:hmmm (0)

Anonymous Coward | more than 4 years ago | (#31248780)

facebook uses casandra

Bye bye Twitter, it was nice knowing you.

I'd rather not get tweets from last week showing up as "latest".

Re:hmmm (1)

clarkkent09 (1104833) | more than 4 years ago | (#31249292)

Yeah, but in those cases if something horrible happens and all data gets deleted, nothing of value will be lost, whereas with slashdot........ok never mind

network issues? (4, Insightful)

QuietLagoon (813062) | more than 4 years ago | (#31248592)

We were originally trying to use the BinaryMemtable interface, but we actually found it to be too fast it would saturate the backplane of our network.

.

First time I have ever heard anyone say that a database was too fast. Maybe there are network problems that also need to be addressed.

Re:network issues? (0)

Anonymous Coward | more than 4 years ago | (#31249602)

Yeah, suck it up. why discount it for that reason? Sounds like there's real room for growth down that path.

Re:network issues? (1)

KermodeBear (738243) | more than 4 years ago | (#31249966)

Could someone explain to me why this kind of speed would be a problem? It seems to me that if BinaryMemtable is so incredibly fast that other things become a bottleneck, then you're in a great position. You have something very fast for storing and retrieving data - you just need to get bigger, faster pipes.

Re:network issues? (2, Insightful)

b0bby (201198) | more than 4 years ago | (#31250096)

I know next to nothing about NoSQL, but what they're talking about there seems to be using BinaryMemtable for the one-time move of data. You can see that you wouldn't want to "saturate the backplane of our network" for several days while that completes, so they're using a slower method & throttling it. It will take a week to do the move, but everything else will keep working.

Re:network issues? (1)

SanityInAnarchy (655584) | more than 4 years ago | (#31250760)

I'm surprised they didn't use the faster method and throtlle it.

Re:network issues? (4, Informative)

ryansking (1752556) | more than 4 years ago | (#31251186)

If we're going to have to slow the system down, we'd rather use the standard interface, because that means the bulk loading doubles as a load test and the tools we build for it can be re-used for normal operations.

Re:network issues? (1)

SanityInAnarchy (655584) | more than 4 years ago | (#31251730)

That actually makes a lot of sense. Thanks!

Re:network issues? (1, Funny)

Anonymous Coward | more than 4 years ago | (#31250150)

you just need to get bigger, faster pipes

That's what she said!

Re:network issues? (2, Informative)

Bill, Shooter of Bul (629286) | more than 4 years ago | (#31250796)

Yes and no. They are specifically talking about importing their data into cassandra. Which will be a one time event, not worth upgrading the network bandwidth. They need to throttle it to allow for more time sensitive traffic to use the bandwidth. The bandwidth to the database in normal use will be much, much less then the import bandwidth.

Re:network issues? (1)

KermodeBear (738243) | more than 4 years ago | (#31251146)

Ah, that makes sense. For some reason I thought they were talking about general usage. Thanks for clearing that up. (o:

Huzzzah! (1)

tthomas48 (180798) | more than 4 years ago | (#31248654)

I look forward to a brand new twitter that randomly doesn't display expected data and sometimes doesn't take my status updates!

Re:Huzzzah! (1, Flamebait)

Target Practice (79470) | more than 4 years ago | (#31250236)

I know you're being sarcastic, but I think some of us around here really do look forward to a non-functioning twitter. Maybe, if it's down long enough, everyone will take a step back and realize what a complete tool they've been, telling the world how their last coffee was, where the Best Place to Buy Things is, or some other third thing equally mundane and self serving.

Here's to them royally screwing up!

And this is front page news, why? (2, Interesting)

Lunix Nutcase (1092239) | more than 4 years ago | (#31248842)

Why is it that whenever twitter makes any random change to some part of its infrastructure that we need a front page story about it?

Re:And this is front page news, why? (4, Funny)

BarryJacobsen (526926) | more than 4 years ago | (#31248990)

Why is it that whenever twitter makes any random change to some part of its infrastructure that we need a front page story about it?

Because the change prevented them from posting it to twitter.

Re:And this is front page news, why? (5, Insightful)

Gruuk (18480) | more than 4 years ago | (#31248994)

Scaling. If something turns out to be robust and fast enough for Twitter, it is definitely of interest to anyone working on significantly large and busy websites.

Re:And this is front page news, why? (3, Insightful)

Lunix Nutcase (1092239) | more than 4 years ago | (#31249246)

Yes, because twitter is the epitome of robustness and speed. Oh wait... Just in the 2 months of this year alone they've had something like 4 outages.

Re:And this is front page news, why? (1)

Gruuk (18480) | more than 4 years ago | (#31249748)

Which is exactly why it would be huge and relevant news if there was something that could make Twitter run way better. It's a perfect example, as it is a very well know websites, with very well known problems related to scalability.

Thank you for helping me prove the point, by the way, that was mighty kind of you.

Re:And this is front page news, why? (1)

Lunix Nutcase (1092239) | more than 4 years ago | (#31250210)

So instead of just seeing what an actual robust and fast site uses we should instead follow Twitter which switches at whim to whatever is the technology de jour of the moment while still being unstable and slow?

Re:And this is front page news, why? (0)

Anonymous Coward | more than 4 years ago | (#31250708)

You're the perfect example of what a morass of stupidity Slashdot has become. Thanks for reminding me why I stopped coming here years ago.

Re:And this is front page news, why? (1)

theshowmecanuck (703852) | more than 4 years ago | (#31252076)

Or is this possibly a case where people are attempting to use technology as a silver bullet against bad design. Your point cannot be made unless we know that they have a good design to begin with and the reason for the outages lies specifically with the database technology. Sometimes the overlooked problem is with bad or dogmatic coding (i.e. too concerned with good form or over-using patterns, and not enough with performance or the 'KISS' principle), and not with the server or hardware technology. Chances are their code is good, but we don't know that. So your premise is not valid.

Re:And this is front page news, why? (4, Insightful)

kriston (7886) | more than 4 years ago | (#31250134)

No way. Their architecture is about as "best guess" engineering as Facebook. I don't think that's actually what engineering is. "Maybe this one will work?"

In the meantime, I have not been able to update my avatar image on Twitter, and TwitPic-like feature is still a faint glimmer in Twitter's amateur eyes. Speaking of missed opportunities, why drive so much traffic to Twitter parasites Bit.ly, TwitPic, TinyURL, Twitition, TwitLonger?

What in the world are Twitter's engineers actually DOING should be the real question.

Re:And this is front page news, why? (1)

e2d2 (115622) | more than 4 years ago | (#31250200)

Which is exactly why developers need to pay attention - So we can avoid these mistakes ourselves.

Re:And this is front page news, why? (2, Interesting)

u38cg (607297) | more than 4 years ago | (#31250930)

Does Twitter really have loads which are more difficult to manage than, say, the BBC, CNN, Google, or Wikipedia? I would have thought serving up a fairly straightforward page, a stylesheet, a background image and the tweets or twits or whatever they're called can't be that difficult compared to, say, Facebook.

Re:And this is front page news, why? (1)

roman_mir (125474) | more than 4 years ago | (#31251148)

Twwweeeeter can also probably generate static pages just as well on some large node and then push them to web servers, that just might have worked better for them.

Do they really need dynamic pages at all or could they live with something that's regenerated every 10 minutes? Just saying.

Re:And this is front page news, why? (1)

DragonWriter (970822) | more than 4 years ago | (#31252814)

Does Twitter really have loads which are more difficult to manage than, say, the BBC, CNN, Google, or Wikipedia?

(1) In some measures , probably;
(2) When Google or Wikipedia makes announcements about technology (whether its a "change" or not) they use in their backend, that's usually often a front-page story on Slashdot, too. The BBC and CNN don't, AFAIK, tend to make big public announcements about back-end technology.

I would have thought serving up a fairly straightforward page, a stylesheet, a background image and the tweets or twits or whatever they're called can't be that difficult compared to, say, Facebook.

Processing the tweets is the big scale issue at Twitter, AFAIK, and while Facebook does something similar with its status updates, ISTR that the scale at Twitter is bigger. But its not really a big issue either way, as when Facebook talks about their technology backend, that also gets attention from Slashdot.

Re:And this is front page news, why? (1)

TheTyrannyOfForcedRe (1186313) | more than 4 years ago | (#31249014)

It's interesting because Twitter is one of the Big Guys and it's cool to know what the Big Guys are up to. Also, a lot of us maintain Twitter based websites and/or apps.

Re:And this is front page news, why? (2, Insightful)

Monkeedude1212 (1560403) | more than 4 years ago | (#31249816)

I suppose then why would we care if any site made any random change to any part of its infrastructure?

Twitter is a -very- busy site.

They are changing their infrastructure to accomodate. Here's what they looked at, here is what they chose. If you are looking for something with equal performance, you don't have to shop around.

Re:And this is front page news, why? (1)

DragonWriter (970822) | more than 4 years ago | (#31252622)

Why is it that whenever twitter makes any random change to some part of its infrastructure that we need a front page story about it?

Because in some areas Twitter is at an extreme of scale, so what they are doing to deal with that extreme of scale (even if it isn't necessarily always the ideal choice) is usually interesting since, if you are looking for things that have been done in production to deal with the kind of scaling they experience, there aren't a lot of other data points to find.

pfffft twatter tweeter (2, Insightful)

roman_mir (125474) | more than 4 years ago | (#31249030)

who cares what twuufter is running off.

The more interesting aspect of all of this 'NoSQL' movement is how they believe that if they achieve some speed improvement against some relational databases, how that makes them so much better.

If you don't really need a database to run your 'website', then who cares if you use flat files or an in memory hashmap for all your data needs? Databases are not being replaced by NoSQL in projects that need databases. The projects that may not have ever needed databases may benefit by this NoSQL idea, but if you actually need a database... well, you better be really good at working around all kinds of problems that this will create for you.

I think that relational databases are good at what they do and that many projects may not need them, but if you do need them on the back end, you will end up with them on the back end. Of-course there maybe some caching/hashmaps/files on the front end but at the back stuff will be sorted out within a real datastore.

Is there really a huge issue with rdbms speeds? Well if there is something there, that's what needs to be looked at. If RDBMSs are not fast enough, that's just an opportunity to work more on them to speed them up.

Re:pfffft twatter tweeter (1)

codepunk (167897) | more than 4 years ago | (#31249170)

Is there really a huge issue with rdbms speeds? I don't know perhaps you should pose that question to google for instance.

Re:pfffft twatter tweeter (2, Insightful)

roman_mir (125474) | more than 4 years ago | (#31250120)

your question is answered in my post: google does not need a database for ACID properties.

Can you complain much if in one location google gives you results that are very different for the same search query as for the same query in a different location at the same time? Well, if you do complain, you can ask google for your money back.

Re:pfffft twatter tweeter (4, Insightful)

AndrewNeo (979708) | more than 4 years ago | (#31249250)

I think their point is not everything needs an RDBMS, whereas before it was the 'go to' method of storing data.

Re:pfffft twatter tweeter (4, Insightful)

Abcd1234 (188840) | more than 4 years ago | (#31250098)

Or: use the right tool for the job. The only difference is, now alternative tools actually exist.

Re:pfffft twatter tweeter (1, Insightful)

roman_mir (125474) | more than 4 years ago | (#31250970)

You know, the truth is, most data is still stored in individual files, not in databases. So RDBMSs were always a very niche thing used for projects because they are understood and it's easier to develop for them if you really have massive data requirements.

Files - that's what many projects even today use, not databases. This is basically what they are going back to - files with whatever window dressing on top - a facade of hashes, it's all key/value pairs. It is, my friends, the old old idea of property files.

I mean, really, I wrote a system in August that uses property files for storage as a database. Property file as a database - because it works. But that's a storage method. So in the NoSQL space they also do clustering by replication across nodes, but it does not really matter much if the data is all the same on all nodes.

But you can do the same with an RDBMS, really, you can skip the principles of ACID and replicate across nodes and hope that it's good enough. Maybe the implementation for things like 'Cassandra' allows faster replication than what is normally done in an RDBMS, but just you wait and see how the RDBMSs of tomorrow provide a few flags to do the same thing in some 'partial ACID mode' with quick replication.

This is intended for applications that do not really care about consistency of data - Google does not care. Twewter does not care. Amazon has to jump through more hoops I am sure than Tweeter, because real money is involved.

Re:pfffft twatter tweeter (0)

roman_mir (125474) | more than 4 years ago | (#31251274)

Let me put it this way, it will make it perfectly clear: if twoofter is regenerating every page on every hit and they are running into issues with speed, then their problem is not their data storage, it's their design. Now that it is clear they don't care about data consistency, I have the solution for them.

They just need to regenerate the pages once in a few minutes on some large node and then push the static content to their webservers. Done. And that's why they sometimes pay me the big bucks :) to think of the obvious.

Re:pfffft twatter tweeter (1)

DragonWriter (970822) | more than 4 years ago | (#31252746)

I think their point is not everything needs an RDBMS, whereas before it was the 'go to' method of storing data.

Except, of course, that it never was the "go to" method of storing data. There was no point in history where RDBMS's were anywhere close to the exclusive method of persisting data. Non-relational document-oriented storage has pretty much always dominated in the era in which relational databases existed, whether it was proprietary binary document formats, fairly direct text-based document formats, or highly structures (XML, etc.) text-based document formats.

Re:pfffft twatter tweeter (1, Insightful)

Anonymous Coward | more than 4 years ago | (#31249390)

Is there really a huge issue with rdbms speeds? Well if there is something there, that's what needs to be looked at. If RDBMSs are not fast enough, that's just an opportunity to work more on them to speed them up.

Surely that's the point. It isn't possible to practically scale RDBMSs up to the sort of scale you need for a huge website such as Amazon. The requirement to continue to meet all of the constraints of the relational model makes it very hard to split databases over a large cluster without a lock-bound hell. There are two solutions to this - either you spend a vast amount of effort trying to get the relational model to scale a bit, or you bite the bullet and relax the relational model's constraints.

Don't get me wrong - there are good reasons why the relational model has constraints in the data model to ensure ACID qualities. However beyond a certain point it is easier to deal with the problems that come from using a different model than it is to stretch a conventional RDBMs and deal with the problems of keeping multiple distributed copies of data consistent.

Take the collection of user reviews and product pictures on a large site like Amazon. Does this need the analytical power of a RDBMS? No. Does it need something a lot more advanced then "flat files or an in memory hash-map" in order to scale to heavy loads across multiple continents? Yes. That's the sort of thing NoSQL databases are working on.

In general your attitude reminds me of the people who thought personal computers would always be toys. "Proper work" would be done on mainframes/supercomputers and trivial office tasks may as well be done on paper. Well, mainframes / supercomputers are still faster than personal computers, but few people would claim the PC had no impact on the office.

Re:pfffft twatter tweeter (0, Flamebait)

roman_mir (125474) | more than 4 years ago | (#31250300)

In general your attitude reminds me of the people who thought personal computers would always be toys. "Proper work" would be done on mainframes/supercomputers and trivial office tasks may as well be done on paper. Well, mainframes / supercomputers are still faster than personal computers, but few people would claim the PC had no impact on the office.

- in general your attitude reminds me of everyone who ever thought that the latest fad is the silver bullet that will devoid them of any responsibility for a hacky design and kludgy implementation, all sprinkled with hair bossy management attitude. Good luck with your new silver bullet, hope you kill the vampire of incompetence.

Re:pfffft twatter tweeter (0)

Anonymous Coward | more than 4 years ago | (#31252936)

i think you mean the "werewolf"

Re:pfffft twatter tweeter (4, Interesting)

azmodean+1 (1328653) | more than 4 years ago | (#31249450)

I think you're missing the point here, the problem with RDBMSs isn't that they are "slow" per-se, which implies that they just need some good ol' fashioned optimization. The problem is that there is a cost associated with the data integrity guarantees they make (usually appears in scalability bottlenecks rather than in pure computational inefficiencies), regardless of how good the implementation is, and if you don't need some of those guarantees, you can dispense with them and end up with better performance (again, this typically means better scalability). Additionally, this is the kind of bottleneck that you just can't throw more resources at. Sure you can find the bottleneck and beef up that particular component to do more transactions/second, but at a certain point you've isolated the bottleneck on a world-class server that is doing nothing but that, and it's still a bottleneck. At that point (preferably long before you reach that point) you have to look at transitioning to an infrastructure that makes some kind of tradeoff that allows the removal of the bottleneck, which is what NoSQL does.

I doubt Twitter wants very many RDBMS-type data coherency guarantees at all. 160-character text strings with a similarly-sized amount of metadata, and no real-time delivery guarantees? Sounds like their database can get pretty inconsistent without messing things up badly. It seems to me they would be well served by using a database that offers just what they want/need in that area and better performance.

Oh and this:

Is there really a huge issue with rdbms speeds?

yes, and what are you smoking that you would even ask this question?

Re:pfffft twatter tweeter (1)

roman_mir (125474) | more than 4 years ago | (#31250180)

As I said, there are projects and then there are projects. Tweater is not the project that requires any real database in the first place, who cares is a commit is transactional there?

As for your last comment: problems with database performance are all about design. You think NoSQL will not hit the same roadblocks in projects that don't do design right? What are they going to move to when that one fails? NoNoSQL++?

Re:pfffft twatter tweeter (1)

einhverfr (238914) | more than 4 years ago | (#31250566)

I am going to add a few other things here. The first is that "not possible to scale" is not really accurate. I believe there are ways to design structures so that write capacity on an RDBMS can scale upward with the nodes on the network. Of course this only works for some types of applications (the approach I have in mind would work with Twitter, for example). And even with Amazon, you would CERTAINLY want RI on purchases even if you don't care about reviews.

However, the larger point is that an RDBMS is a tool which is useful for certain types of applications and not others. For example, managing the financial data on product purchases at Amazon is going to be integrity critical but it still has to work and perform well all the time. Reviews, OTOH, won't but integrity problems add to the load of work for the tech support guys.

So there are solutions to this problem which involve middleware, but even in Amazon's case unless you want to cobble together something with bailing twine and duct tape, the RDBMS is going to likely be the go-to solution there. With twitter? Not so much.

Re:pfffft twatter tweeter (0)

Anonymous Coward | more than 4 years ago | (#31251208)

In principle, a database itself has no cost associated with integrity compared to Cassandra or the others. If you do away with foreign keys, the only "slowdown" would be due to primary/unique key constraints, which *any* map type storage with incur, because checking unicity is O(1) if you're indexing at the same time. Now, there *is* a cost associated with transactional integrity, but that is a latency and not throughput problem. To simplify, if you require transactional integrity, you need to flush and thus wait for the seek and the platter to rotate to wherever the data needs to be written. If matters little if you're committing 1 or 1000 transactions, once you're there the disk bandwidth can take it, it's getting there that is the issue, and that means latency (ignoring SSDs).

Why does it matter then?

Because every single DB interface in existence is synchronous. So while the DB can handle 1M TXs a second, that would require 10000 threads on the web side, each working for a negligible amount of time and waiting for 10ms. And that doesn't work.

Since they can't fix that, the only option is to have the request complete in .1 ms. Then you only need 100 threads. But then you need to do away with transactional integrity. They decided they could do away with that. Fair enough. But the problem there is not RDBMS themselves, it's the sorry APIs and drivers we have to work with.

Speed *isn't* scalability (1)

Colin Smith (2679) | more than 4 years ago | (#31250556)

Speed is latency. (how long it takes)
Scalability is throughput. (how many concurrent). Or put another way; Speed is the quality, throughput is the width.

who cares what twuufter is running off.

Well, developers, and their managers do. They're nothing if not fashion victims.

RDBMS aren't the be all and end all of scalability (or speed, they perform a shit load of management functions you may or may not need). While attempting to scale conventional rdbms you get into write consistency problem, lookup performance problems unless you specifically design your data structures properly. You end up fighting with the relational data model.

Most developers never even think about it, they just develop against their local mysql install and are overjoyed that their app actually runs. Not all apps even need an rdbms. I've seen apps with a single table, two columns, one of which is a key and it's running on an rdbms, because that's what you do... The words WTF sprang to mind.

Re:pfffft twatter tweeter (1)

tokul (682258) | more than 4 years ago | (#31250644)

Is there really a huge issue with rdbms speeds? Well if there is something there, that's what needs to be looked at. If RDBMSs are not fast enough, that's just an opportunity to work more on them to speed them up.

WW2 and Korea called.

Is there really huge issue with those propeller plane speeds. We can always speed them up, right. Fastest prop planes reach 850-870km/h. me-262 reached 900 km/h. Mig-15 went to 1075 km/h.

If other tools are faster and better than rdbms, then why people should waste their time with slower option.

Re:pfffft twatter tweeter (1)

roman_mir (125474) | more than 4 years ago | (#31250774)

you really should have left that comment to the 'bad analogy guy', he could make it sound good.

If other tools are faster and better than rdbms, then why people should waste their time with slower option.

- faster and better, ha? So you don't really mind if your bank switches from its datastore to a 'faster and better' NoSQL system, whatever the latest fad name is? I mean what's a few dollars not rolling back in a transaction that fails when your employment check is deposited?

Propeller would have been a file in a makeshift file system, we use jet engines now for large commercial aircraft now, you don't see the ramjets on those though or rocket boosters, do you?

Re:pfffft twatter tweeter (1)

Knowbuddy (21314) | more than 4 years ago | (#31251480)

I don't think you understand the niche that NoSQL databases are trying to fill.

The more interesting aspect of all of this 'NoSQL' movement is how they believe that if they achieve some speed improvement against some relational databases, how that makes them so much better.

It's not a black and white, panacea-type situation. Relational databases are good at some things, non-relational databases are good at others. Where non-relational databases are better is at solving very specific problems, many of which happen to map directly to the needs of web developers.

A Viper is a fun car to take you to and from work, but it's probably not the best to shuttle around a little league baseball team--that's what minivans are for. (Whether the Viper is the relational or non-relational database in the analogy is up to you.)

I teach a course titled Advanced Database Concepts, so I'll give you the same example I give my students: blogs. It's the sort of canonical example--I didn't make it up.

To show a blog's home page, you need a list of recent posts. Each post is probably associated with a category, maybe some tags, and and author. Just to get that data, you're looking at joining 3 tables: Posts, Categories, and Users. What if you want a comment count? That's another join, and the query just got hairier--do you do a simple aggregation (join then group), or do you see that might be inefficient and so transform it into a harder-to-read-but-more-efficient subquery? That might even involve a fifth join, if you have registered user accounts and avatars for your commenters.

All of which is fine and good until you're running LiveJournal or WordPress.com and you have millions of bloggers generating hundreds of millions of posts and who knows how many comments. With beefy machines and proper indexes you're probably okay ... but I wouldn't want to be the DBA who had to tell management that a new column needed to be added to any of those tables.

Enter NoSQL/non-relational databases: why not fetch everything with just one query? (I'd show you some JSON, as that's what many of the NoSQL databases speak, but the /. filter considers it too much junk.) You put your comments in the same document as your posts, and the replies to those comments in child arrays, and the user info right inside the comments. If your users can't change their username, this isn't a bad solution. There are other tricks, but the point is that you reduce everything down to a single denormalized query.

This design makes it trivially easy to build data-driven web pages, as effectively every web language has a JSON deserializer. No ORM impedence mismatch, and you get horizontal scalability pretty much for free.

If you don't really need a database to run your 'website', then who cares if you use flat files or an in memory hashmap for all your data needs?

Because it's still a database, even if it's non-relational. You're still doing inserts and updates and deletes, you just get a nice hunk of denormalized clay to play with instead of the normalized rigidity of Tinker Toys.

I think that relational databases are good at what they do and that many projects may not need them, but if you do need them on the back end, you will end up with them on the back end.

But that's the point I think you're missing: until relatively recently, relational databases were the only game in town. Relational databases are ubiquitous because they solved the problems of the 60s-90s. They aren't going anywhere, as those types of problems (financial, transactional, etc) aren't going anywhere. But now we have a relatively new class of problems (graphs, etc) that need to be nailed down just as thoroughly. Many web applications are straining to fit within the relational model, and this explosion of NoSQL software is because people are realizing that all that straining can't be good for them.

Is there really a huge issue with rdbms speeds?

Others may disagree with me, but I make the point that it's not the speed that is the problem--it's the scalability. Sure, you can keep throwing memory and horsepower at the problem and hope that covers it. Partitioning data across database servers is a tough problem, with no perfect solutions. But why keep fighting on the bleeding edge of technology like that? NoSQL databases are designed to be easily horizontally scalable from the start (in many cases you wouldn't believe how easy it is), even though they may lack the huge feature set of an Oracle or Teradata solution.

But just to reiterate: non-relational databases aren't a panacea, either. Each type of database has it good points and bad points. Each solves a set of problems, some of which overlap and some don't. The reason you should care is because now you have more options--you're not stuck trying to wedge your system into a relational model if you don't want to. And isn't /. all about freedom of choice?

Re:pfffft twatter tweeter (1)

roman_mir (125474) | more than 4 years ago | (#31251664)

Freedom of choice, definitely. I had projects just recently I used property files as a database - inserts, deletes, updates, all in a property file. Easy enough because it is just a hash map. You don't impress me with any of it, it's not in any way new first of all, but it does not replace any RDBMS where RDBMS is needed.

My entire point is that Twooter never needed an RDBMS in the first place. They should be just fine without any database usage on the front end, and forget about JSON. The problem with them, if they can't scale right now, is that they don't do design, they just jumped on a silver bullet train. Tweuter can just as well serve static content, that's as fast as you can get. The design is obvious - generate static content and periodically replace it on the front end web servers. Done, no need for anything else. Who cares what's on the back end? They never needed an RDBMS, you see, that's my point. Just like google. To your point - we always had the choice, it's all the same stuff in different wrapping, so fine, who fights that?

I'm Reluctant (1, Insightful)

Anonymous Coward | more than 4 years ago | (#31249160)

I'm reluctant to believe that Twitter is a good technology bellwether. Twitter seems to have so many technology issues, fail whales, outages, security breeches...

SO, I'm left wondering; what does this move say? Does it say that Cassandra is so bad that Twitter is using it? Or does it say that a fail whale population boom is imminent?

Re:I'm Reluctant (2, Insightful)

binarylarry (1338699) | more than 4 years ago | (#31249496)

Twitter's only moving to this new database written in Java because everyone else is.

Intersystems Caché (1)

paugq (443696) | more than 4 years ago | (#31249600)

They should move to Intersystems [intersystems.com] Caché [intersystems.com] . SQL, objects, XML and even MUMPS. It will make equally happy SQL and NoSQL fans. And it's damn fast. Much leaner than Oracle, DB2 or Informix, too. Excellent support. Extremely good. Not cheap, thought.

Re:Intersystems Caché (1)

edmicman (830206) | more than 4 years ago | (#31250342)

Not cheap, though.

That might be part of it....

Re:Intersystems Caché (0)

Anonymous Coward | more than 4 years ago | (#31252274)

How does the write performance scale across several servers ("scale horizontally")? That is what NoSQL is all about.

Don't want to install Cassandra (2, Funny)

einhverfr (238914) | more than 4 years ago | (#31249820)

I hear Cassandra is really a trojan. Can anyone verify? I don't want a trojan on my computer.....

To the clueless mod (Homer Simpson, is that you?) (1)

einhverfr (238914) | more than 4 years ago | (#31251392)

Flamebait?

Do I have to spell out the joke to people?

Or is it just that nobody reads Homer anymore.

Twitter needs scalability experts (5, Interesting)

Heretic2 (117767) | more than 4 years ago | (#31250104)

I love how ass backwards twitter has always been with learning how to scale their 90s infrastructure up. I remember when they called out the Ruby community because they didn't understand MySQL replication and memcached.

I guess without a profit model they couldn't use a real RDBMS like Oracle. EFD (Enterprise Flash Drive) support anyone? 11g supports EFD on native SSD block-levels. Write scale? How about 1+ million transactions/sec on a single node Oracle DB using <$100K worth of equipment and licenses? Anyway, I've built HUGE databases for a long time, odds are most of you have interfaced with them. Just because it's free and open-source doesn't make it cheap.

I love FOSS don't get me wrong, but best-in-class is best-in-class. I only use FOSS when it happens to be best-in-class. I laugh at how none of the requirements included disaster recovery. No single point of failure does not preclude failing at every point simultaneously. EMP bomb at your primary datacenter anyone?

Re:Twitter needs scalability experts (1)

MindVirus (1424817) | more than 4 years ago | (#31250408)

If you have dealt with an EMP at your primary datacenter, you are a more hardened sysadmin than us all.

Re:Twitter needs scalability experts (2, Insightful)

guru42101 (851700) | more than 4 years ago | (#31250848)

I've never dealt with an EMP but a more realistic threat with similar effects would be planning for a hurricane or earthquake. I used to work at a international bank and we had to deal with both (offices in FL and CA). For the most part the best solution was to have an identical setup at another office and having all applications available via VPN and/or web access. We had a separate pipe that was used only for backup data transfers. The DB transaction logs were written both locally and remotely. All user files saved to the server were immediately copied to the backup server. On several occasions the systems were tested due to black/brownouts. The users were sent home where they could work just as effectively as the office.

Our general emergency plan for hurricanes (I worked at the FL office we used the CA office as our backup). Was to let the users go well in advance of the hurricane and switch CA to being our primary servers with FL as the backup. Once the users were settled then they could continue working from home. The only way we would be screwed is if a hurricane and earthquake happened simultaneously. At that point we'd have to restore VM backups on hardware located at the main corporate offices in NYC or Sydney.

Re:Twitter needs scalability experts (1)

Jazz-Masta (240659) | more than 4 years ago | (#31251034)

you are a more hardened sysadmin than us all.

You have no idea...

http://xkcd.com/705/

Re:Twitter needs scalability experts (1)

msimm (580077) | more than 4 years ago | (#31250618)

...real RDBMS like Oracle...

Holy fuck, the right tool for the right job, please? Oracle does somethings for some markets really well but for the rest of us who don't need such a high degree of transactional safety that $90k + two-node RAC price tag might just end up taking your great web 3.0 business through development, maybe early beta before you begin liquidating assets. That's per-processor licensing too on a database that scales vertically well (very well really) but not horizontally well (sharding anyone?) so the better your project does the more processor licenses you'll be looking at, and using higher-priced hardware to do it too (because cheap boxes scale best horizontally).

So sure, if you can afford it get the big iron and with any luck your industry works with the kind of margins you'll never even need to know the cost of going that way. Personally, after having done the Sun/Oracle thing I hope to never find myself sitting at a business meeting trying to figure out how we can meet capacity demands after we've run out of money paying for high priced hardware and license fees.

I'm glad products like Oracles exist, even somewhat impressed by them, but not every project will need them and there are very real cost considerations that should also be taken into account. Know your business and for the love of God, do a thorough survey of all the available tools before you commit to one.

Re:Twitter needs scalability experts (0)

Anonymous Coward | more than 4 years ago | (#31251042)

...real RDBMS like Oracle...

So sure, if you can afford it get the big iron and with any luck your industry works with the kind of margins you'll never even need to know the cost of going that way. Personally, after having done the Sun/Oracle thing I hope to never find myself sitting at a business meeting trying to figure out how we can meet capacity demands after we've run out of money paying for high priced hardware and license fees.

Oracle runs on almost anything, including cheap Linux boxes, so hardware costs are rarely the bottleneck unless you're locked into a contract with a hardware vendor. (RDBMS license costs are a different matter.)

Then again, most apps I've seen are badly designed around the DB - especially if you let the developers design their own schema - so I'm not surprised they have scaling problems. Most well-designed apps running on Oracle can take huge loads without sweating.

Re:Twitter needs scalability experts (1)

msimm (580077) | more than 4 years ago | (#31252124)

You're either intentionally missing my point or have simply missed it.

Sure, you could run Oracle on x86 desktop hardware. You could even go so far as buying two cheap e-machines and run them both as Linux-based RAC nodes. Granted, those shitty nodes with the fibre-channel attached disk array would still have set you back probably over $90k with the additional hardware and licenses.

Eventually, if you're lucky enough to see rapid or continued growth you'll find yourself moving onto better enterprise grade gear and the best way to push performance (aside from application/network/database optimization) it to begin your vertical scaling by increasing your available memory, disk speed (SSD cache disks, etc) and total number of cores.

Oracle is amazing with multi-core architecture. I've personally performed days, probably several weeks, of stress testing and watching Oracles core/page/memory use which, as compared to MySQL for example, is a thing of beauty.

But that all comes with a per-core license cost attached which can make capacity increases cost significantly more in database licenses then in physical hardware. That's fine if you have a business that really needs some of the features that come with Oracle. But I'd argue that the majority of businesses like Twitter or Facebook need flexible, cheaply scalable, high-volume read-writes more then they need the reliability or datamining/statistical features that come with the Oracle price tag.

But to each their own. The premise of my post was proper evaluation, the right tool for the right jobs and there are certainly times Oracle is/will be that tool.

Re:Twitter needs scalability experts (1)

mini me (132455) | more than 4 years ago | (#31250684)

They are seeing about 1/2 million transactions per second with this setup based on the information given, but no word of what their cluster consists of. If it is just a handful of generic PCs, $100,000 for your setup looks pretty expensive.

Re:Twitter needs scalability experts (2, Informative)

ryansking (1752556) | more than 4 years ago | (#31251402)

You're right, I failed to mention disaster recovery– it was something we looked at, its just been awhile since we went through the evaluation process, so I've forgotten a few things. We actually liked Cassandra for DR scenarios – the snapshot functionality makes backups relatively straight forward, plus multi-DC support will make operational continuity in the case of losing a whole DC a possibility.

Re:Twitter needs scalability experts (1)

codepunk (167897) | more than 4 years ago | (#31251406)

I love oracle it is a fine database, would I personally buy it? Nope, but as long as
it is OPM (Other Peoples Money) I am perfectly fine with it. Now say I was designing something
like a medical records system oracle would be a no brainer. Missing a couple of tweets here
and there who is really going to care.

Re:Twitter needs scalability experts (0)

Anonymous Coward | more than 4 years ago | (#31252514)

Cool! Let's design twitter in a slashdot forum! That sounds fun!

EFD's are 10x cheaper now than when Twitter started. But, we'll ignore that. You need two nodes for fail-over, probably. So $200k.You also need some hardware and licenses for test and development environments. So, probably $250k minimum?

Twitter is doing ~50 million tweets per day. That's about 600 tweets per second, probably with peek load around 6k/s. I'm just guessing about the peek load. So, your point load capability of 1 million simple row writes per second with no indexing is extreme overkill. But, you aren't measuring the right things. The critical problem is read throughput and latency when under write load. Twitter currently supports an average page read load of 1600/s. So, there are probably bursty point loads approaching 10k/s.

You need to write those raw tweets into a FollowTweet table like |ID|UserID|Date|AuthorID|AuthorName|TweetID|Tweet|, or you'll end up doing essentially random searches across the core tweet table for every page view. The dominant query pattern for this table is

SELECT TOP 50 * FROM FollowTweet WHERE UserID = @pUserID ORDER BY Date DESC

You probably need a clustered covered index to make that query fast or you'll do a lot of random seeks on read. I've already denormalized that a bit for you to get rid of the join against the core Tweet table. You can argue about that if you like. Test both if you disagree.

So, what's your write rate to a table that looks like that while maintaining a clustered covered index, after writing 50 million tweets per day for 6 months? It's far lower than 1 million transactions/sec. But, I'm curious about the exact number.

Twitter also provides free text search. So, add in a free text search index on tweets, ordered by time of post descending.

Now that you're maintaining a free text index on the core Tweet table, how many writes per second can you sustain, after writing 50 million tweets per day for 6 months?

Much more importantly, how many user queries per second can you sustain against the FollowTweet table, and what is the 99.9% latency of a read? And, how many free text searches per second can you sustain, and what is the 99.9% latency of a free text search? And, how long does it take to completely write a tweet and make it available to all followers and in the free text index?

I'm sure you know that latencies are much more important than throughput to the user's subjective experience. I bet you can't sustain the necessary write load on a single node while maintaining good read latency, So, you're going to need some read only mirror nodes to scale reads. So, add that hardware and license cost to your system.

I could, however, take 12 commodity 1u boxes with cheap disks and 8GB ram, install mongo on them and turn them into mirrored pairs, and handle all of the load I just described with consistently low latency. That'd be about 133/s direct key lookups per mongo box for a page view. And, 100/s writes per mongodb box. Each mongodb box should scale to 10x that load without worry, so we should be fairly safe for point load. I'd have to see the real twitter numbers to be sure. They do seem to have high burst peeks. We can snapshot the mongodbs for disaster recovery, and I'm mirrored for fast failover if a single node goes down.

You're going to need another 6-12 front end boxes to render pages, with either storage system.

The total cost for the release environment machines would be ~$24k from a reputable server hardware builder. So, we're looking at $24k for the NoSql FOSS design, vs. $200k for the Oracle, big iron design. We're both ignoring network equipment, bandwidth, and hosting costs.

Also, with an oracle design, you get to spend your $500k a year on operational specialists with pagers. With FOSS, you can probably spend $240k a year on that staff.

Re:Twitter needs scalability experts (1)

lawpoop (604919) | more than 4 years ago | (#31252634)

I only use FOSS when it happens to be best-in-class

Just curious, what FOSS have/do you use?

Too bad they dont about TPF/ZTPF and TPFDB/ACPDB (1)

emes (240193) | more than 4 years ago | (#31250600)

It's always funny to read things written by people who obviously are inexperienced with high volume transaction processing in the mainframe environment. The systems behind airline, rail, and hotel reservations as well as emergency response messaging often are built on IBM mainframes using TPF/ZTPF as the operating system and
TPFDB(formerly known as ACPDB) as the underlying database. If someone would take the time to study TPFDB, they would notice its nonrelational character, as well as some interesting similarities to what the Cassandra developers unknowingly chose to do. By the way, these systems are happily handling 10K-12K transactions per second without bunny farm racks of servers.

Sometimes progress is not always about what will be done, but understanding the benefits of older things that have been done.

Re:Too bad they dont about TPF/ZTPF and TPFDB/ACPD (0)

Anonymous Coward | more than 4 years ago | (#31251190)

> By the way, these systems are happily handling 10K-12K transactions per second without bunny farm racks of servers.

The airline systems have entire *data centers* instead, to say nothing of the enormous transaction processing infrastructure inbetween.

Re:Too bad they dont about TPF/ZTPF and TPFDB/ACPD (0)

Anonymous Coward | more than 4 years ago | (#31251816)

Nice point. Thanks for this. Data processing/transaction is not really my area of expertise, but I've always worked with the thought that nothing I'm doing is new on a technical level. This goes to show it. What the F/OSS community should focus on, be it through research groups is the human computer interaction. This is a relatively new field of study - maybe 20 years old, and there's a lot less catch-up. My conspiracy theory hat of yester-year would probably take a stab that this is why oracle cut funding to the accessibility projects of sun/gnome. Just to extend the gap between free and commercial HCI offerings.

Re:Too bad they dont about TPF/ZTPF and TPFDB/ACPD (1)

einhverfr (238914) | more than 4 years ago | (#31252518)

Teradata seems to win typical OLTP and OLAP benchmarks. I would think for airline reservations and such that would be my choice of platform.

NiG6a (-1, Troll)

Anonymous Coward | more than 4 years ago | (#31250706)

Preferrably with an Inw eternity...Romeo IS DYING LIKE THE

Java / JVM Wins Again ... (1)

zuperduperman (1206922) | more than 4 years ago | (#31251430)

It's fascinating how after initially being a posterboy for the post-Java revolution Twitter is gradually moving their architecture to the JVM, piece by piece. I think it's actually a credit to them that they seem to have level heads and are evaluating technology on it's merits (where as if you talk to most of the ruby / python crowd they would rather stick toothpicks in their eyes than endorse a solution that involves java).

Re:Java / JVM Wins Again ... (3, Funny)

codepunk (167897) | more than 4 years ago | (#31251566)

Until recently I thought the same way, I would never endorse a solution that involves java. However
a recently came to the same realization that sun did when they created it. Java is a fantastic
way to over sell gobs of expensive hardware. I am a system administrator so the more hardware it takes to
run a solution the better off I am, more machines, more money and better job security. So I have now
fully jumped on the java bandwagon, java makes me smile.

Re:Java / JVM Wins Again ... (2, Informative)

zuperduperman (1206922) | more than 4 years ago | (#31252544)

Sure - but I think the whole point is that you'd be smiling even more if they were using one of the modern & trendy dynamic languages because you'd likely have 2 - 3 times the amount of hardware to look after. I'm not sure what alternative you would propose that uses less hardware but there actually aren't many that are better than the JVM these days.

Re:Java / JVM Wins Again ... (1)

DragonWriter (970822) | more than 4 years ago | (#31252550)

It's fascinating how after initially being a posterboy for the post-Java revolution Twitter is gradually moving their architecture to the JVM piece by piece.

I think its fascinating, too -- but probably in a very different way than you do. You seem to think that it is a repudiation of some mythical "post-Java revolution", when in many ways I think it is a validation of exactly the approach that was common to pushing Ruby, Python, and similar languages as more agile alternatives to Java. The appeal of tools noted for their suitability for rapid development of software that works and is maintainable, even if it isn't going to set any kind of performance records, is that it supports getting new functionality (and, thus, often new businesses) of the ground, and supports the kind of rapid change that is often necessary when a product is first exposed to a mass market, gets used in new and unexpected (by the developers) ways, etc., and that the right time to optimize performance is often once the concept is validated, and trying to do too much of that too early means you lose agility in introduction and early development of the product.

Shifting, component by component, to more "enterprisey" solutions as a service/product matures is entirely consistent with that understanding.

(where as if you talk to most of the ruby / python crowd they would rather stick toothpicks in their eyes than endorse a solution that involves java).

I don't think that's particularly true. Sure, some of the people in the any language community are going to be partisans for that language exclusively, but the Ruby community (which I'm more familiar with than the Python community) seems particularly friendly to Java as a platform, and to Ruby being used in the role of a "glue" language instead of an exclusive language.

In the case of the Ruby community, I think that the appearance of anti-Java sentiment there stems largely from the the early days of Rails, where lots of people were pushing Rails by extolling (often in a rather hyperbolic manner) its virtues as compared to enterprise-oriented, XML-configuration-heavy, Java frameworks.

They considered Voldemort (0)

Anonymous Coward | more than 4 years ago | (#31251860)

But found that its backup policy required horcruxes.

Open Source Parallel Databases (1)

cervo (626632) | more than 4 years ago | (#31252652)

A lot of the complaints from NoSQL seem to be regarding DBMSses being too slow and SQL being too hard. And yet a lot of them invent query languages/query languages similar to SQL. Supposedly Oracle scales up really well. There is a paper that compares mapreduce to parallel databases and Hadoop takes a huge beating via the RDBMSes in performance. Now the funny thing is that Oracle was not included, yet most content that if you pay enough Oracle scales really well. DB2 also scales, because in 1999 I worked at a place with terabytes of database space and they had a few nodes running DB2 on AIX boxes and seemed to be getting adequate performance.

But most open sources databases seem to not be able to compete with the likes of the commercial parallel databases. But it seems like an open source parallel database would do a lot to silence many nosql critics. There is still the complaint about needing to define a schema, however if you are not exploring the data and are processing the same data over and over again, it seems like a good idea to define a schema anyway, that way you can better detect files that don't conform.

How hard can it be? (1)

FloydTheDroid (1296743) | more than 4 years ago | (#31252822)

I was going to try to write something funny about twitter only needing three tables to run and how hard is it to change but then I thought about how much money they're going to make off those three tables and I started to cry.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>