Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Twitter Not Rocket Science, but Still a Work in Progress

ScuttleMonkey posted more than 6 years ago | from the time-to-start-over-on-some-things dept.

The Internet 111

While it may not be rocket science, the Twitter team has been making a concerted effort to effect better communication with their community at large. Recently they were set-upon by a barrage of technical and related questions and the resulting answers are actually somewhat interesting. "Before we share our answers, it's important to note one very big piece of information: We are currently taking a new approach to the way Twitter functions technically with the help of a recently enhanced staff of amazing systems engineers formerly of Google, IBM, and other high-profile technology companies added to our core team. Our answers below refer to how Twitter has worked historically--we know it is not correct and we're changing that."

Sorry! There are no comments related to the filter you selected.

twitter hate... (4, Funny)

bsDaemon (87307) | more than 6 years ago | (#23630915)

Man, who *IS* this guy that his trolling has gotten its own front page article?

Re:twitter hate... (1, Funny)

nawcom (941663) | more than 6 years ago | (#23631167)

heh. as soon as i read the title I just had to see who got first post on it, and which insult they are going to use on twitter.

Re:twitter hate... (4, Interesting)

Jeff DeMaagd (2015) | more than 6 years ago | (#23631251)

Dude, I'm not seeing the hate here. May be an eye of the beholder kind of thing.

I do have two concerns about the service, apart from the supposed outages.

One is that they don't seem to have any plan that can make money on their service. I suppose they are trying to build up a service that they can sell to Microsoft, Yahoo or Google, but how much of the user base is going to tolerate ads if any of them bought it as a platform to serve ads?

Another is that they are very quick to cancel passwords due to inactivity, requiring a request to get it reset. I don't know if it's one week or one month, but it's as if they don't want you at all if you don't log in at least once a week. However often they do it, I've not seen any other service do it so quickly.

Re:twitter hate... (2, Informative)

gEvil (beta) (945888) | more than 6 years ago | (#23631285)

It's a joke... [slashdot.org]

Re:twitter hate... (2, Funny)

bsDaemon (87307) | more than 6 years ago | (#23631299)

I was attempting to make a joke based on the fact that everyone seems to hate the user named "twitter" who allegedly has a vast army of fake accounts he uses to reply to himself and create little Socratic dialogs to solicit karma.

of course, now its not funny anymore.

twit, twitter, twittest? (1)

tepples (727027) | more than 6 years ago | (#23632333)

But is the twit named after the service, or is the service named after the twit?

Re:twitter hate... (1)

niceone (992278) | more than 6 years ago | (#23636099)

Another is that they are very quick to cancel passwords due to inactivity

Dude, if you can't be bothered to tell people what you had for breakfast each day maybe twitter's not for you.

Re:twitter hate... (1)

neuromancer23 (1122449) | more than 6 years ago | (#23637735)

>> One is that they don't seem to have any plan that can make money on their service

Who cares when you've still got venture capital to burn through?

Re:twitter hate... (3, Funny)

rts008 (812749) | more than 6 years ago | (#23631355)

I have been confused about this also.
I DO hang out at places other than /., but until all of the '/. twitter' troll posts, I had never even heard of the twitter weblog. I assumed they were connected somehow and totally dismissed the twitter weblog.

Now that I have seen the twitter weblog, I'm still tempted mentally to associate the two, and want to spam it with goatse.cx links to somehow balance the shite. (just kidding-sort of)

I could not get an answer from the twitter weblog about how many sock puppet accounts you could register though, so I guess I will pass on the whole thing, and attempt to just filter 'anything twitter' from my web experience, as any more in my mind twitter==mindless rant.

And before some twitter (weblog) fanboy tries to bust my chops, yes I know I could be 'missing out on something' with that attitude, well that also goes for a million and one other websites that I currently don't know about. Save it!

Re:twitter hate... (3, Funny)

infonography (566403) | more than 6 years ago | (#23631643)

I thought that the Twitter product was a free roving troll for all weblogs and somebody had figured out a way to profit from it.

So I went to the site and found I was correct. Its a AI-based troll that talks up products. Sadly there is a glitch where it also talks about toilet related obsessions.

Re:twitter hate... (1)

Em Adespoton (792954) | more than 6 years ago | (#23632633)

Funny thing; I knew about the twitter blog before the twitter troll, and figured someone was using the twitter service to spam slashdot instead of uploading posts to the blog. Took me a while to figure out it was two different things.

Then again, I guess someone could set up a twitter-like slashdot posting service....

Re:twitter hate... (0, Flamebait)

rdavidson3 (844790) | more than 6 years ago | (#23631461)

Is that you Twitter?

And Twitter is... (1)

Colin Smith (2679) | more than 6 years ago | (#23630933)

Answers on a postcard please.

 

Re:And Twitter is... (1)

Hatta (162192) | more than 6 years ago | (#23631035)

...for twits.

Re:And Twitter is... (2, Insightful)

Pope (17780) | more than 6 years ago | (#23631095)

Will "a rather pointless waste of time" fit on a postcard? I guess I'm not 'hip' enough to care about Twitter, it all seems a bit pointless for anyone who's not some interweb celebrity.

Re:And Twitter is... (5, Informative)

gEvil (beta) (945888) | more than 6 years ago | (#23631227)

...a geek-celeb circlejerk?

Alright, poop-time (4, Funny)

spun (1352) | more than 6 years ago | (#23631895)

I'm a twitter-shitter! [penny-arcade.com]

Re:And Twitter is... (0)

Anonymous Coward | more than 6 years ago | (#23633381)

I thought you might like an answer that was actually, you know, /helpful/. Take a look at this [commoncraft.com] .

This is correct (1, Funny)

Anonymous Coward | more than 6 years ago | (#23630937)

Making 10 slashdot accounts is not rocket science.

Re:This is correct (1, Funny)

Anonymous Coward | more than 6 years ago | (#23631931)

... says an AC.

What do you expect with RoR? (0)

Anonymous Coward | more than 6 years ago | (#23630949)

Scaling. *snicker*

it's true... (0)

Anonymous Coward | more than 6 years ago | (#23630959)

...social networking sites affect any hope of better communication.

Oh well, as long as people will doggy-paddle the slough of mediocrity, less competition for me.

That's "effect", not "affect" (5, Informative)

XanC (644172) | more than 6 years ago | (#23631003)

This is one of those cases where "effect" is a verb.

Obligatory xkcd (5, Insightful)

Krishnoid (984597) | more than 6 years ago | (#23631193)

Grammar [xkcd.com] can be fun!

Re:That's "effect", not "affect" (2, Insightful)

devotedlhasa (1298843) | more than 6 years ago | (#23631223)

affect better communication.... negatively.

Re:That's "effect", not "affect" (1)

morgan_greywolf (835522) | more than 6 years ago | (#23631225)

Grumble, grumble! Just who effected the invention of this stupid language anyway?

Re:That's "effect", not "affect" (1)

zurtle (785688) | more than 6 years ago | (#23631439)

I don't know about you, but the use of "effect" as a verb always makes me think of going to the toilet. Incidentally, so does twitter, Twitter and curries from the dodgy Indian restaurant down the road...

Re:That's "effect", not "affect" (1)

SheeEttin (899897) | more than 6 years ago | (#23631281)

Maybe they're taking said "better communication" and attempting to affect (i.e. change) it...?

Re:That's "effect", not "affect" (1)

MattPat (852615) | more than 6 years ago | (#23632011)

Well then I suppose they would just be stupid, wouldn't they? :P

Big Brother(s) (-1)

Vectronic (1221470) | more than 6 years ago | (#23631059)

No offence to the people at Twitter, and maybe not the people who recently migrated there. But am I alone in being hesitant to welcome people who formerly worked at IBM, et al, and even Google?

Granted, they seem to more directed at the physical performance of the system, but I see this as somewhat of an inroads to having large companies control how Twitter works, and I would much rather have some no-name 'genius' from a basement come and work for them, than people who could (possibly) be 'tainted' by how a large coproration works and handles things, and what friends, they may have (who married into, etc)

I appreciate Twitter, and what it allows for (much like WikiLeaks, or even Slashdot), But I just find having big boys come in and having control over something thats (potentially, and already has been) so useful.

Re:Big Brother(s) (1)

Vectronic (1221470) | more than 6 years ago | (#23631089)

uh... "kind of scary" (to finish my sentence)

Re:Big Brother(s) (5, Insightful)

Anonymous Coward | more than 6 years ago | (#23631229)

Hiring folks who used to work at IBM or Google is not the same thing as "large companies control[ling] how Twitter works." Some day, you'll have a job and you'll understand that. [Sorry to be an asshole about this, but your comment just shouts "teenage kid who's never had a serious job."] People with experience with large-scale applications may already know solutions to some of the problems Twitter is seeing. Those solutions aren't always in the text books; and if they were trivial and obvious, then such applications would be much more common.

Re:Big Brother(s) (1)

Vectronic (1221470) | more than 6 years ago | (#23631421)

Well, I didn't say it was the same, just that it is a possibility.

And am aware that people who formerly worked at places such as Google (especially) and IBM would most likely have a better understanding of how a large-scale system like this works.

But, Google itself (along with a lot of other major internet/software/hardware companies) were started in a 'grass roots'/garage/basement method.

It seems to me that more often than not, bringing in the help of (formerly employees of) large corporations tends to conform them to that previous corporation, even if its not intentional.

Not that it doesnt happen in smaller ones, "back when I used to work for [...], we did it this way..."

To argue against myself on your behalf though, having someone from a larger corporation, can be benificial aswell, because they may have been fired, or quit because the larger corporation wasn't listening to them, or was doing something in a way that went against their ideology, thereby possibly preventing said smaller company from becomming similar to the larger one.

My main concern is that it will lean towards the former though, rather than the latter.

Re:Big Brother(s) (3, Insightful)

Otter (3800) | more than 6 years ago | (#23631563)

To argue against myself on your behalf though, having someone from a larger corporation, can be benificial aswell, because they may have been fired, or quit because the larger corporation wasn't listening to them, or was doing something in a way that went against their ideology, thereby possibly preventing said smaller company from becomming similar to the larger one.

Like the AC said, I think you're wildly exaggerating how ideological workplaces are, particularly from the point of view of a server monkey.

Re:Big Brother(s) (2, Insightful)

Antique Geekmeister (740220) | more than 6 years ago | (#23633341)

Oh, ideology affects cage monkeys. The use of open source versus closed source, burchasing enough licenses for the software you use, and making '99.999%' uptime actually mean that instead of simply hiding downtime, and forcing people to spend more time documenting how much time they spent on a task than actually doing the task are all policies I've seen affect server work. Those may not be idologies per se, but they certainly arise from philosophies about how things should be done.

Re:Big Brother(s) (1)

edittard (805475) | more than 6 years ago | (#23635997)

For Pete's sake, learn how to use commas. Reading that penultimate paragraph makes me think of William Shatner.

Re:Big Brother(s) (0)

hkgroove (791170) | more than 6 years ago | (#23631377)

"I would much rather have some no-name 'genius' from a basement come and work for them, than people who could (possibly) be 'tainted' by how a large coproration works and handles things, and what friends, they may have (who married into, etc)"

Would you choose Anthony Anderson or Kevin Smith?

Re:Big Brother(s) (1)

Vectronic (1221470) | more than 6 years ago | (#23631499)

To marry? probably Kevin Smith. :P he's kinda sexy when he's quiet.

If you mean, someone who has experience directing movies, as apposed to someone who (seems) to only have experience being directed, and probably has "lots of ideas" on how to direct.

I'd be willing to take advice from either with an equal amount of salt.

Unless of course I was planning on directing, or writing a movie similar to one of Kevin Smiths, enwich case, the answer is obvious.

However, as for "working with", I would probably be more comfortable working with Anthony, given that we would both be "sort new" to the idea (directing, making our own movie, etc)

If you meant something else, well, enlighten me.

Re:Big Brother(s) (1)

hkgroove (791170) | more than 6 years ago | (#23631711)

They both played basement dwelling 'geniuses' in two abysmal movies last summer, Die Hard 4 and Transformers.

Re:Big Brother(s) (1)

hkgroove (791170) | more than 6 years ago | (#23631751)

And the correct answer however is actually Laslo Holyfeld

Fixed it for you.... (1, Funny)

adamofgreyskull (640712) | more than 6 years ago | (#23631459)

"No offence to the people at Twitter, and maybe not the people who recently migrated there. But am I alone in being hesitant to welcome people who formerly worked at IBM, et al, and even Google?

Granted, they seem to be more directed at the physical performance of the system, but I see this as somewhat of an inroad to Twitter actually working, thereby requiring me to acknowledge its existence. And I would much rather have some no-name website, where people living in their mother's basements come to tell a world that would rather ignore them that they just had a shit, go under, than sap the life-blood of a decent technology company.

I have no idea what Twitter could be useful for, but I just find having big boys come in and having control over something that's (potentially, and already has been) so useless, uh...kind of boring."

Honestly though, what? They hired some people from real technology companies who might have the first clue of how to do what they should already have been doing. I have only the vaguest idea of what Twitter is, based on this [penny-arcade.com] and having checked it out, have had my suspicions confirmed. Call me a curmudgeon, but they just sound like they're trying desperately to survive in a marketplace already brimming with (better) competition by hiring experienced technologists. Seriously, how long before 'phones that can handle Facebook are the norm?

Re:Fixed it for you.... (1)

Vectronic (1221470) | more than 6 years ago | (#23631703)

I can agree that for the most part Twitter is just another black hole like YouTube, filled with endless amounts of drivel, but they both have good uses.

Earthquake In China (Twitter Related)
http://news.slashdot.org/article.pl?sid=08/05/13/0245240 [slashdot.org]

YouTube Refuses To Remove Terrorist Videos
http://tech.slashdot.org/article.pl?sid=08/05/20/224218 [slashdot.org]

YouTube Fires Back At Viacom
http://tech.slashdot.org/article.pl?sid=08/05/27/2350225 [slashdot.org]

And when they become "Big" they fall prey to that greed, cencorship in favor of profit.

Im in favor of anarchistic sites similar to them both (which are both declining), a sort of "anything goes" motto, with bad, comes the good. Google already has YouTube, and they may very well take Twitter along with it, they are already working together on some projects.

That said, I dont give a damn about the Social Networking side of it, just the "Micro-Blog", free-speech, etc, etc.

Re:Big Brother(s) (1)

pembo13 (770295) | more than 6 years ago | (#23631677)

because your twitter conversations are so secure right now

Twitter lulz (0)

Anonymous Coward | more than 6 years ago | (#23631079)

please tag article "whogivesashit"

Re:Twitter lulz (1)

aeskdar (1136689) | more than 6 years ago | (#23631131)

Agreed

Re:Twitter lulz (1)

hostyle (773991) | more than 6 years ago | (#23631495)

onlyrampanttaggerswouldcare

Re:Twitter lulz (1)

leamanc (961376) | more than 6 years ago | (#23631699)

If there was ever an AC that deserved some up-modding, this is the one. Don't be afraid next time and log in!

Re:Twitter lulz (1)

fractoid (1076465) | more than 6 years ago | (#23635983)

wouldn't im(not)givingashitrightnow be more apt?

No mention or Ruby/Rails? (1, Interesting)

pembo13 (770295) | more than 6 years ago | (#23631105)

Kinda surprised. I use neither Twitter nor Ruby, but seems like the talk has been than Rails wasn't scaling well. Thought the Q&A would confirm or deny this.

Re:No mention or Ruby/Rails? (1, Insightful)

Anonymous Coward | more than 6 years ago | (#23631587)

I use neither Twitter nor Ruby, but seems like the talk has been than Rails wasn't scaling well.
So you're basically flamebait. Rails might had been a factor, but this is not what it's about. It's not as easy as, rewrite it all in java or php and it will work blazing fast. Twitter needs to find a better way to reorganize or perhaps rewrite their code and it may still be in ruby as far I am concerned. Dare I also mention that their biggest overhead is not the rails framework, nor the ruby language, but their database, wich lead to their downtime in the first place. Nevertheless, big players such as IBM or Google already faced these issues, and this is what this article is about.

It's the algorithm, stupid (4, Informative)

QuoteMstr (55051) | more than 6 years ago | (#23633195)

Ruby is not Twitter's problem. The algorithm Twitter uses is the problem.

When I started irately writing this post, I wrote it in a tone that would have gotten me modded into oblivion. But then I realized that ignorance, not idiocy, drives the particular myth I'm debunking. let me educate, not flame, those of you who haven't formally studied computer science.

It's become fashionable to blame Ruby for Twitter's problems, but that's wrong. The particular choice of language doesn't matter a bit when you talk about scalability, no matter what the language or the problem.

First, sending a twitter message is an algorithm. An algorithm is just a recipe for doing something to some data. Although most computer science literature deals with more abstract and general algorithms, like those for sorting and searching, the same principles applies to even the most mundane processes, like what rm foo does to a file system, or how a database engine runs an INSERT.

One way we can talk about algorithms is to use something called Big-O Notion [wikipedia.org] , which describes the relationship between how much stuff an algorithm processes and how long it takes to run.*

It's easier to see things with examples. Say we have an algorithm and we give it three sets of data, D1 and D2, and D3, each twice as large as the last, so that D2 is twice as large as D1, and D3 is four times as large as D1.

If we call the algorithm O(1), it will take the same amount of time to process D1, D2, and D3. If we instead say it's O(N), D2 will take twice as long to run as D1, and D3 will take four times as long.

If N represents the number of users for a web application, and we want to double N, twice as many users, we'd need twice as many web servers if the bottleneck algorithms are O(N). If the database is the bottleneck, we'd need a twice-beefier database server, or some partitioning.

Things start to get interesting with O(N^2). In that case, D2 takes four times longer to run than D1, and D3 takes four times longer than D2, which sixteen times longer than D1.

That means that if we want to support twice as many users, we need four times as many web servers, or more likely, a four-times beefier database server.

It can get a lot worse than O(N^2) too, especially if you're not paying attention to complexity. For example, many graph (think social networking) algorithms can easily become O(2^N), which is a lot worse than N to a constant power.

When you try to scale a poorly-designed algorithm (pretty much anything worse than O(N)), you start running out of cores, rack space, electricity, and atoms in the universe.

One useful bit about big-O notation is that it lets us ignore piddly details that don't matter. Say we had an O(2N) version of the O(N) algorithm. Sure, the O(2) algorithm might take twice as long to run, but it can still handle double the data with double the capacity or double the time. Even if it's O(10N), you don't start boiling the oceans to cool your data center when you want to increase your visit capacity a thousandfold.

This observation is why the choice of language doesn't matter. If a language implementation is slow, all it does is add a constant factor to any algorithms written in that language. A Python application might be ten times slower than one written in C, but its big-O complexity will be the same.

At the worst, that means you'll need ten times as many servers as with the C web application. The increase in development efficiency writing in Python (or Ruby on Rails, or Lisp, or anything else) might make the trade-off worth it. You can deal with a constant factor slowdown.

If on the other hand, you code a wicked fast implementation of an O(N^3) algorithm in C, no amount of hardware will save you. You'll hit a number of users beyond which your servers slow to a crawl and you lose blagosphereic karma. Even if you double your capacity, or buy a four-times-beefier database server, that number of users you can service will only increase a tiny bit.

That's the situation Twitter is in. I don't know Ruby, but I hear its main implementation is pretty slow. But that's not the problem, and never is.

What is Twitter's problem is the poor algorithm. Blaming or even optimizing Ruby doesn't help the situation one damn bit. Twitter could be written in hand-tuned assembler or DOS batch files and we'd still be seeing the same scalability problems.

I'm glad to see they're addressing the issue and finally choosing a new algorithm.

But the whole discussion surrounding Twitter's performance reminds me of a thread I read once, in which some PHP developers were benchmarking "echo" versus "print". The funny thing? Right above the line in question was a "SELECT * FROM huge_table".

It's important to be literate in computer science (even without a degree) in order to better understand concepts like algorithmic complexity. A good algorithm can make the difference between good and great software.

Granted, without knowing computer science, a smart programmer might identify an algorithm that's slowing things down and rewrite it. But he won't recognize the slowness as a general property of algorithms. He won't label in O(N^2). If he finds and fixes enough slow algorithms, he might develop an intuitive revulsion toward "slow" algorithms.

But that intuition takes a long time to develop, and it's never quite as good as having formally studied complexity. Someone who's studied complexity can immediately pick out potential bottlenecks, or realize that combining two algorithms can yield a terribly bad one. Someone who's been exposed to what "O(2^N)" will be much less likely to write an O(2^N) algorithm.

It's a shame that there isn't a non-intimidating introduction to computing theory. The Wikipedia article I referenced above is correct, but inaccessible. Nobody without a math background will get much out of it other than "these geeks are making it harder for me to do my job".

--
* Yes, technically, big-O means worst case running time, but the distinction is irrelevant in this basic overview.

Re:It's the algorithm, stupid (0)

Anonymous Coward | more than 6 years ago | (#23634319)

There are so many variables that you don't know in twitter's case that there's no way you could speak from any position of authority. Unless you work for them, in which case, fix your shit instead of posting on Slashdot.

Re:It's the algorithm, stupid (0, Redundant)

recoiledsnake (879048) | more than 6 years ago | (#23635219)

Some web frameworks make it easy to scale and some don't. I don't why you need to get all worke d up when people say Rails does not scale easily as well as other frameworks. Rewriting Twitter in C may not yield much benefits but if it makes it easier to go out and buy 10 machines and make it scale even with a algo that isn't super perfect or elegant, that's better than having a framework that doesnt' do that.

Re:It's the algorithm, stupid (2, Insightful)

QuoteMstr (55051) | more than 6 years ago | (#23635307)

That's the beauty of the web. At a basic level, it's stateless. Web frameworks don't have any concept of communicating with other clients. The highest level they'll work at is the individual session. You only see complexities worse than O(N) when clients communicate with each other, and that communication must be an entirely application-specified thing. The web framework and language have nothing to do with this communication between clients, and so have nothing to do with scalability writ large.

Re:It's the algorithm, stupid (1)

recoiledsnake (879048) | more than 6 years ago | (#23635613)

Uh? Who was talking about communication between clients here? Did u reply to the wrong post by accident?

Re:It's the algorithm, stupid (2, Insightful)

mcrbids (148650) | more than 6 years ago | (#23635243)

Well written post! It's amazing how few people really understanding the difference between performance and scalability. Getting good performance isn't all that hard. Getting good scalability is much harder.

Good scalability is not about how fast something processes, it's about how much the speed degrades as the load increases. It sounds simple - but it's NOT.

Re:It's the algorithm, stupid (2, Insightful)

Gazzonyx (982402) | more than 6 years ago | (#23635415)

First let me say, that was a very well thought out and informative post. I only would like to bring up a counter point for the sake of discussion.

The language you choose may affect your ability to scale when you take its concurrency model (or lack thereof in some cases) in to account. For instance, I can have a O(1) algorithm, using a hashmap, but that doesn't mean that I'll be able to have the runtime performance of constant time. For a solid example, let's use Java (Java 6, with java.util.concurrent, as this is the concurrency framework I'm most familiar with).

I can have a ConcurrentHashMap (a regular HashMap with a concurrency wrapper) with constant time access (get and put). However, every time that I modify the internal structure of the HashMap, every other thread that is accessing the HashMap at that time is kicked out. That means I've got my optimal algorithm, but due to competition for resources, it'll have the runtime performance of an algorithm chugging along at exponential time if everyone wants to write to that data structure.

Granted, there are ways around this, but you can't just throw hardware at the problem and pretend it doesn't exist. In reality, more hardware raises the communication overhead to the process. Your language of choice may or may not scale at the same pace as others when you take in to account your concurrency needs and perhaps the backend you are given to interface. All that being said, I'm not sure how varied the runtime characteristics of various languages (say, an interpreted JIT versus native machine code) scale WRT each other. I would suspect that a JIT would have much more overhead when you layer it on top of an OSes concurrency model (handling locks within your own code, the JIT handling resources, and the OS doing the same, all at the same time... probably with a database and filesystem doing so at varying degrees). Of course, I could be wrong.

Re:It's the algorithm, stupid (3, Insightful)

QuoteMstr (55051) | more than 6 years ago | (#23635563)

Thank you for your polite reply, but I feel like I haven't adequately communicated my point.

The language you choose may affect your ability to scale when you take its concurrency model (or lack thereof in some cases) in to account. For instance, I can have a O(1) algorithm, using a hashmap, but that doesn't mean that I'll be able to have the runtime performance of constant time.


If your whole web application is bottlenecked by one hashmap, you're going to run into scalability problems as soon as you need more than one machine anyway. On the other hand, If the performance of the web application as a whole does not depend on the hashmap, then your argument is irrelevant to the scalability of the application as a whole.

I concede that a more efficient runtime environment might make better use of the same hardware, supporting, say, 70 clients instead of 50 per machine. But that's not the kind of scalability I'm talking about. Even a platform that achieved only one client per machine that scaled linearly would be better than one that handled 70 clients per machine, except that you were limited to one machine.

And yes, on one machine, a bad choice of data structure can affect scalability. But the blame for that rests on the data structure itself, not the language in which it is implemented. As an associative array, a Python hash table (dict) will scale far better than a C linked list. Why? Because one's a hash table and one is a linked list!

Which data structures are available in which language might factor into the choice of language, but it's only a convenience: you can always create your own data structure implementations.

Granted, there are ways around this, but you can't just throw hardware at the problem and pretend it doesn't exist.


Creating a scalable application means being able to throw hardware at the problem.

Let's assume you've gotten your application to scale beyond one machine anyway. That's a prerequisite for this section.

Now, if the machines don't communicate and users don't care, you automatically win O(N) scalability.

If your machines must communicate, they do so over some kind of network. The way this communication is achieved determines the scalability of the application. While some environments might have more intuitive network facilities than others (think Erlang), ultimately one can use any approach to networking with any language.

Again, we're reduced to choice of data structures and algorithms, not language, as the marker of scalability.

The choice of language does not dictate the data structure the designer of the application uses, and so the language is not a serious barrier to scalability. I concede it may be more difficult to implement efficient protocols in some languages than in others, but we're dealing with turing-complete languages here, aren't we?

I should note that languages typically thought of as "slower" are often more expressive. It often takes less effort to write efficient algorithms in expressive languages.

(Returning to our previous example, since writing a hash table is more complex than writing a naive linked list in C, a C programmer is more likely to use a linked list at the expense of scalability. In Python, using a hash table is as simple as writing {}, so an equally-skilled programmer is more likely to use the more efficient data structure, resulting in better performance in a "slower" language.)

The bottom line is that if communication between nodes is required, complexity must be > O(N). And if complexity is greater than O(N), then as N increase without bound, the communication overhead approaches infinity anyway. The key is to make that growth as slow as possible.

The tools and techniques used to slow that growth --- thinking about the problem, designing efficient algorithms --- are features of the human mind, and not any particular language.

Saying that one language is better at scaling than another is like arguing that one human language is better for building cars than another!

Re:It's the algorithm, stupid (1)

xouumalperxe (815707) | more than 6 years ago | (#23635679)

That is almost a good counterpoint, but for one thing: What you're effectively saying is not that the language itself is poor, or slow (in the usual "bytecode is slow" sense), but rather that the data structure itself is slow or poorly implemented, or whatever. The same data structure implemented in C with the same algorithms would yield the exact same performance bottlenecks. The problem with Java and .NET and similar architectures is that you have issues separating the concept of "Language" with the concept of "Library".

i wonder what they mean by.. (4, Interesting)

trybywrench (584843) | more than 6 years ago | (#23631145)

I wonder what they mean by "elegant filesystem-based approach"? Maybe their going to treat tweets like an email and store it all in the filesystem rather than a database? There's certainly some proven extremely high volume email servers so you know that method scales.

I wonder what the disadvantages of setting up a front end to an email system and covert incoming tweets to actual an actual email is. On the retrieval side you just read the mailbox and convert back to the tweet format then send them on to the destination.

Re:i wonder what they mean by.. (1)

trybywrench (584843) | more than 6 years ago | (#23631521)

I was on hold, with CIHost no less, while posting so ignore my bad spelling and grammer :(

Re:i wonder what they mean by.. (2, Informative)

Balerion (25115) | more than 6 years ago | (#23631891)

This post [hueniverse.com] , from the author of a similar system, breaks down the pros and cons of an email-like approach to the problem.

One simple but painfully restrictive solution is to duplicate the data for each user. Basically what this means is turning the service into an email system. Each user is given a mailbox and whenever someone they are following publishes a status, it is copied into their inbox, as well as into all the other followers' inboxes. This brings the solution in line with existing systems such as webmail services where data partitioning alone can accomplish great scaling. But this comes at a price. First the data storage grows significantly faster and requires much more disk space, together with increased backup cost. Second, it makes other actions much more complex such as erasing a message once sent.

Re:i wonder what they mean by.. (3, Interesting)

PhotoGuy (189467) | more than 6 years ago | (#23633683)

I was founder of a top 100 internet site during the .COM era. We grew to serve millions of hits a day, millions of users, yadda, yadda, yadda.

We initially used a simple file based approach for user data (on vanilla Linux commodity boxes). It worked well.

As we grew, there was some pressure to move to a database approach, so we switched to Sybase (free on Linux). It worked well, and scaled us through a lot of growth.

However, eventually, when the database bogged down and no amount of tuning would help, rather than clustering, we looked at the nature of our data (a user's data was self-contained, generally not related to any other user's data), so having a massive relational database of hundreds of millions of records wasn't really necessary. So we went back to the file-based approach (with a good central "locking daemon" to ensure atomicity of writes), and gained a lot of performance (and simplicity). Even with thousands of bits of information or transactions for a user, a flat file is pretty darn manageable.

Generally people jump at databases as a default way to store data. If a single user's data is fairly manageable, and you have millions of users, there are times when plain old files suffice. (Doing some smart things like ensuring that directories don't grow arbitrarily and such also help, but that stuff is generally a lot easier than db design and maintenance.)

It sounds like Twitter didn't have well-thought out foundations, and they're reworking some of that. Good for them. (I've actually found some good consulting work in helping companies like them deal with scalability issues, from my experience with such things...)

Re:i wonder what they mean by.. (1)

Hal_Porter (817932) | more than 6 years ago | (#23635125)

I wonder what they mean by "elegant filesystem-based approach"?
Part of me thinks "what a load of pretentious bullshit" and part of me thinks "wow, some lucky bastard will have fun coding that".

Twitter Google App Engine (3, Interesting)

revscat (35618) | more than 6 years ago | (#23631191)

You know, I wonder how hard it would be to do a Twitter clone on Google's App Engine. It seems like it would be the perfect fit: relatively simple application that needs to be massively scalable.

Re:Twitter Google App Engine (3, Interesting)

Anonymous Coward | more than 6 years ago | (#23631427)

http://meow.appspot.com

Re:Twitter Google App Engine (1)

tobiasly (524456) | more than 6 years ago | (#23631493)

You know, I wonder how hard it would be to do a Twitter clone on Google's App Engine. It seems like it would be the perfect fit: relatively simple application that needs to be massively scalable.

They wouldn't have Twitter's killer feature, which is free SMS updates. That's the only reason we don't already have 100 Twitter clones.

Re:Twitter Google App Engine (0)

Anonymous Coward | more than 6 years ago | (#23632451)

uh, free sms updates aren't hard. the cell phone companies love services that send sms messages through them, as they make lots of money that way. in fact, many companies get paid a small amount for each sms they send users.

Re:Twitter Google App Engine (3, Informative)

Anonymous Coward | more than 6 years ago | (#23631527)

http://www.jaiku.com/blog/2008/04/08/wroom-were-moving-to-google-app-engine/

Re:Twitter Google App Engine (0)

Anonymous Coward | more than 6 years ago | (#23631745)

Dude, it would be easy to do a twitter clone using rock,paper,scissors technology. I'm surprised no one has cloned this thing out of significance already.

J

And the biggest irony is... (2, Funny)

JamesP (688957) | more than 6 years ago | (#23631233)

...twitter blog is hosted on blogger (Google), and this morning it was out of service.

I've never used Twitter (0)

Anonymous Coward | more than 6 years ago | (#23631263)

and I fail to see what's so special about it. I have plenty of other methods to keep in touch with those I care to keep in the know.

yuck! (0)

Anonymous Coward | more than 6 years ago | (#23633293)

.... oh, the website.

Too ittle too late. (1)

wattrlz (1162603) | more than 6 years ago | (#23631295)

...the Twitter team has been making a concerted effort to affect better communication...

From what I've heard the only affect twitter's had on communication isn't one worth bragging about.

Translation (0)

Anonymous Coward | more than 6 years ago | (#23631347)

We just hired a bunch of guys who couldn't cut it at IBM/Google/other large successful companies. We're trusting them to completely overhaul our system because they must have been so smart to have worked there.

Re:Translation (1)

Hal_Porter (817932) | more than 6 years ago | (#23635149)

It'll be OK if they hired technical people rather than (shudder) Business Method consultants. Mind you, the idea of Web 2.0 people being put on a death march to ISO 9001 certification fills me with demonic glee.

I find it odd that this article is tagged effect. (3, Funny)

bistromath007 (1253428) | more than 6 years ago | (#23631449)

It seems to be meant to suggest that the article's use of "affect" is incorrect. Surely this is mistaken. If suggesting that twitter has anything to do with better communication isn't an affectation, I don't know what is.

Re:I find it odd that this article is tagged effec (0)

Anonymous Coward | more than 6 years ago | (#23631839)

This confused me too, but I check dictionary.com.

Effect is used as a verb when there's no direct object. When used as a verb you can replace it with "bring about" and see if it makes sense. So in this case, "The Twitter team has been making a concerted effort to bring about better communication with their community at large." So, effect is the right spelling.

Using affect as a verb here, would suggest that the better communication is being acted on, which it isn't. Well... that's not the intention of the author's words anyway.

Re:I find it odd that this article is tagged effec (0)

Anonymous Coward | more than 6 years ago | (#23633131)

WHOOSH

Re:I find it odd that this article is tagged effec (0)

Anonymous Coward | more than 6 years ago | (#23633399)

Instead of dismissing the OP with "WHOOSH", why don't you take the time to educate yourself about what he means? You can even take the radical step of asking a question! All your "WHOOSH" tells me is that you believe willful ignorance is acceptable. That attitude harms not only you, but society.

Re:I find it odd that this article is tagged effec (1)

Phroggy (441) | more than 6 years ago | (#23633719)

WHOOSH!

You, also, failed to get the joke the original poster was making. Allow me to explain:

It seems to be meant to suggest that the article's use of "affect" is incorrect. Surely this is mistaken. If suggesting that twitter has anything to do with better communication isn't an affectation, I don't know what is.
This is a joke. The person who wrote it is well aware of the difference between "affect" and "effect". He deliberately used the word "affectation" (which looks related to "affect" but really means conspicuously artificial [reference.com] or unnatural speech or conduct [merriam-webster.com] ). The WHOOSH was not a dismissal of the response; the WHOOSH was intended to be the sound of the joke passing over the respondent's head.

Whether the joke is funny or not is another matter entirely.

Re:I find it odd that this article is tagged effec (1)

QuoteMstr (55051) | more than 6 years ago | (#23633369)

Effect is used as a verb when there's no direct object. When used as a verb you can replace it with "bring about" and see if it makes sense. So in this case, "The Twitter team has been making a concerted effort to bring about better communication with their community at large." So, effect is the right spelling.


You mean with a direct object, right? When saying "Bob effects X", X is the direct object.

Lore Sjöberg on Twitter (1)

Scareduck (177470) | more than 6 years ago | (#23631537)

Fargin' hilarious [youtube.com] . Twitter seems to me to be absolutely February 2001, just milliseconds before the crash stupid.

ergh Twitter (0, Flamebait)

aztektum (170569) | more than 6 years ago | (#23631555)

Talk about sites the US Gov should shut down. Facebook and MySpace are up there too.

Plurk seems pretty stable so far (4, Interesting)

the JoshMeister (742476) | more than 6 years ago | (#23631641)

Plurk [is.gd] has been gaining popularity in the past 24 hours, and it's handling scalability rather well so far (after having been mentioned by Leo Laporte, Robert Scoble, TechCrunch, and others). I'm very curious to see how well it would hold up if it had the same number of users as Twitter, though.

Re:Plurk seems pretty stable so far (2, Insightful)

Bill, Shooter of Bul (629286) | more than 6 years ago | (#23632549)

That is interesting. Sounds like were still running on in house development level servers when the load hit -- ouch. If I do end up using such a site, I d rather say I'm Plurking than tweeting. It sounds much cooler, like something to do after having too many beers, rather than trying to imitate a bird.

Nice try... (3, Funny)

wigginz (730819) | more than 6 years ago | (#23631847)

but who would quit Google to work for Twitter???

working for Twitter (0)

Anonymous Coward | more than 6 years ago | (#23632379)

but who would quit Google to work for Twitter???
How about Erris and Gnutoo?

Re:Nice try... (1)

Phroggy (441) | more than 6 years ago | (#23633729)

Not all former Google employees quit voluntarily.

poop (0, Troll)

jaguth (1067484) | more than 6 years ago | (#23632447)

Poop is coming out now...

Re:poop (1)

legallyillegal (889865) | more than 6 years ago | (#23635681)

apparently the mod and/or mods (who may or may not be on crack) didn't get the reference [penny-arcade.com]

grammar nazi (1)

zuperduperman (1206922) | more than 6 years ago | (#23632745)

Why would they try to "affect" better communication when they could actually put it into effect instead?

Re:grammar nazi (1)

Hognoxious (631665) | more than 6 years ago | (#23636045)

The triumph of style over substance is a curse of our times. They're probably Apple users.

"Our biggest mistake... (0)

netcrusher88 (743318) | more than 6 years ago | (#23634747)

...was using Ruby on Rails"

I doubt they'll come out and say that. But look at how flaky twitter is (try using it for a while). The two biggest sites on the internet that are built on RoR - Penny Arcade and twitter - are flaky as hell. I've met the PA coder, and I'm not willing to believe the twitter guys are incompetent, so it's obviously not their fault.

RoR is known for obscenely high resource usage. I can't believe it'd be a good choice for large sites like twitter, long term.

Re:"Our biggest mistake... (0)

rogerbly (1080765) | more than 6 years ago | (#23635371)

agreed. Rails is awesome for prototypes and small apps, but you need to move to java stack at some point. I think twitter did it right... starting with Rails, but now it's time to do what they are doing... bring in the pros and refactor.

rails to java? (0)

Anonymous Coward | more than 6 years ago | (#23635301)

bottom line... move from Rails to Java and continue to work on real-time social networking... all will be well.

'team' (2, Informative)

spazdor (902907) | more than 6 years ago | (#23635525)

While it may not be rocket science, the Twitter team has been making...

Psst, there's actually no "Twitter team." It's just one guy with like ten accounts.

Re:'team' (0)

Anonymous Coward | more than 6 years ago | (#23637351)

WOOSH... to whoever modded this "informative".
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?