Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Handling the Loads

CmdrTaco posted more than 13 years ago | from the when-it-all-hits-the-fan dept.

Slashdot.org 890

On Tuesday, something terrible happened. The effects rippled through the world. And Slashdot was hit with more traffic than ever before as people grabbed at any open line of communication. When many news sites collapsed under the load, we managed to keep stumbling along. Countless people have asked me questions about how Slashdot handled the gigantic load spike. I'm going to try to answer a few of these questions now. Keep reading if you're interested.

I woke up and it seemed like a normal day. Around 8:30 I got to the office and made a pot of coffee. I hopped on IRC, started rummaging through the submissions bin, and of course, began reading my mail. Within minutes someone told me on IRC what had happened just moments after the impact of the first plane. Just a minute or 2 later, submissions started streaming into the bin. And at 9:12 a.m. Eastern Time, I made the decision to cancel Slashdot's normal daily coverage of "News for Nerds, Stuff that Matters," and instead focus on something more important then anything we had ever covered.

I couldn't get to CNN, and MSBNC loaded only enough to show me my first picture of the tragedy. I posted whatever facts we had: these were coming from random links over the net, and from Howard Stern who syndicates live from NY, even to my town. Over the next hour I updated the story as events happened. I updated when the towers collapsed. And the number of comments exploded as readers expressed their outrage, sadness, and confusion following the tragedy.

Not surprisingly, the load on Slashdot began to swell dramatically. Normally at 9:30 a.m., Slashdot is serving 18-20 pages a second. By 10 we were up to 30 and spiking to 40. This is when we started having problems.

At this point Jamie and Pudge were online and we started trying to sort out what we could do. The database crashed and Jamie went into action bringing it back up. I called Krow: he's on Western time, but he knows the DB best, and I had to wake him up. But worst of all, I had to tell him what had happened in New York. It was one of the strangest things I've ever done: it still hadn't settled in. I had seen a few grainy photos but I don't have a TV in my office and hadn't yet seen any of the footage. After I hung up the phone I almost broke down. It was the first time, but not the last.

The DB problem was a known bug and the decision was made to switch to the backup box. This machine was a replicated mirror of Slashdot, but running a newer version of MySQL. We hadn't switched the live box simply because it meant taking the site down for a few minutes. Well we were down anyway, and the box was a complete replica of the live DB, so we quickly moved.

At this point the DB stopped being a bottleneck, and we started to notice new rate limits on the performance of the 6 web servers themselves. Recently we fixed a glitch with Apache::SizeLimit: Functionally, it kills httpd processes that use more then a certain amount of memory, but the size limit was to low and processes were dying after serving just a few requests. This was complicated by the fact that the first story quickly swelled to more than a thousand comments ... we've tuned our caching to Slashdot's normal traffic: 5000-6000 comments a day, with stories having 200-500 comments. And this was definitely not the normal story. Our cache simply wasn't ready to handle this.

Our httpd processes cache a lot of data: this reduces hits to the database and just generally makes everything better. We turned down the number of httpd processes (From 60 on each machine, to 40) and increased the RAM that each process could use up (From 30 to 40 and later 45 megs) We also turned off reverse hostname lookups which we use for geotargetting ads: The time required to do the rdns is fine under normal load, but under huge loads we need that extra second to keep up with the primary job: spitting out pages as fast as possible.

This was around noon or so. I was keeping a close eye on the DB and we noticed a few queries that were taking a little too long. Jamie went in and switched our search from our own internal search, to hitting Google: Search is a somewhat expensive call on our end right now, and this was necessary just to make sure that we could keep up. We were serving 40-50 pages/second ... twice our usual peak loads of around "Just" 25 pages a second. I drove the 10 minutes to get home so I could watch CNN and keep up better with what was happening.

We trimmed a few minor functions out temporarily just to reduce the number of updates going to frequently read tables. But it was just not enough: The database was now beginning to be overworked and page views were slowing down. The homepage was full of discussions that were 3-4x the average size. The solution was to drop a few boxes from generating dynamic pages to serving static ones.

Let me explain: most people (around 60-70%) view the same content. They read the homepage and the 15 or so stories on the homepage. And they never mess with thresholds and filters and logins. In fact, when we have technical problems, we serve static pages. They don't require any database load, and the apache processes use very little memory. So for the next few hours, we ran with 4 of our boxes serving dynamic pages, and 2 serving static. This meant that 60-70% of people would never notice, and the others would only be affected when they tried to save something ... and then they would only notice if they hit a static box, which would happen only one in 3 times. It's not the ideal solution, but at this point we were serving 60-70 pages a second: 3x our usual traffic, and twice what we designed the system for. We got a lot of good data and found a lot of bottlenecks, so next time something that causes our traffic to triple, we'll be much more prepared.

At the end of the day we had served nearly 3 million pages -- almost twice our previous record of 1.6M, and far more then our daily average of 1.4M. During the peak hours, average page serving time slowed by just 2 seconds per page ... and over 8000 comments were posted in about 12 hours, and 15,000 in 48 hours.

On Wed. we started to put additional web servers into the pool, but that ended up not being necessary. We stayed dynamic and had no real problems on all 6 boxes all day. We peaked at around 35-40 pages/second. We served about 2 million pages. Thursday traffic loads were high, but relatively normal.

Summary So here is what we learned from the experience.

  • We have great readers. I had only one single flame emailed to me in 24 hours, and countless notes of thanks and appreciation. We were all frazzled over here and your words of encouragement meant so much. You'll never know.
  • Slashteam kicks butt. Jamie, Pudge, Krow, Yazz, Cliff, Michael, Jamie, Timothy, CowboyNeal, you guys all rocked. From collecting links to monitoring servers, to fixing bits of code in real time. It was good seeing the team function together so well ... I can't begin to describe the strangess of seeing 2 seperate discussions in our channel: one about keeping servers working, and another about bombs, terrorists, and war. But through it all these guys each did their part.
  • Slash is getting really excellent. With tweaks that we learned from this, I think that our setup will soon be able to handle a quarter million pages an hour. In other words, it should handle 3x Slashdot's usual load, without any additional hardware. And with a more monstrous database, who knows how far it could scale.
  • Watch out for Apache::SizeLimit if you are doing Caching.
  • Writing and reading to the same innodb MySQL tables can be done since it does row-level locking. But as load increases, it can start being less then desirable.
  • A layer of proxy is desirable so we could send static requests to a box tuned for static pages. For a long time now we've known that this was important, but its a tricky task. But it is super necessary for us to increase the size of caches in order to ease DB load and speed up page generation time ... but along with that we need to make sure that pages that don't use those caches don't hog precious apache forks that have them. Currently only images are served seperately, but anonymous homepages, xml, rdf, and many other pages could easily be handled by a stripped down process.

What happened on Tuesday was a terrible tragedy. I'm not a very emotional person but I still keep getting choked up when I see some new heart breaking photo, or a new camera angle, learn some new bit of heart breaking information, or read about something wonderful that somebody has done. This whole thing has shook me like nothing I can remember. But I'm proud of everyone involved with Slashdot for working together to keep a line of communication open for a lot of people during a crisis. I'm not kidding myself by thinking that what we did is as important as participating in the rescue effort, but I think our contribution was still important. And thanks to the countless readers who have written me over the last few days to thank us for providing them with what, for many, was their only source of news during this whole thing. And thanks to the whole team who made it happen. I'm proud of all of you.

cancel ×

890 comments

Sorry! There are no comments related to the filter you selected.

fp (-1, Offtopic)

Anonymous Coward | more than 13 years ago | (#2299052)

2789452nd post!

I am going to pull down my pants... (-1, Flamebait)

Anonymous Coward | more than 13 years ago | (#2299056)

and you can handle MY load...

suck my ass (-1, Flamebait)

Anonymous Coward | more than 13 years ago | (#2299058)

bitch

Something terrible happened (-1)

Coma of Souls (201241) | more than 13 years ago | (#2299065)

Slashdot was overloaded. Oh my god.

A request (4, Insightful)

rosewood (99925) | more than 13 years ago | (#2299068)

I know that a lot of shackers and other people on the net aren't christian or don't even beleive in God. Thats fine. Tomorrow (now today) you will hear a lot of people praying, asking you to pray, etc. This isn't the snickers comercial where they bring in a representative of every religion before the big game. It will feel weird. I feel that a week ago that if NBC was showing a service that someone would whine. Today, I ask ya just let it slide. When they say pray, interperate that as 'do what makes you feel comfortable. Please just be respectful like your mama would want you to be. But for today, just kinda chalk it up to all those people burned, crushed, flateneted, chocked, suffocated, etc. to death.

Thank you

Re:A request (0)

Anonymous Coward | more than 13 years ago | (#2299111)

(doh..forgot my password and account info and being i'm at work and not home..well..you get the drift...)

i agree. :)

BTW, great work Slash crew!

~Valien

Re:A request (1)

rosewood (99925) | more than 13 years ago | (#2299121)

Yes - this was a bit OT and I suspect it will get moded as such - but I think it needed to be said

gg to the slash crew that was able to focus on something like bandwidth when I felt like the world stopped

Re:A request (-1, Offtopic)

Anonymous Coward | more than 13 years ago | (#2299128)

I respect them by dancing on the street.

Re:A request (1)

rosewood (99925) | more than 13 years ago | (#2299136)

Come back to this when you lose someone dear to you

Re:A request (2, Insightful)

Nicolas MONNET (4727) | more than 13 years ago | (#2299240)

"Come back to this when you lose someone dear to you"

If it happened, odds are it would be due to some religious fanatics.

Well Said (5, Insightful)

Wyatt Earp (1029) | more than 13 years ago | (#2299144)

That was really well said.

During my life I've always taken "bow your head and pray" as "shut up and look serious".

And I thought Slashdot did a very good job this week. I woulda emailed Taco that, but I figured there was enough traffic over the Internet.

Really good job guys. Between Slashdot and Drudge I felt as informed as a guy can be.

Anne Tomlinson (-1)

trollercoaster (250101) | more than 13 years ago | (#2299071)

good thing you lost that bitch.

Welcome back Trolls! (-1, Offtopic)

Anonymous Coward | more than 13 years ago | (#2299236)

Glad to see slashdot is back to normal :-)

I've been impressed overall (3, Insightful)

WinDoze (52234) | more than 13 years ago | (#2299073)

Not only with Slashdot (did that REALLY say 2-thousand-something comments on the front page?!?!), but with CNN, ABCNews, the NY Times, and just about every other major news source I can think of. Tuesday afternoon was tough. By Tuesday evening all these sites were responding as though I was the only connected user. The server power that must have been thrown at some of these sites is staggering.

Re:I've been impressed overall (1)

part!cle (473500) | more than 13 years ago | (#2299157)

/. was great. Expecially with the people keeping it online, but all of the intelligent conversations that transpired in the last few days as well. ./, along with abc and bbc for non techie stuff are now where I will get my news.

Re:I've been impressed overall (1)

kootch (81702) | more than 13 years ago | (#2299261)

don't forget the BBC! While other people were trying to get CNN to load and the coverage on the tv stations was still iffy, my entire office was watching a BBC streaming video (via Real Player) that was incredible.

CNN you couldn't get a page view, but the BBC you could watch a live video stream.

THAT is incredible. My only question for the BBC... how did you guys manage to find brits to interview in the area? Everyone they managed to interview while live in NYC had a british accent... or was it dubbed over?

ALLAH WAKBAR! (-1, Offtopic)

Anonymous Coward | more than 13 years ago | (#2299076)

ALLAH WAKBAR!!!

All Praises be to ALLAH!

Couldn't get to CNN (0, Funny)

Anonymous Coward | more than 13 years ago | (#2299080)

Have you heard of that wonderful new invention, television?

Kudos to Slashdot and the Slashteam (5, Insightful)

Whyte Wolf (149388) | more than 13 years ago | (#2299089)

I've spent the last few days in something of a daze, waiting for the real ramifications of Tuesdays horror to sink in. Many of my collegues up here in Canada are not sure what to make of the events, and possible response, but we're sure it will be bad.

That said, in all my experiences on the net over the last couple of days, it was Slashdot I came back to for my info feed/dump. Who had their site up and running in the face of the massive demand? Slashdot.

CNN was there during the Gulf War. Slashdot was there for the start of this new era, and I'm sure will be there in the face of whatever is to come. You guys are just another indication of the strength the US can have in the face of adversisty.

Thank you.

Enemies of the USA (0, Informative)

CmdrTaco on (468152) | more than 13 years ago | (#2299091)

The following list is of slashdot users who are terrorist sympathisers and enemies of freedom and democracy:

greenrd
abdousi
IntlHarvester
Angry White Guy
delmoi
Cederic

You make it sound like a bad thing (-1, Flamebait)

Anonymous Coward | more than 13 years ago | (#2299112)

Democracy sucks.

Re:Enemies of the USA (0, Offtopic)

IntlHarvester (11985) | more than 13 years ago | (#2299130)

I knew I shouldn't have posted something about Lotus Notes!

Lotus notes! Symbol of freedom everywhere! (-1)

CmdrTaco on (468152) | more than 13 years ago | (#2299148)

Damn straight!

The Community Was Served. (5, Interesting)

pgrote (68235) | more than 13 years ago | (#2299093)

Slashdot did provide a very valuable service the day of the attack.

Take into consideration that during the day at some point all major media web sites died.

Many people found Slashdot as their only source of updated information that was staying up.

This sentiment was echoed in pieces by Salon and Wired writers that mentioned Slashdot specifically as a site that had what people were looking for.

You should be proud and satisfied that what you have created did provide a needed service. Thanks, again.

Re:The Community Was Served. (1)

brain damage (222855) | more than 13 years ago | (#2299229)

I agree entirely.

I could watch TV and see what was happening, but after a few repetitions of the same horrifying videos of the crash and subsequent collapses I simply wanted to know what other people had to say.

It comforted me to know that there was a world out there just as outraged as me. Just as dismayed with humanity.

The stories of /. members who experienced the events first hand really brought everything home. It also allowed for a moment of thanks that some people were able to survive.

Thank you for struggling to keep this incredible resource up and running.

Congratulations. (1)

chill (34294) | more than 13 years ago | (#2299095)

and thanks for the information. If I couldn't load CNN due to traffic, I was able to get through to /. and at least get updates.

did you forward this ... (3, Interesting)

Rev.LoveJoy (136856) | more than 13 years ago | (#2299097)

to CNN or MSNBC?

:-)

This is a great writeup. It covers all the things you could have done on your end to make /. fly. I guess the only prerequisite that most of us have trouble with are the Phat Pipes you folks can afford.

Cheers,
- RLJ

Very interesting stuff. (2, Insightful)

XretsiM (459192) | more than 13 years ago | (#2299098)

As a part owner of an internet developer/consultant, one of the more interesting things about Tuesday's tragedy was watching how various sites responded to the incredible load demands placed on them. Even watching the situation from the outside, it was clear that clear heads at Slashdot were doing something remarkable behind the scenes. Thanks for the insite into what was actually going on. I'll be passing this on to our staff, many of who came to rely on Slashdot's coverage on Tuesday morning.

thank you very much. (4, Insightful)

garcia (6573) | more than 13 years ago | (#2299100)

It is not common for people to recieve thanks for the great service that they do for a community but I am going to go ahead and give you thanks for feeding us the information that I was not able to get through TV and the basically non-exitant other news-sites.

I am normally a critic of /. and the editors but this entire week I felt that they did an extraordinary job of keeping us informed. For once I am going to applaud you.

I got links to personal experiences on the tragedies, movies, images not seen on TV, and personal reflection on the entire ordeal by people that seem to have valid ideas (not the crap that you hear from most people about the attacks)

Thank you again /. for making sure we got the news we needed.

other sites... (3, Interesting)

swright (202401) | more than 13 years ago | (#2299101)

Part of my job is monitoring various web sites. They aren't news related and the average traffic levels fell by around 60% - rising to 50% under average after a day or so. They're only just returning to normal.

Thought y'all might be interested (the sites are generally eCommerce sites in Europe)

Re:other sites... (2)

Roblimo (357) | more than 13 years ago | (#2299169)

This seems to have been a Net-wide pattern; news sites got hard-hit, but all other kinds saw big traffic drop-offs.

There will be a story up at Online Journalism Review [ojr.org] before long talking about that -- and about how Slashdot served as an ad hoc news portal in a way no traditional news site could.

- Robin

Staying focused was tough (4, Redundant)

sphealey (2855) | more than 13 years ago | (#2299104)

I also congratulate you guys for staying focused on getting your jobs done under very difficult circumstances. I would estimate my own productivity was 25% of normal that day, along with most people I was working with.

sPh

My only source (2)

nate1138 (325593) | more than 13 years ago | (#2299106)

At least for me, slashdot was the ONLY news source I had, no TV or radio in my office, and all of the usual suspects collapsed under the load. Thanks also to google for providing Cached pages from CNN, Wash Post, MSNBC, Etc. a little later in the day. And thanks to all the /.'ers that had personal reports from the area, that really helped to put a perspective on things.

Great Job, Cmdr Taco (2, Informative)

Petrus (17053) | more than 13 years ago | (#2299107)

I was getting all my early informatins and initial links to working news sites from slashdot. Everybody in the office was surprised, where do I get working connection, since they could not get through any major news channels.

Regards,

Petrus

Hopefully a dumb question... (2)

cperciva (102828) | more than 13 years ago | (#2299108)

But I'm still stuck on it. Why is /. running a per-Apache-process cache? Doesn't that mean that it would be keeping 50 copies of the same data in memory on each machine? I would have thought that having a single-process cache at the front-end (something like SQUID) which holds on to a much larger cache and then passes requests to non-caching Apache processes would have done much better.

CNN and MSNBC take note. (2, Funny)

AltGrendel (175092) | more than 13 years ago | (#2299116)

This is how the REAL pros do it.

Slashteam, we salute you.

Re:CNN and MSNBC take note. (0)

Anonymous Coward | more than 13 years ago | (#2299177)

Well, not to put too fine a point on it, but if slashdot got hit with the same traffit that regular news sites did, I suspect it would not have handled it. They have an order of magnitude more visitors than /.

Way to go Slashdot (2, Insightful)

Sentry21 (8183) | more than 13 years ago | (#2299119)

When I first heard about this, after being woken up to it, I checked CNN's homepage - which was down. I checked several other news sites, and the only working one was CTV News. Then I thought to check slashdot - lo and behold, it was the only other site I could get to. I posted in one of the discussions that ctvnews.ca was working, and by the time I had hit 'submit', it wasn't.

Kudos to the Slashdot team for having the only satisfactorally working news service on the net. Combined with the people that made their own websites and posted their own pictures, and the people that mirrored news reports they COULD get to, it was an amazing triumph of technology. It's just too bad that this great moment in Slashdot history had to come at such a horrid moment in world history.

--Dan

A request for future (5, Insightful)

tzanger (1575) | more than 13 years ago | (#2299122)

Could you please cache the stories in NESTED mode instead of threaded? When the site is being hammered I would imagine it is far better to have guys grab a single large, cached page than a smaller cached page and then have to try to have teh system survive thosands of clicks for more information.

I really do thank you guys for this site and your decision to carry the news. I have a new respect for the amount of bandwidth you throw around with impunity on a daily basis.

one final request: get search back online so I can get to the old stories! Google doesn't have them (even now!)

Re:A request for future (1)

sulli (195030) | more than 13 years ago | (#2299178)

Not sure this would make sense as many readers just read the threaded stories and don't click thru...

So why then is Slashdot always down ? (1, Flamebait)

tmark (230091) | more than 13 years ago | (#2299124)

So Slashdot held up well on Tuesday. I'm not really sure what that means since the hits you cited are small potatos compared to a news outlet like CNN or MSNBC. My question is, if the Slashcode is so good and your team rocks as you say, why then is your site so often down ? Today, especially ? Why are there so many problems where you click on a follow up article and get bounced to the front page ? Why are there so many follow-up articles that clearly appear underneath the *wrong* articles ? If the Slashteam and Slashcode are so great under fire, why are they so mediocre from day-to-day ?

Why was this moderated down? (0)

Anonymous Coward | more than 13 years ago | (#2299160)

It's a very valid remark. Inbetween cheering themselves, the Slashdot guys should do something to make Slashdot more stable on normal days.

Re:So why then is Slashdot always down ? (1)

sgups (449689) | more than 13 years ago | (#2299168)

You are right on. For some reasond espite constant attempts to login I keep getting the static page. I.E no comment whis reads 'This page was generated by so and so for sgups'. However at the rime of writing this reply, I do not see this line, but I see I am not posting as anonymus. SO I guess something is working.

Re:So why then is Slashdot always down ? (1)

part!cle (473500) | more than 13 years ago | (#2299176)

oh shut up.

MODERATION: I don't think this is flamebait (2)

MarkusQ (450076) | more than 13 years ago | (#2299188)

To whoever modded this as flamebait:

While I don't claim to support all his points, the post is on topic he has a right to ask these sorts of questions.

-- MarkusQ

Re:So why then is Slashdot always down ? (0)

Anonymous Coward | more than 13 years ago | (#2299222)

You've confused Slashdot with Stileproject Forums. And you've confused Slashdot readers with people caring about your tripe.

Good job. (2)

standards (461431) | more than 13 years ago | (#2299127)

Good job keeping things up.

I found Slashdot, BBC, and Boston.com to be the most available sites. ABCNews and CNN and Foxnews, etc, were all pretty much overwhelmed and unusable.

Fairly quickly, CNN went to a simple static page with 1 image, and that helped them out quite a bit.

Funny where the news comes from (3, Funny)

ellem (147712) | more than 13 years ago | (#2299129)

I got a lot of my news from Howard Stern and from you guys when I locked my office door and shut the windows.

/. and Howard Stern had the best coverage -- I think that is just weird.

But true.

Re:Funny where the news comes from (2)

zpengo (99887) | more than 13 years ago | (#2299207)

I have to agree. When the big news sources (CNN et al.) got smashed with traffic, an amazing amount of responsibility came down to the amateur journalists [slant-six.org] of the world, who did a wonderful job of reporting what they saw on their own websites.

Hats off to the webloggers and other people whose generally mundane daily updates suddenly became a source for an information-starved world.

Re:Funny where the news comes from (0)

Anonymous Coward | more than 13 years ago | (#2299211)

Howard Stern has proven that he is a fuckwad and a racist. His "towelhead" jokes were in bad taste, even for him, considering the millions of innocent americans who are now being attacked for the color of their skin and the fashion of their dress. Despite his history of doing stuff simply to push buttons, he has gone too far this time and should be truly ashamed of himself. I used to respect the guy for being politically incorrect, but now I see the truth behind it, and he is worse than any of his religious adversaries have made him out to be.

Re:Funny where the news comes from (3, Funny)

Skyshadow (508) | more than 13 years ago | (#2299246)

This reminds me of John Stewart's reaction on hearing that many people consider the Daily Show thier prime news source:

"Don't do that!"

Stephen King, author, dead at 54 (-1, Troll)

Anonymous Coward | more than 13 years ago | (#2299131)


I just heard some sad news on talk radio - Horror/Sci Fi writer Stephen King was found dead in his Maine home this morning. There weren't any more details. I'm sure everyone in the Slashdot community will miss him - even if you didn't enjoy his work, there's no denying his contributions to popular culture. Truly an American icon.

For my own personal edification. (1)

thesolo (131008) | more than 13 years ago | (#2299133)

Thanks for explaining what happened with the site on Tuesday. And much more importantly, thanks for staying up & running!

I didn't have a TV or radio here at work, and I live 60+ miles away, so I couldn't just drive home to watch the coverage. Slashdot was the only site that kept me informed of what was going on. My coworkers & I really appreciated having our favorite site covering these events.

Thanks again!

Thank you (4, Insightful)

sulli (195030) | more than 13 years ago | (#2299137)

I posted this elsewhere, [slashdot.org] but I'll say it again: this has been Slashdot's finest hour. The eyewitness accounts and individual stories have been so meaningful, and the readers have been great - almost zero harassment and trolling (a bit more in the last day) and very honest, heartfelt comments. Also great were the mirrors in the first day that many participants posted, to handle the excess load for the news sites; the many Red Cross donation links; the updates and corrections of the news; and more.

Slashdot itself did very, very well in my experience. I experienced far fewer delays and errors than on other sites. Thanks to everyone who worked so hard to keep it running. You've made a huge difference for thousands of people.

You guys ruled!! (1)

brindle (8241) | more than 13 years ago | (#2299145)

I was very impressed with the information available from your site. However I was not surprised, you guys have always done a great job with this site.

Its amazing what you are able to do with apache/perl and whatever else you are using.

Again, congrats on a job well done.

-JL

To paraphrase a theme.... (1)

DA_MAN_DA_MYTH (182037) | more than 13 years ago | (#2299146)

that I've been hearing over and over again

For what does not break us only makes us stronger...

Great job Slashdot, CNN couldn't handle their load and had to change the format of their site.... Maybe they should switch to Slash :)

enlighten me... (1)

__soup_dragon__ (112372) | more than 13 years ago | (#2299150)

first congrats to you all for the hard work, i wasnt hable to read slash at that time but i'm sure a lot of people relied on it as the only source of info. my question is, was the load you were under comparable to the load cnn was under? if yes... wow... :-D

Big thanks to Slash Team (2)

Alien54 (180860) | more than 13 years ago | (#2299154)

You guys have down a marvelous job.

Although, being on dial up, getting into any site was a struggle [smile]

as a secondary note, I have seen a few random reports of senseless actions. I trust that the Slash Crowd is wise and intelligent and educated enough to avoid this.

One should never accept the invitation to hate, especially in conditions like these. it becomes a slippery slope.

We all have exceptions that we make, for our favorite pet peeves and political causes. Even so, This is a big step to making things right. This does not mean that we do not take action to save ourselves and our friends. People may appoint us as their enemies, their opponents, even as their executioners. We should hate them for their lack of good sense, or for their own hatred.

- - -
Radio Free Nation [radiofreenation.com]
an alternate news site using Slash Code
"If You have a Story, We have a Soap Box"
- - -

Re:Big thanks to Slash Team (2)

Alien54 (180860) | more than 13 years ago | (#2299171)

We should hate them for their lack of good sense, or for their own hatred.

DUH!

should read:

We should NOT hate them for their lack of good sense, or for their own hatred.

CNN's problems (5, Interesting)

crow (16139) | more than 13 years ago | (#2299155)

CNN's main problem was that they had canceled their contract with Akamai a month or two ago to save money. Akamai works by having servers at or near most major ISPs so that the majority of traffic is served locally.

While the load was heavy, it wasn't anything Akamai wasn't prepared to handle.

Unfortunately, Akamai's co-founder was one of the passengers flying out of Boston on a hijacked flight Tuesday. I have friends who work at Akamai for whom he was not just a boss, but a friend.

Bravo (1)

kTag (24819) | more than 13 years ago | (#2299156)

For everything you did and everything you said.

Some Good News (5, Informative)

bahtama (252146) | more than 13 years ago | (#2299161)

A reminder and fyi, the current totals at Amazon.com [amazon.com] [amazon.com] are:

Total Collected: $4,528,374.96
# of Payments: 124408

I think that is truly amazing and by the time you go there it will be even more. I donated my $100, did you? Even 10 dollars could help buy all these guys [time.com] [time.com] a cup of coffee, what's a couple bucks compared to the cause.

This is pathetic (-1, Flamebait)

Anonymous Coward | more than 13 years ago | (#2299192)

The military just received another $20 billion, and the Red Cross has to beg for donations. Something's wrong here.

Thanks (2)

arete (170676) | more than 13 years ago | (#2299164)

I didn't email you a thank you - partially 'cause I figured your box would be fairly full. But thank you. I do appreciate you keeping me informed better than anyone else was doing on the web.

Wonderful.

Primary source (2, Interesting)

Anixamander (448308) | more than 13 years ago | (#2299166)

It was interesting to see Slashdot move from a secondary source to a primary source. Throughout the day, I would check it for updates that folks had posted, and to all those individuals who constantly posted working links. I spoke with my wife several times throughout the day, and as she was only familiar with the standard sources (CNN, MSNBC, etc) I was able to give her URL's that worked. While those kept changing throughout the day, Slashdot remained available and useful.

Kudos to the slashdot team for their tireless efforts here...while work came to a halt everywhere, you guys managed to troubleshoot problems that would have given ordinary people fits on an average day. I am amazed at how quickly you adapted and improved, even though you no doubt would have preferred just to watch TV in saddened silence like the rest of us.

Instant news from the net... (1)

bradasch (516015) | more than 13 years ago | (#2299184)

I'm very impressed with slashdot: I live in Brazil, and I got word of the attacks as soon as you posted the first news: ten minutes later a friend called and told me what I already knew.
Now, think about it: I'm very far from NY (about 10-12 hours on a plane) and was reading news about it almost instantly. Most news sites on Brazil got overloaded too, so slashdot, for a while, was my only source of info...
Congratulations to the slashdot team... you did a very cool job for the whole world!

Thank you for giving us the news (1)

Peter Simpson (112887) | more than 13 years ago | (#2299187)

I couldn't read CNN, I couldn't read Boston Globe,
I couldn't read BBC.
Slashdot kept me informed (so did Sydney Morning
Herald site). I couldn't believe what I was
reading, but at least I could read it.

If there's one thing that sickens me more than the
terrible events of Tuesday themselves, it's the
anti-Muslim attacks. Granted there have been only a few
of these, but there should be NONE! Anyone tempted
to lash out in this way should think about why
they feel that they deserve to call themselves an
American. Then they should ask themselves how
it must feel to be an Arab-American.

Peace,
Peter

No need to reverselookup hostnames for geotarget (5, Informative)

bodin (2097) | more than 13 years ago | (#2299189)


There's no need to reverselookup just to be able to geotarget ads. Build up a reverse-database, and you are all set.


See http://www.ipindex.net/ [ipindex.net] for an updated index.


You just need country or so location anyway, right? I mean there are a lot of .com-domains in europe now, and that's when reverse-lookups does WRONG instead of looking at where the actual nets are allocated.

Good job to /., but forgive CNN and MSNBC (5, Insightful)

Jayde Stargunner (207280) | more than 13 years ago | (#2299193)

First off kudos to Slashteam. You kept a valuable news source up and running while most people were too stunned to do anything other than watch, horrified, at the TV. Good work. You provided a valuable service to many people in this crisis.

Also, to those who are getting down on CNN and MSNBC... From what I've heard, those sites are already tuned--and regularly do--serve around 45 pages per second...even with loads of media.

Crashing them was likely no small feat, either. Likely every person with internet typed in the very familiar cnn.com or msnbc.com just on instinct. It probably didn't help MSNBC or CNN that the MSN and AOL/Netscape portals, respectivly, link to them directly.

I was actually pretty impressed with how they handled the load...it was a little slower than /.'s recovery, but it was rather impressive given the HUGE load they were experiencing. First, they stripped down the page content to low-bandwidth versions, then phased in their site. I'm not sure about CNN, but MSNBC added static mirrors to their pool, and got Akamai servers to serve all their media. By around noon, both sites were running their normal full-content versions, even though they were probably still getting hammered to high-heaven.

Personally, I give many thanks to all the techs for all the news sites who worked like mad to ensure that people were able to understand what was happening. It must not have been easy to work in conditions like that: especially considering the stress that was put on them.

-Jayde

Other sites went to stripped down initial pages (2, Informative)

tcyun (80828) | more than 13 years ago | (#2299194)

One of the things that I noticed were that many of the major sites reduced the content size on their home pages to the smallest size possible. I know that the NYT, MS-NBC and other sites removed most all of their images and went to fairly small home pages with a few lines of text.

I think it is a credit to Slashcode, the Slash coders and great up-front planning that Slashdot was able to handle the load as well as it did. I know that Slashdot was one of the few sites where I could get a collection of information when many of the other sites were down.

Kudos to all of you.

thanks /. (2)

Dr. Awktagon (233360) | more than 13 years ago | (#2299197)

Usually when something big happens, I instantly turn to the net and usually slashdot for the news links and especially for the reader comments, which usually give the best picture of whatever happened. I'm glad you guys were able to stay up.

I must admit though, the TV coverage, especially MSNBC, was excellent during the first day. Usually I avoid it.

After a while though, the ratings grabbing kicked in and they added the graphics and the special music and the "let's get them to cry on camera" bits and I remembered why I don't usually watch TV news, and have come back to the 'net and slashdot.

Anyway thanks Slashdot!

Thank you /. (1)

sho-gun (2440) | more than 13 years ago | (#2299198)

Thank you. Your efforts and talents are GREATLY appreciated.

Surprising lack of contingency plans (1)

warpSpeed (67927) | more than 13 years ago | (#2299199)

I was somewhat surprised that I could not get to the main news sites during the aftermath (with the exception of www.WashingtonPost.com). You would think that CNN would have figured out that they will get hit first and furious during a crisis.

I'll bet that there will be quite a bit of retooling various web sites to handle this type of load soon. LEts just hope that this type of load is not seen for a long time.

~Sean

Great Job! (1)

lavaboy (21282) | more than 13 years ago | (#2299200)

Thanks for keeping the site up, and making the decision to focus on the Tuesday's only important news. I was also at work, unable to get to a TV or radio, and couldn't get to any of the more traditional news sites. Thanks to your work, and the constant influx of working links posted by other users, I was able to follow the situation as it happened.

Way to go guys!

Slash team kicks butt... (2)

srvivn21 (410280) | more than 13 years ago | (#2299203)


* Slashteam kicks butt. Jamie, Pudge, Krow, Yazz, Cliff, Michael, Jamie, Timothy, CowboyNeal, you guys all rocked.

And Jamie rocks twice as much. ;o)

My thanks to the lot of you. The Slashteam, and the /. community.

Thank you (0)

Anonymous Coward | more than 13 years ago | (#2299204)

thank you for providing such a great source of information. I too, when unable to reach nyt.com or cnn.com, headed here.

thank you.

DNS & mod_gzip (4, Informative)

drwho (4190) | more than 13 years ago | (#2299206)

Everyone knows that you should turn off hostname lookups. I was wondering why slashdot would often be some damned slow first thing in the morning -- well there's why. Because the PTR record had expired overnight. Another way we suffer for advertisers. Oh well.

static content can be stored and transmitted in gzip format, to be uncompressed by the browser (all modern browsers support this). HTML coompressed very well -- pages here end up averaging 28% of their original size! This not only saves slashdot bandwidth, but saves it for the end user as well. Some people out there are still using crufty old 28.8 modems, and need every bit of help they can get. Anyhow, do a search for apache mod_gzip and you'll find all you need to know.

Once again ... (2, Interesting)

Buck2 (50253) | more than 13 years ago | (#2299210)

I received my first indication that something was happening in the world through Slashdot.

I heard that the other sites weren't available, but I wouldn't have known this empirically since there were already so many quality posts, comments, and bits of information to sort through in the Slashdot forum.

Any long term Usenet denizen will know what I speak of when I refer to that rare, but addictive, experience that seems to be only able to be brought by these forums of such spatially disconnected people, joined only by common interests. When, seemingly all of a sudden, someone writes something so perfect, so funny, so outrageous, so wonderful, so _different_, or so incredibly informative, that you all of a sudden feel justified for all of the times that you wondered why you just kept coming back.

Slashdot did it again.

BTW, perhaps the moderators have been out in force, or maybe I'm just getting old and much more interested in politics than before, but it seems that the quality of posts in the last few days have been much more thoughtful and interesting.

Kudos to both the Slashteam and community.

Yes, your contribution is VERY important. (1)

!Xabbu (1769) | more than 13 years ago | (#2299214)

In this day and age sifting through the bullshit online has become increasingly harder and harder as every day passes. I hit Slashdot almost as much as I hit CNN for news in the past 4 days and while CNN had up to date information that was over all quite good, it as usual is filtered to fit their format only (you can gather much more from other sources). You guys had the links to mirrors, donation pages, real people stories, original thoughts on everything on how all this may have happened to why it was possible that the towers collapsed. All this before the mainstream media thought it important to report on. I swear I must have seen the crash from every angle possible about a hundred times without any new news.

Thank you.

wow... (0)

Anonymous Coward | more than 13 years ago | (#2299215)

...mysql finally does locking?

Stuff that Mattters (2)

gorilla (36491) | more than 13 years ago | (#2299217)

I made the decision to cancel Slashdot's normal daily coverage of "News for Nerds, Stuff that Matters,"

If this wasn't "Stuff that Matters" I don't know what is.

Of Slash and Slashdot (5, Insightful)

ajs (35943) | more than 13 years ago | (#2299221)

When I started reading this, I was disgusted. I was expecting something like CNN's ads after the Gulf war, touting the fact that they were the ones who got most of the scoops.

By the time I got half-way through the actuall content (not the front-page piece) I was in awe of how much went on. Usually when a massive load spike happens on my watch, I try to get everyone's fingers out of the pie so that we have a good chance of the machines just doing their jobs. The fact that these folks were able to make emergency changes in real-time to compensate for the load is just astounding.

CNN should be rolling out a Slash-based discussion forum for top stories. Heck, so should Whitehouse.gov!

Thanks guys, and good luck with your ongoing coverage of News For Nerds, Stuff That Matters!

Pictures? (-1)

HarveyNeon (106183) | more than 13 years ago | (#2299223)

Hey cmdrtaco, how about some pictures of your server room? Sorry if this is off topic but your valiant story and compliments to your team are screaming for photos to back it all up. What does it look like in the /. server room? This is a question I've had for a long time.

Thanks.

Harveyneon

rhettnye.org [rhettnye.org] updated the day before 911.

More Kudos (1)

haus (129916) | more than 13 years ago | (#2299224)

Thanks!

I just wanted to let you know that I appreciated the great work that the ./ crew has done this week. ./ Is where I found out about the tragedy on Tuesday morning, and through the links in the stories and by the fellow ./ posters, how I was able to get more detailed news on what ad actually happened.

You should feel proud of the job that you performed and the service that you provided.

Slashdot: Service to the community (1)

Matey-O (518004) | more than 13 years ago | (#2299225)

Stuck in the airport, all TVs in the councourses we were just evacuated out of, My Ipaq and Nokia 8290 were ONLY able to reach Slashdot. Don't kid yourselves, you provided a GREAT service...for free.

Irony (5, Funny)

Skyshadow (508) | more than 13 years ago | (#2299227)

Knock on wood, Taco -- saying stuff like this tends to be the perfect cue for your servers to crash an burn.

Other phrases to avoid:
- Boy, sendmail's been rock solid for months!
- Hey, I've been driving years without a ticket/accident.
- Wow, this economy is unstoppable!
- I don't have to run in to apply that patch; what are the odds some script-kiddie will notice before Monday?
- Alright! I'm worth millions in stock options!
- Pft, what are the odds she'll get pregnant from just that one time?

Wow. Thanks- That feels good. (1)

Zzootnik (179922) | more than 13 years ago | (#2299228)


To Quote:

Slashteam kicks butt. Jamie, Pudge, Krow, Yazz, Cliff, Michael, Jamie, Timothy, CowboyNeal, you guys all rocked.

Truly people, you do. If at ANY time, we could have expected the site to be "Slash-Dotted" itself, this was it.

This is good code, and you know how to use it.

One of the few marvels that's made me smile since Tuesday.

thank you (1)

insane4no1 (26385) | more than 13 years ago | (#2299232)

i just wanted to add my thanks to everyone elses. thoughout the day i relied on slashdot to learn the newest information, and pointed several friends to it. thank you for all the hard work and great jobs you guys have done on such a constant basis. as one of many people who looked to you for the latest news i was never disappointed and always well pleased with the performance, and the content.

Good job, thanks! (1)

ToasterTester (95180) | more than 13 years ago | (#2299235)

What we saw on TV and things we read on sites like /. will remain with us the rest of our lives. Thanks for all the work you did to keep /. up Tuesday and thanks for summarizing what it took so we can all learn as well.

thanks Slashteam (1)

caseydk (203763) | more than 13 years ago | (#2299238)

Thanks Slashteam, you guys did a great job of keeping up under the load and keeping the information flowing...

I, for one, am in the dc area and phone lines were blocked and i was at work without a tv or radio... you guys managed to keep many of us in touch with the outside world and the breaking news (even if some of it was a bit wrong)

Thanks.

Just a thank you (1)

Hangtime (19526) | more than 13 years ago | (#2299242)

It doesn't seem like much but its what I feel right now. April 19th, 1995 I was a sophomore in high school living on the southwest corner of Oklahoma City watching wall-to-wall coverage of what happened to my city and trying to do anything to help. Tuesday brought back many bad memories that I never really wanted to resurface but did. Being able to get information and find out what was happening......I'm going to start rambling so Im going to stop and just say this. I wanna thank the Slashdot Team for being just about the only place where I could find information on Tuesday and to each one of the posters who were mirroring reports here that people couldn't get to. (A special thanks to the person who linked to Sky News Broadband; I couldn't get to a TV and they were using FoxNews coverage). God Bless each of you that contributed reports and God Bless America

HT

exactly. (1)

zama (244613) | more than 13 years ago | (#2299245)

That's all I can say. Whyte Wolf said it all. Thanks Slashdot - you pulled through beautifully when it was crunch time. And thanks to all the Slashdotters for the insights, commentary, diverstity, and community at this time.

Hey JackAss! (-1)

sucko (257144) | more than 13 years ago | (#2299249)

How about an article describing the problems you've been so quiet about since the switch over? This self-congratulatory crap sickens me.

YATAC (1)

bsletten (20271) | more than 13 years ago | (#2299252)

This is just yet another thanks and congrats.

I was impressed at how well /. held up on Tuesday
and I appreciate your taking the time to explain
how.

Kudos.

I was at work (2)

einTier (33752) | more than 13 years ago | (#2299255)

Since I was at work, I had no TV, and no radio -- everyone thought someone else would bring one (it's a small office). So, I was stuck with no way to get any real data. Slashdot was the only site working, and the only place to get my news updates. At the time, I didn't think much of it -- this WAS slashdot, after all. I even directed my friends to it, saying "Slashdot has news and they are up." I really took the hard work for granted, though I wondered how you stayed up when CNN, MSNBC, and all other major news sites were completely down.


CmdrTaco, thank you so much for the hard work that went into making slashdot the site it was Tuesday. I would have been lost without it.

Its Been Said (2)

Cylix (55374) | more than 13 years ago | (#2299258)

Slashdot held under the tremendous load. Yeah, it was sluggish at times. It was sluggish when all the other sites were failing.

I stayed home a little earlier on tuesday to watch the news. I didn't want to leave for work, but it wasn't really an option to stay home. I was really hoping I could keep up on the events with most of the major news sources being online.

Slashdot served as a place of information. Many posters were pasting articles as they were able to retrieve them from sites as they opened up temporarily.

All of slashdot team's efforts and the posters deserve a good deal of thanks.

As always, keep up the good work.

Enders Game (0)

Anonymous Coward | more than 13 years ago | (#2299259)

In "Enders Game" there there is frequent reference to 'the nets' where people join in discussions and how important these are to the public opinion, basically the place where democracy is going to take place.
Especially over the last few days, I was under the impression that 'the nets' will be something very similar to Slashdot.
I haven't seen another place where such a lively and for the most part, reasonable discussion took place, and no other system that seemed to handle the rush apparantly without a glitch.

AC

Hmm... (0)

Anonymous Coward | more than 13 years ago | (#2299260)

But only the computer community reads slashdot. The rest of the world do not.

Yes it is a tragedy what happened. We will fight it. It is also a tragedy that 20 Palestinans were killed 2 days ago and even more Iraqi children died in the past week. This has to stop.

Just Another (1)

grubby (121481) | more than 13 years ago | (#2299262)

When I first heard of the tragedy I had users coming to me wondering why they couldn't get to the news sites. I checked myself and sure enough most were dead as could be. So I checked slashdot and sure enough got right in. I just want to say GREAT job and you guys deserve a pat on the back for this. I get so tired of hearing all the people complain about /. and ct and michael, etc... It sis really great that we were able to share the news all together. Thanks again /. !

Emergency Mode (3, Interesting)

datavortex (132049) | more than 13 years ago | (#2299265)

Rob touched on this in what he said above, but I also wanted to bring to attention this story [slashcode.com] that is of similar subject matter posted Wednesday on the Slash site. It's an idea for a slash feature to automatically do several of the things that the Slashteam did manually on Tuesday to keep the site alive. Things such as serving static HTML, disabling or changing the functionality of the search and other dynamic functionality, etc.

I'd also like to throw in my public note of thanks to everyone who kept the site up on Tuesday. We thirsted for answers, and you were there to provide, as always. Your work and dedication are wholly appreciated.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?