Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Introducing The New Slashdot Setup

CmdrTaco posted more than 14 years ago | from the its-been-a-long-time-coming dept. 306

At the request of countless users, we're happy to finally present a summary of the new setup over at Exodus. It's the result of over 6 months of work from a lot of people, so shadouts to Adam, Kurt, and Scoop, Team P:Pudge, PatG & Pater for the code, and Martin BSD-Pat and Liz for getting the hardware and co-loc taken care of. Now hit the link below and see what these guys did:

the original version of this document was written by Andover.Net Alpha Geek Kurt Grey. The funny jokes are his. The stupid jokes are mine.

The Backstory
We realized soon that our setup at Digital Nation was very flawed. We were having great difficulty administering the machines and making changes. But the real problem was that all the SQL traffic was flowing over the same switch. The decision was made to move to Exodus to solve these problems, as well as to go to a provider that would allow us to scatter multiple data centers around the world when we were ready to do so.

Meanwhile Slashcode kicked and screamed its way to v1.0 at the iron fists of Pudge (Chris Nandor) and CaptTofu (Patrick Galbraith). The list of bugfixes stretches many miles, and the world rejoiced, although Slashdot itself continued to run the old code until we made the move.

The Co-Loc
Slashdot's new co-location site is now at Andover.Net's own (pinky finger to the mouth) $1 million dedicated datacenter at the Exodus network facility in Waltham, Mass, which has the added advantage of being less than 30 minute drive for most of our network admins -- so they don't have to fly cross-country to install machines. We have some racks sitting at Exodus. All boxes are networked together through a Cisco 6509 w/ 2 MSFCs and a Cisco 3500 so we can rearrange our internal network topology just by reconfiguring the switch. Internet connectivity to/from the outside world all flows through an Arrowpoint CS-800 (which replaced the CS-100 that blew up last week) switch which acts as both a firewall load balancer for the front end Web servers. It also so happens that the Arrowpoint shares the same office building with Andover.Net in Acton so whenever we need Arrowpoint tech support we just walk upstairs and talk to the engineers. Like, say, last week when the 100 blew up ;)

The Hardware

  • 5 load balanced Web servers dedicated to pages
  • 3 load balanced Web servers dedicated to images
  • 1 SQL server
  • 1 NFS Server

All the boxes are VA Linux Systems FullOns running Debian (except for the SQL box). Each box (except for the SQL box) has LVD SCSI w/ 10,000 RPM drives. And they all have 2 Intel EtherExpress 100 LAN adapters.

The Software
Slashdot itself is finally running the latest release of Slashcode (it was pretty amusing being out of date with our own code: for nearly a year the code release lagged behind Slashdot, but my how the tables have turned).

Slashcode itself is based on Apache, mod_perl and MySQL. The MySQL and Apache configs are still being tweaked -- part of the trick is to keep the MaxClients setting in httpd.conf on each web server low enough to not overwhelm the connection limits of database, which in turn depends on the process limits of the kernel, which can all be tweaked until a state of perfect zen balance has been achieved ... this is one of the trickier parts. Run 'ab' (the apache bench tool) with a few different settings, then tweak SQL a bit. Repeat. Tweak httpd a bit. Repeat. Drink coffee. Repeat until dead. And every time you add or change hardware, you start over!

The Adfu ad system has been replaced with a small Apache module written in C for better performance, and that too will be open sourced When It's Ready (tm). This was done to make things consistant across all of Andover.Net (I personally prefer Adfu, but since I'm not the one who has to read the reports and maintain the list of ads, I don't really care what Slashdot runs).

Fault tolerance was a big issue. We've started by load balancing anything that could easily be balanced, but balancing MySQL is harder. We're funding development efforts with the MySQL team to add database replication and rollback capabilities to MySQL (these improvements will of course be rolled into the normal MySQL release as well).

We're also developing some in-house software (code named "Oddessey") that will keep each Slashdot box sychronized with a hot-spare box, so in case a box suddenly dies it will automatically be replaced with a hot-spare box -- kind of a RAID-for-servers solution (imagine... a Beuwolf cluster of these? *rimshot*) Yes, when it'll also be released as open source when its functional.

Security Measures
The Matrix sits behind a firewalling BSD box and an Arrowpoint Load balancer. Each filters certain kinds of attacks and frees up the httpd boxes to concentrate on just serving httpd and allows the dedicated hardware to do what it does best. All administrative access is made through a VPN (which is just another box).

Hardware Details

Type I (web server)
VA Full On 2x2 Debian Linux frozen
PIII/600 Mhz 512K cache
9.1GB LVD SCSI w/ hot swap backplane
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter

Type II (kernel NFS w/ kernel locking)
VA Full On 2x2
Debian Linux frozen
Dual PIII/600 Mhz
(2) 9.1GB LVD SCSI w/ hot swap backplane
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter

Type III (SQL)
VA Research 3500
Red Hat Linux 6.2 (final release + tweaks)
Quad Xeon 550 Mhz, 1MB cache
6 LVD disks, 10000 RPM (1 system disk, 5 disks for RAID5)
Mylex Extreme RAID controller 16 MB cache
Intel EtherExpress Pro (built-in on moboard)
Intel EtherExpress 100 adapter

cancel ×


Sorry! There are no comments related to the filter you selected.

MySQL in General. (2)

viper21 (16860) | more than 14 years ago | (#1063852)

Ok, so now that we have established the fact that MySQL isn't beefy enough. Why was the choice made to go with MySQL instead of Oracle or PostgreSQL... etc? Just because the code was written for MySQL, or for some other reason that eludes us all?


Scott Ruttencutter

Re:Interesting Setup (2)

Anonymous Coward | more than 14 years ago | (#1063855)


The site recieves somewhere between 1 and 2 million page views a day. The mySQL server has to be 'beefy' to take that kind of load under mySQL while we wait for replication to be stable and complete.

If you ask "why not __X__" for the database.. our brave programmers are working on a general abstraction layer to allow different databases to be used with slash that is NotReadyYet(tm). When that is complete, people will be able to port slash to the database of choice. For now, the only db supported is mySQL.b

Martin B.

Re: MySQL Server. (1)

jlanng (130635) | more than 14 years ago | (#1063857)

You don't need replication support for data partitioning, which is actually a better choice in most cases. Don't know how Slashdot works, so couldn't comment either way. (To the uninitiated, an example of data partitioning would be storing all the odd-numbered records on ServerA, and the even-numbered records on ServerB. This is clearly a pain for table joins, however) The big question is (and I'm not trying to be a troll here)... why not use a proper database in the first place?

Testing (1)

LaNMaN2000 (173615) | more than 14 years ago | (#1063879)

Have you tested the setup under a variety of loads? Approximately what was its capacity? After the recent group of DDOS attacks, have you installed any safeguards to revent similar attacks in the future.

Hopefully this setup will be able to handle Slashdot's growth and end the insufferable page loading delays that have become all too common. Kudos to the IT staff for successfully migrating to the new system.

Seems faster. (2)

tcd004 (134130) | more than 14 years ago | (#1063883)

Imagine a beo... oh nevermind.

I wonder how many peoplehours go into running slashdot each week?


Here's my Microsoft Parody [] , where's yours?

Next on the list... (2)

tringstad (168599) | more than 14 years ago | (#1063886)

Update the "about" page on Slashdot, which still says, "Bandwidth provided by Digital Nation."


"I do not think much of a man who is not wiser today than he was yesterday."

no, silly, they mean shadouts (2)

georgeha (43752) | more than 14 years ago | (#1063963)

a bunch of old, wrinkly, smelly violent fremen hosuekeepers are on their way now to help out the network admins.


SQL Server (1)

jstepka (20825) | more than 14 years ago | (#1063967)

I'm wondering if you are thinking of moving to a db with ACID support? Also, how are you replicating data should you take a hit on your one SQL server?

what's funny is that... (3)

T.Hobbes (101603) | more than 14 years ago | (#1063973)

... the amount of relevant information has remained the same ever since /.'s been on a dusty ol' 386 with a 14.4kb/s pipe...

(that is not to say the trollers and such are not fun, which they are.. they're just not useful outside the context of /. ...)

Why Debian/RH? (4)

waynem77 (83902) | more than 14 years ago | (#1063974)

Forgive me if this has been asked elsewhere, but why did y'all choose those distributions for those servers? I'm genuinely curious; I'm unfamiliar with the large-scale differences between distributions. (My computer runs Mandrake... that decision was based on the single factor that my friend happened to have a Mandrake CD on him.)

what broke the code? (1)

Fat Lenny (150637) | more than 14 years ago | (#1063977)

There are a number of things that don't work, like my comment preferences defaulting to Threaded instead of Nested, and when you click a CID link, the Moderation Totals can't be seen because it's in white text...

Maybe this is a screwup, maybe it's because I prefer browsing with Windows IE -- so sue me.


For the record -- "Exodus"?? (2)

devphil (51341) | more than 14 years ago | (#1063985)

Who/what exactly is Exodus?

All I can see from their homepage is that they're vaguely ISP-looking. But I have a personal policy of Not Bothering With Flashy Graphics-Laden Web Pages, so I didn't push any further through the morass.

From the context of the previous play-by-play article, I take it Exodus provides physical storage space and connectivity for your machines, and not much else...?

/. garage sale, or maybe ebay, (1)

georgeha (43752) | more than 14 years ago | (#1063995)

Can I borrow the Quad Xeon? Maybe just one of the DUAL PII systems? Surely you have enough systems from old slashdot servers to put them together ;)

So when is the /. garage sale?

Or are you gonna ditch the old stuff on ebay, or

How much for the ugly couch in the WiReD article?


MySQL Server. (3)

viper21 (16860) | more than 14 years ago | (#1064001)

Why was the choice made to use one beefy-as-hell SQL server instead of multiple lesser powered systems?

Scott Ruttencutter

NFS Question (2)

Johnboy (15518) | more than 14 years ago | (#1064002)

Can anybody tell me what purpose the NFS box serves? Multiple load-balanced webservers connecting to a single SQL server -- that's commonplace, but I can't figure out what is on the NFS box or which other boxes need that data. Am I missing something? How about a Slashdot topology diagram?

Interesting Setup (3)

Elik (12920) | more than 14 years ago | (#1064004)

I looked over the setup for those servers and frankly, they seems to be quite an overkill, for this site itself. Maybe I am wrong, but if this site handles over that many people, since I have no clue on how many been hitting the site, then the setup is not bad. But Quad Xeon Processors for MySQL? What wrong with Dual Pentium II/III or single Athlon MySQL server?

I can understand the need for failover for the MySQL which is a major requirement, but the computer itself is quite a major overkill. I do adminstration for over 16 different servers over 3 different clients and we uses Mandrake Distros on those and only problem I usually finds is the MySQL can be really stressful when there is many people using it on the same server with the apache server. I am in process of moving the MySQL over to new delicated server so it can handle that among several servers, and maybe that might call for beefy processor to handle the load.

If the site is setup to handle... let say.... 100,000 hits a day, what server configuration is needed for this? With MySQL on it own server, of course. :)

dual nics? (1)

miss_america (177725) | more than 14 years ago | (#1064007)

ok i see they have dual nics? is that for fault-tolerance or load-balancing, or what?

Re: MySQL Server. (1)

InitZero (14837) | more than 14 years ago | (#1064011)

Why was the choice made to use one beefy-as-hell SQL server instead of multiple lesser powered systems?

Didn't you hear? MySQL doesn't do replication. Without such tools, creating multiple servers and keeping them in sync would be a massive pain in the buttocks.


Re:MySQL Server. (1)

The Dev (19322) | more than 14 years ago | (#1064013)

That is because it is difficult/tedious to write code that handles multiple db writes, especially to multiple servers. Expecially with MySQL where you would have to explicitly verify all writes to make sure everything is in sync. This will hopefully be fixed when replication is added to MySQL.

Pictures!! (3)

a.out (31606) | more than 14 years ago | (#1064017)

Everybody Chant ..

"We Want Pictures.. We want Pictures.. We want Pictures"

Seriously though: it would be nice to actually see this setup. Don't forget to have CowboyNeal give us an oh so sexy pose near the almost as sexy VA SQL server... :)

Thank you Slashdot! (3)

colindiz (162137) | more than 14 years ago | (#1064018)

This series is very useful for people just starting to work in linux admin environments, even if it's just a personal server.

Lots of varying methods are discussed of how to properly protect or run a server, and now we get a real life scenario of what happens when the shit hits the fan.

Don't just publish the Hellfire series, package up this one too.

What is all the hardware needed for? (1)

pyronicide (104978) | more than 14 years ago | (#1064021)

I am sorry, but i do not know anything about web serving, anyways, why are all those SWEET systems needed when a 386 can fill a 10meg pipe? It is cool to list/know everything at Slashdot is running on sweet systems like that, why is so much horsepower needed?

Blown up arrowpoint? (1)

getha (97821) | more than 14 years ago | (#1064023)

Just goes to show how little I know about these things...
[..] through an Arrowpoint CS-800 (which replaced the CS-100 that blew up last week)
Do you really mean to say that one of those arrowpoints is a piece of junk after a DDos? Or did this have to do with something else?

Well, since my ignorance is now fully established, I'll just say I think you've got some nice stuff there...

xchg .,@

BSD Firewall.... (2)

Anonymous Coward | more than 14 years ago | (#1064025)

What is the config of the BSD Firewall? Hardware/Software...

I have a couple of questions and a comment. (3)

Dante (3418) | more than 14 years ago | (#1064028)

I wonder why there is a red hat box in the mix? what is the reason? Now I am a Debian bigot, but my guess is that so are you guys, is there somthing specific about Redhat and mysql that I don't know?

Second why not mylex cards in all the box's? mylex's new DAC110 SCSI cards are simply the fastest I have ever seen.

Why not Gigabit? I use it with Linux it works, it makes all that heavy duty hardware sing, 100mbit is just a passe :>

I am proud that you chose Potato for most of your box's

Re:what's funny is that... (1)

slpalmer (6337) | more than 14 years ago | (#1064030)

ever since /.'s been on a dusty ol' 386 with a 14.4kb/s pipe I seems to recall a post from CmdTaco way back when talking about the original /. server being a DEC Multia (Alpha).
Here is the result of your Slashdot Purity Test.
You answered "yes" to 86 of 200 questions, making you 57.0%

Why did you choose MySQL? (1)

gimpboy (34912) | more than 14 years ago | (#1064031)

Fault tolerance was a big issue. We've started by load balancing anything that could easily be balanced, but balancing MySQL is harder. We're funding development efforts with the MySQL team to add database replication and rollback capabilities to MySQL (these improvements will of course be rolled into the normal MySQL release as well).

If this was a big issue why did you choose MySQL over PostgreSQL? I dont know about replication, but Postgres does support transactions/rollback.


Re:I got two words for ya... (2)

EngrBohn (5364) | more than 14 years ago | (#1064033)

Mmm. Slashquake. Imagine the possibilities. []
Christopher A. Bohn

Oddessey? (1)

Matthew Weigel (888) | more than 14 years ago | (#1064039)

So we all know CmdrTaco is horrible at spelling, my question is, is "Oddessey" a knowing and humorous play on that, or is it an error? Given that the correct (or accepted as correct in English) spelling would be odyssey, is there any plan to correct the spelling?

I'm genuinely curious... with some people, it's difficult to tell the intentional from the unintentional spelling errors.

Re:Thank you Slashdot! (1)

colindiz (162137) | more than 14 years ago | (#1064043)

Er, I meant to say Hellmouth. Blorf.

Re:For the record -- "Exodus"?? (2)

Elian (10270) | more than 14 years ago | (#1064045)

It's a big colocation facility. Lots of redundant network connections to several of the major backbones, power, 24-hour security, that sort of thing. It's the place you'd build for your own datacenter if you only had huge gobs of cash and time.

Re:what's funny is that... (3)

Rendus (2430) | more than 14 years ago | (#1064047)

Actually, Slashdot started on a Multia :) Not far off from The 386 In The Corner(TM) though. It was actually the mail server for his employer at the time (The Image Group I believe, can't remember for sure), and he ran /. off it (this was back when 100 comments on a story was unheard of) until it couldn't handle the load anymore. Personally though I think Slashdot had more interesting topics back then as well. -Shrug-

Why spend all that $ to fix MySQL? (5)

Evro (18923) | more than 14 years ago | (#1064048)

Fault tolerance was a big issue. We've started by load balancing anything that could easily be balanced, but balancing MySQL is harder. We're funding development efforts with the MySQL team to add database replication and rollback capabilities to MySQL (these improvements will of course be rolled into the normal MySQL release as well).

Just out of curiosity, wouldn't it be easier to use something like PostgreSQL [] (which is just as freely available) that already has rollback & atomicity than to pay the MySQL people to develop it? Didn't y'all read the article on here a few weeks ago, "Why not MySQL? [] "

__________________________________________________ ___

Re:For the record -- "Exodus"?? (4)

Zico (14255) | more than 14 years ago | (#1064050)

I'd tell you, but I have a personal policy against helping lazy luddites who think they're taking some kind of principled stand because they don't visit sites that use <img> tags. At least read the damn FAQ linked to on their home page.


3 Image servers? (2)

BoLean (41374) | more than 14 years ago | (#1064053)

,i>>3 load balanced Web servers dedicated to images

Why three image servers? Slashdot isn't exactly the most graphics intensive site I've seen. A few icons and the banner ads. Are you planning more graphics/art? Or, is this just to ensure that the Ads are loaded quicker than the rest or the page (jab to the ribs!).

Bonehead question... (1)

sid crimson (46823) | more than 14 years ago | (#1064062)

Since I'm still on the steep end of the learning curve and the top is not in sight... :-)

I'm curious why there would be 5 servers dedicated to serving pages and 3 more just to images? As opposed to 8 servers dedicated to serving up the complete page?

Or maybe a reverse-proxy arrangement?

Seriously. This stuff is great and I wanna know more!

Open source? (2)

Anonymous Coward | more than 14 years ago | (#1064063)

It sounds like you've got some interesting software products in the works. Instead of releasing them When They're Ready, why not release them Now? I always thought that collaborative development was crucial to open source -- that's certainly a recurring theme on these pages, especially when Microsoft code blows up.

Why following a "cathedral" development model when there are many folks out here willing and eager to help out with your code?

Re:Why Debian/RH? (1)

rtscts (156396) | more than 14 years ago | (#1064064)

so would I..

why RH for the SQL server when the rest are Debian? RH runs Postgres, not MySQL...

switching requires a recompile of Apache - a royal PITA IME to get all the modules to compile. On my box, I ended up copying the Apache RPM's from my Mandrake CD instead (can't run mandrake, it won't install and running the non-GUI installer requires more effort than just booting off the CD)

Re:MySQL Server. (1)

nweaver (113078) | more than 14 years ago | (#1064065)

I think the article made this clear: MySQL is not set up to act as a distributed system, and would take considerable work to do so (work which, out of enlightened self interest, andover is funding).

So currently, the solution of "buy a big beefy machine" is the only one which works.

and what's the root password, too? (2)

georgeha (43752) | more than 14 years ago | (#1064066)

just curious

Think Different, man! (1)

Electric Eye (5518) | more than 14 years ago | (#1064067)

You could've saved yourselves a lot of money and got a nice Beowolf cluster of green iMacs..... You rich guys like to go overboard, don't ya? ;-)

1 Gig on web servers? (1)

The Dev (19322) | more than 14 years ago | (#1064068)

Do you really need 1 Gig of ram on the web servers? Wouldn't the extra memory be better
used on the DB server? You're probably not using
more than 150MB for the httpd's.

Re:What is all the hardware needed for? (3)

Dr. Sp0ng (24354) | more than 14 years ago | (#1064069)

Well, a 386 can't fill a 10meg pipe - a 486 could, but not a 386. Also, a 486 could only do it if it was only static content (i.e. pre-written HTML and images.) Slashdot isn't static content by any stretch of the imagination. The pages are dynamically generated for each user using data from MySQL. MySQL is very CPU/IO intensive, as is mod_perl, which generates the HTML.

The specs for the Type III box.... (1)

Salgak1 (20136) | more than 14 years ago | (#1064071)

. . .can we safely assume that you're running the same 9.1 GB SCSI hotswaps ??

And why no Athlon boxen ... or is that for the future ????

In any case, looks like nice, solid technology, as opposed to bleeding-edge tech, and thus extremely reliable. My hats off to your network architect: VERY nicely done. . . .

More important question (2)

Ford Prefect (8777) | more than 14 years ago | (#1064072)

I wonder how many peoplehours go into running slashdot each week?

More importantly, how many people-hours are lost due to reading Slashdot each week? :-)

Ford Prefect

Waltham, MA? (2)

Mark F. Komarinski (97174) | more than 14 years ago | (#1064076)

Heck, I work in Waltham. Next time you're out this way, give me a call/email. I'll buy you a beer. You deserve it.


Total cost (3)

ajakk (29927) | more than 14 years ago | (#1064078)

I am sure everyone wants to know how much everything here costs. Here are the calculations for the linux boxes(info is off of the va linux custom configuration program):

webservers (type 1) = $4505 each

NFS server (type 2) = $7040

database server (type 3) = $25739

So the grand total is $68819. I haven't found the prices for the switches and firewall. I would suspect that the BSD box is close in price to the webserver (prob. a bit less).

Re:1 Gig on web servers? (3)

Dr. Sp0ng (24354) | more than 14 years ago | (#1064079)

Do you really need 1 Gig of ram on the web servers? Wouldn't the extra memory be better used on the DB server? You're probably not using more than 150MB for the httpd's.

Not true at all. I'm running a slash server [] which doesn't get very many hits (~3000 in the last two days, chump change compared to Slashdot) and right now httpd is using 270MB of RAM.

Re:For the record -- "Exodus"?? (2)

RoufTop (94425) | more than 14 years ago | (#1064081)

Exodus is a datacenter company in the real estate business. Essentially, they build buildings, put in huge air conditioners, fat digital pipes, and security guards, then sell cages within for individual sites' boxes. They'll monitor servers but generally are pretty hands off, so its a good deal for companies that have their own admins but don't want to build their own datacenters. In short, your estimation is on the money.

Re:MySQL Server. (5)

irix (22687) | more than 14 years ago | (#1064082)

This isn't flamebait - I'm honestly wondering why you wouldn't go away from MySQL here. It is obvious that money isn't a concern - so why not fork over the money for some Oracle licenses and a competent Oracle admin?

It's not like it is for some sort of open-source reason - MySQL isn't released under an open-source license. I'm curious why slashdot/Andover are spending money funding a closed source project rather than funding an open-source one or forking over the $ for a more capable database like Oracle.

Exodus is a *BIG* ISP (5)

DragonHawk (21256) | more than 14 years ago | (#1064086)

Who/what exactly is Exodus?

Exodus is one of the world's biggest (in terms of service capacity available) Internet Service Providers.

"We're going to need bandwidth. Lots of bandwidth."

Exodus specializes in having more bandwidth then most of the third world. They've got NAPs (Network Access Points, i.e., backbone connections) all over the continental United States, and a few outside the US as well. They link this all together using both external and internal networks. The end result is, most anywhere on the net that has a good connection, has a good connection to Exodus.

They provide servers. Do you need to host downloads for ten million users? Exodus can give you servers to do so.

They provide co-location space. If their standard server packages just won't cut it -- bring your own. They'll give you a rack, a dedicated co-loc cage, or a dedicated high security vault.

Their web page [] has a lot of graphics because they have a lot of pictures of their equipment and graphs of their capacity. It is actually justified. You may want to make a return trip.

Re:Blown up arrowpoint? (2)

gavinhall (33) | more than 14 years ago | (#1064088)

Posted by BSD-Pat:

Essentially the CS-100 had content checking on, combined with the SYN (and the fact it was a loaner unit/refurb until the 800 came) it died.

Arrowpoint is doing a post-mortem now.


Re:MySQL Server. (1)

kirwin (71594) | more than 14 years ago | (#1064089)

MySQL cannot handle such features as datbase replication, and parallel server tasking. This will probably change when the linux kernel supports raw devices (2.4 from what I have heard).

I suffer from the same issue with an Oracle database.

Interesting... (2)

Psarchasm (6377) | more than 14 years ago | (#1064092)

You don't really talk a whole lot about where and for what you are using the NFS box for. How is linux's NFS working for you?

All I've ever heard were horror stories about its poor implementation. Water under the bridge? Righty right?

I assume your sharing static content and image serving using the NFS shares. Perhaps you've even gone so far as to utilize qmail or the sendmail/procmail NFS mail hack in order to store your email on the NFS share?

What shyed you away from the Linux Virtual Server Project's implementation of load balancing? Or the mon+hearbeat+fake+coda solution for HA? Or is your "Oddessey" work based on this in any way?

Are you loadbalancing your firewalling BSD box? Or have we reached a critical point of failure at the firewall?

Very interesting stuff, I'd like to know more details though.

Re:Why Debian/RH? (1)

matticus (93537) | more than 14 years ago | (#1064095)

slashdot runs debian, to my knowledge, mainly because it's rob's favorite. he is a debian die-hard. the sql server running redhat is new-i'm not sure why they picked it other than i've heard redhat is good for sql.

Re:For the record -- "Exodus"?? (3)

phungus (23049) | more than 14 years ago | (#1064096)

Exodus is a very high-class co-location provider. They have lots of datacenters across the world which are all 1000's of square feet each and house sites like Hotmail, Lycos, etc. In each of their datacenters they bring in many pipes to all the major internet backbone providers as well as AOL, etc.

They have, literally, 1000's of racks and TONS of machines. At the Sterling, Virginia facility which we moved into (which, BTW, is one of their newest and flagship facilities) I saw SGI Origin machines, countless Sun enterprise-level machines, mainframes, small machines, etc, etc, etc. They run OC-12, 48, or more between each of their centers. They offer "Datacenters within the Datacenter" which are little rooms constructed on the raised floor which offer a secure environment that companies can pay (lots!) for. They have fibre coming in to multiple points in most buildings, many generators, huge UPS's, fingerprint readers to get into the network rooms, etc, etc, etc.

It's pretty phat. Their service is top notch as well. I had to fly up there from Dallas and it was an immense pleasure (even though I had to work. :)

They're pretty awesome (and no, I don't work for them).


Re:Testing (1)

geoffeg (15786) | more than 14 years ago | (#1064099)

>Have you tested the setup under a variety of loads?

Uhm, they installed a BSD box as a firewall to filter out unwanted and off packets. Read the article from yesterday.

Re:dual nics? (2)

bluGill (862) | more than 14 years ago | (#1064100)

Most likely one nic connects to the internet (through the firewall), and the other to the database backend.

Assuming the above is true, there are several advantages. First, data traffic to the sql server isn't generating colision with data from the internet, this allows both less ethernet collisions for more bandwidth, and more security. The Sql machine can run the every buggy service imanginable. It won't matter because nothing can get to it. The web servers only listen on ssh and httpd ports, and don't forward anytying. The sql network can be a private (192.168.x.x) network which isn't routable. You can still break it I suppose, but much more difficult.

Re:what broke the code? (1)

albamuth (166801) | more than 14 years ago | (#1064102)

Gee, I don't seem to have ANY problems at all surfing with my relatively feature-low NetPositive...

until I go anywhere with Javascript, Applets, Flash, etc.


Cachedot? (3)

Mycroft-X (11435) | more than 14 years ago | (#1064104)

I remember back when there was a that maintained a replica in case the server was having trouble. Is there any chance that this could be implemented again but in a different location?

Of course such a thing would not need to be as powerful as the main slashdot systems, but would provide some additional backup in case of another DDOS or a network outage of some sort.

Sort of a "battle bridge" for those of you who remember the days when Star Trek was good. (startrek.version = ST:TNG)


Debian not on the SQL server? (4)

Oskuro (26466) | more than 14 years ago | (#1064105)

There has been a small talk about this story in the debian-devel channel on IRC just some minutes ago, and of course the great question was:
Why isn't the SQL server Debian as well?
If there's any problem with Potato's MySQL, I think Debian would be pleased to hear, whether it's a bug report in the BTS or whatever.


Re:Why spend all that $ to fix MySQL? (1)

einstein (10761) | more than 14 years ago | (#1064107)

I think it's because Taco is an admitted MySQL junkie, and the fact that the Slashcode is already so embedded in MySQL that a transition would be sorta painful.

Automate the benching (1)

Anonymous Coward | more than 14 years ago | (#1064108)

Run 'ab' (the apache bench tool) with a few different settings, then tweak SQL a bit. Repeat. Tweak httpd a bit. Repeat. Drink coffee. Repeat until dead. And every time you add or change hardware, you start over!

I don't want to be a spoil-sport, but this kind of mechanic work sounds like a job for software!

Re:NFS Question (2)

Bluecoat93 (140994) | more than 14 years ago | (#1064109)

It's very commonplace to have multiple load-balanced web servers talking to a single NFS-mounted disk storage backend. It prevents problems with replicating content to all your web servers. At my company, we use a farm of 32 or so Sun web servers all sharing a NetApp filer for storage. All of your files and scripts are pulled from the same source. Plus, you don't have to invest in huge amounts of disk for each of your web servers. You just put your money into the NFS server. Also makes backups much easier. Cheers, Brian

Some Feedback (1)

neowintermute (81982) | more than 14 years ago | (#1064110)

I for one think that the new setup kicks ass. It's much much faster than the old setup. great job guys!

unfortunately the "highest score first" still doesn't work right, but I'd rather have faster loading pages than anything else. It's a pretty big feat when you think about how big the slashdot pages really are, with 300 comments in an html page generated on the fly. wowza.

Michael Cardenas

Re:Interesting Setup (1)

grarg (94486) | more than 14 years ago | (#1064111)

On that note, a basic question: how many hits does Slashdot get daily? And is there a file somewhere containing the total number of hits since the site's inception?

What a cool thought...

Re:1 Gig on web servers? (3)

gavinhall (33) | more than 14 years ago | (#1064112)

Posted by BSD-Pat:

The web servers are running mod_perl, each process takes up alot of RAM (hrrrm...something to streamline with mod_perl/slash?)

So as a result, the machines that need the most amount of RAM are the webservers and MySQL machines.

essentially we need enough RAM to run up to the MaxClients set in apache *and* have file cache ;)


Re:Cachedot? (1)

Mycroft-X (11435) | more than 14 years ago | (#1064113)

Stupid nonfunctioning ill-previewing HTML code...

That is supposed to be (Startrek.version <= ST:TNG)

Now watch this one not work either...

Re:dual nics? (1)

The Dev (19322) | more than 14 years ago | (#1064114)

Most likely for out-of-band management network.
Possibly Exodous's, probably used for management(duh) and backups. The system should also have another set of nic's for the owners out-of-band management network, connected via private line (or dialup) to their office/noc/whatever.

Re:NFS Question (1)

kirwin (71594) | more than 14 years ago | (#1064115)

I would assume that the database itself rests on the database server, and the data files are NFS mounted from some sort of NAS server.

I do this. It works very well. The database connects to the NAS server (a Network Appliance F760) via NFS over 1000baseSX ethernet. CPU usage is offloaded to the NIC processor for accessing all of the data files. It frees up *a lot* of CPU usage. NAS servers have great hot-swap backplanes, tough as nails drives, and remarkable stability. Most vendors throw in nifty little features such as drive mirroring, integrated backup, etc.

Quick question... (2)

fReNeTiK (31070) | more than 14 years ago | (#1064116)

For the MySQL box you list:

Red Hat Linux 6.2 (final release + tweaks)

Is that only a reference to tweaking the max number of processes in the kernel or did you apply some alien-technology-from-outer-space-experimental kernel patches?

If so, details pleeze!

Re:For the record -- "Exodus"?? (2)

slaker (53818) | more than 14 years ago | (#1064117)

Exodus is a colocation facility.
Basically, they have connectivity coming out their ears (one or more OC48s to the other Exodus facilities, OC3s to "most" of backbone carriers, at each site) the in a number of buildings that offer things like highly secured access (full time security staff, kevlar lined walls, 2" thick electronic-keyed steel doors), mutliple power circuits and UPS, extremely regulated environmental conditions. You have to sign an NDA to even walk in their building.

Basically, the Exodus facilities are going to be better than anything a single company could afford to do on their own.

They host sites like Ebay and (at the one I was at) all the stuff.

To me the best part was that everything in their
vending machines only cost a dime. =)

Re:Why Debian/RH? (2)

Devil Ducky (48672) | more than 14 years ago | (#1064118)

I was more wondering why you would want to switch systems.

Here at work I'm setting up a Linux server for each of our locations to use for sendmail and apache. Each of our machines is using Red Hat 6.2 (final) because Red Hat was easier for me to explain to my non-linux enabled co-workers.

But why would you want two of your systems to be Debian and one to be Red Hat? Wouldn't that just make it harder to figure out when something goes wrong with one of them. Personally I like to be able to look at a machine that works and compare in case I miss something.

Is Red Hat better prepared for mySQL than Debian?

Devil Ducky

Re:Seems faster. (1)

CmdrPinkTaco (63423) | more than 14 years ago | (#1064119)

Imagine a beo... oh nevermind.

Imagine a wolfpack cluster of....oh I think I need to shuddup too.

Re:dual nics? (1)

kwalker (1383) | more than 14 years ago | (#1064120)

Considering the rest of The Matrix's setup, the dual NICs are probably for bandwidth. One NIC (100mb/sec) for the connection to the Internet and the other NIC (100mb/sec) to share between themselves so no matter how full the Internet side gets, the servers can still swap data between themselves.

Re:NFS Question (1)

romerom (151264) | more than 14 years ago | (#1064121)

i BELIEVE it would be so that the SQL server only had to publish the data ONCE instead of posting to each individual web server... the web servers get the information from the SQL servers off of the NFS server and everything works out great.

Re:For the record -- "Exodus"?? (4)

Reality Master 101 (179095) | more than 14 years ago | (#1064122)

Others have said it well, but I'll add this: Exodus hosts Yahoo. 'Nuff said.


Important Question... (1)

CokeBear (16811) | more than 14 years ago | (#1064123)

How secure is the location where the servers are located?
Are they in a nuke-proof bunker?
You might think this is a joke, but what if someone posts government secrets to Slashdot and the Secret Service (or whoever...) decides to pull a Steve Jackson on Andover HQ?
How would Slashdot stay up? Is there a mirror at another location ready to go up at a moment's notice? Would any of the authors not arrested post notice to
I must sound really paranoid, but this is the kind of contingency plan that needs to be considered if Slashdot is to become the flag bearer for Free Speech around the world.

Re:What is all the hardware needed for? (1)

pe1rxq (141710) | more than 14 years ago | (#1064124)

Actually a 386 can fill an 10meg (bits, not bytes) pipe. I tried it. It also depends a lot on the type of nic and protocol you use. You can't put any real data over it however since it can't do anything usefull anymore....


Re:shadouts? (1)

FattMattP (86246) | more than 14 years ago | (#1064125)

This isn't flamebiat. The correct term is shoutouts.

Re:Why Debian/RH? (1)

mansemat (65131) | more than 14 years ago | (#1064126)

1) RH Runs MySQL.
2) The RPMs for Apache just don't cut it with a high volume site. You need to optimize Apache for whatever it is you really need. You include the modules you need and exclude the one's you don't. Plus I'm sure they have a few custom modules they need to include.

Re:Thank god! (1)

jamesm (31089) | more than 14 years ago | (#1064127)

He did. Do you think that he tarred up and released the first Linux kernel the minute that it worked on his machine? I don't think so. Release Early is important, but so also is releasing code when it's ready - and if you KNOW that your code needs improvement before it's ready for consumption, why would you release an unpolished, buggy, incomplete package? Sometimes a little bit of work would push it above the usability threshold, below which people probably wouldn't take a second look at it.

Re:1 Gig on web servers? (1)

The Dev (19322) | more than 14 years ago | (#1064128)

Oops, sorry. I forgot about mod_perl.
That's one nice thing about coding in C or
Java Servlets, no httpd bloat, but that's another topic :)

Suggestions for the future... (2)

Ron Harwood (136613) | more than 14 years ago | (#1064129)

1. Cluster the SQL box for high availability and fail-over.
2. Switch from NFS to SMB - even the apache site recommends this for speed.
2a. Get rid of NFS and just sync all your web-servers from one server - hence having local copies of all the code.
3. Look into having local instances of SQL running on the web-servers - read-only copies that are replicated from the main DB... then the central DB would only be used for write (aka comments and postings...)

Just my less than humble ideas...

Lies (4)

Anonymous Coward | more than 14 years ago | (#1064130)

You're using Win2000, IIS, and Active Server Pages. We all know it. Quit making stuff up.

Re:Interesting Setup (2)

kirwin (71594) | more than 14 years ago | (#1064131)

Slashdot is the 2779th most visited website on the internet, according to pcdataonline []
Check it [] out. Only 2736 places to go before they tie with

Of course yahoo [] is #1. Google is better though ;(

Re:Why did you choose MySQL? (2)

LizardKing (5245) | more than 14 years ago | (#1064132)

I imagine they chose MySQL because:

1) It's very simple to install
2) It's very fast
3) PostgreSQL only really came of age recently

The lack of features in MySQL are only now biting them, but rather than switch to PostgreSQL they're funding the MySQL guys to add those features. Rather a nice way of rewarding them for producing the software at the heart of Slashdot, and one that will benefit others who may find themselves in a similar position one day.

Chris Wareham

Re:Why spend all that $ to fix MySQL? (1)

valis (947) | more than 14 years ago | (#1064133)

Well PostgeSQL is (technically speaking) slower than death. The performance advantages of MySQL outweigh the (obvious) feature advantages of PostgreSQL. At least for the stuff I do...

Re:Total cost (1)

austad (22163) | more than 14 years ago | (#1064134)

Depending on their vendor, and what blades they bought with it, it would be around $85-$110k.

Network Topology (4)

howard_wwtg (175831) | more than 14 years ago | (#1064135)

I don't know about the other slashdottes, but I for one would love to see how the Slashdot network is configured topologically.

NFS: What are you using it for? (1)

danbeck (5706) | more than 14 years ago | (#1064136)

We are considering using NFS as a backend storage facility for our web cluster nodes, but we are concerned with the performance of that type of setup as opposed to replicated data on the local web server nodes. Are you using the NFS server for this purpose?

Re:Why spend all that $ to fix MySQL? (2)

MattJ (14813) | more than 14 years ago | (#1064140)

PostgreSQL certainly has its good points. I would guess one reason they're sticking with MySQL would be speed. MySQL is significantly faster than PostgreSQL, and everything else. Although recently MS SQL Server 7 has caught up in some kinds of operations.

MySQL is extremely fast if you are doing mostly SELECTs (versus a lot of INSERTs and UPDATEs), and that's what Slashdot tends to do. The site spends much more time retrieving messages (and displaying them) than it does storing new messages. Even after the MySQL team spent lots of time tweaking benchmarks to make *PostgreSQL* look better (!), MySQL continues to be much faster, especially in SELECTs.

But as the friendly MySQL [] team will tell you (on the excellent, excellent mailing list), there are many scenarios where PostgreSQL is the best choice. And hey, there are situations where industrial-size Oracle is the best choice (although it will cost you 5-7 figures).

Re:Why spend all that $ to fix MySQL? (3)

hbruijn (30472) | more than 14 years ago | (#1064142)

From the MySQ L Documentation [] :

PostgreSQL has some more advanced features like user-defined types, triggers, rules and some transaction support. However, PostgreSQL lacks many of the standard types and functions from ANSI SQL and ODBC. See the database comparison with crash-me [] for a complete list of limits and which types and functions are supported or unsupported.

Normally, PostgreSQL is a magnitude slower than MySQL. This is due largely to their transactions system which is not as sofisticated as Berkely DB's. In MySQL you can decide per table if you want the table to be fast or take the speed penalty of making it transaction safe. If you really the rich type system PostgreSQL offers and you can afford the speed penalty, you should take a look at PostgreSQL.

I can imagine with the loads on slashdot on order of magnitude speed difference makes a world of difference. Second there's already an complete MySQL code base for slash, so you get stuck with "industrial inertia".

Re:Interesting Setup (1)

nuprin24 (144287) | more than 14 years ago | (#1064144)

It's not overkill if you want it to scale up. As with all things internet, growth tends to follow an exponential curve. Rather than get by with what you have and have to keep adding to it, you buy whatever you can afford and hope you don't have to upgrade every week to meet demand.

Personally, I would have gone for Sun and Oracle on a E3500-E6500 for a DB server, pricey, but well worth the reduced headaches. but I suppose this *IS* Slashdot... you gotta eat your own dog food.

Re:dual nics? (1)

CybeRHiDe (66849) | more than 14 years ago | (#1064146)

I would think that one of the nics would be for internet traffic, and the other for NFS (storage) traffic.

So... how much traffic? (1)

Djinh (92332) | more than 14 years ago | (#1064149)

So, how much traffic and hits/sec does this setup do?

Re:Why spend all that $ to fix MySQL? (2)

Exanter (2171) | more than 14 years ago | (#1064150)

Why? Because unless they wanted to kill that beefy SQL server, mysql works just fine. Yes, most of us read that nice little article on "Why not mysql?", and lots of people rightly concluded that they guy had a stick up his arse about something, and decided to take it out on the mysql team. Mysql does just fine for slashdot. It's a hell of a lot faster than postgres, and more server friendly too. Remember that for all the hoopla and glitz and glamor, slashdot is a relatively easy setup, not requiring really complex database setups, or "ACID" compliance.

Besides, what's wrong with helping the mysql team? pretty soon, mysql gets most of the features everyones been bitching about (including the replication, thanks Andover!), and it still blows the pants off other dbs in terms of raw speed, and suddenly, the recently much maligned mysql does kick some serious arse.

Re:I have a couple of questions and a comment. (1)

kirwin (71594) | more than 14 years ago | (#1064152)

Gigabit ethernet for network attached storage, which is most likely how the database server is set up. See my explanation above

Re:SQL Server (1)

Eponymous, Showered (73818) | more than 14 years ago | (#1064153)

I assumed when they talked about working with and funding the MySQL team to add "rollback capabilities," they were looking at full ACID support. Maybe I'm on ACID, though.

Nothing to do with data modeling (2)

daviddennis (10926) | more than 14 years ago | (#1064154)

I haven't looked at Postgres SQL specifically, but the mySQL API was designed to look just like the mSQL API. Most likely, they would have to change a lot of stuff around to make it compatible with any other database besides mSQL, and of course mSQL has been an inferior product to mySQL for quite some time now.

That's the most likely reason he did it; that's also why I've been sticking to mySQL. I don't have time to learn something new when I'm asked to take on 3,000 new projects every day :-).


Load More Comments
Slashdot Login

Need an Account?

Forgot your password?