×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Linux Clustering Hardware?

Cliff posted more than 8 years ago | from the expensive-highway-or-the-cheap-road dept.

Hardware Hacking 201

Kanagawa asks: "The last few years have seen a slew of new Linux clustering and blade-server hardware solutions; they're being offered by the likes of HP, IBM, and smaller companies like Penguin Computing. We've been using the HP gear for awhile with mixed results and have decided to re-evaluate other solutions. We can't help but notice that the Google gear in our co-lo appears to be off-the-shelf motherboards screwed to aluminum shelves. So, it's making us curious. What have Slashdot's famed readers found to be reliable and cost effective for clustering? Do you prefer blade server forms, white-box rack mount units, or high-end multi-CPU servers? And, most importantly, what do you look for when making a choice?"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

201 comments

Imagine a beowulf cluster of linux clustering (-1, Offtopic)

Anonymous Coward | more than 8 years ago | (#12600709)

Then repeat recursively.

Depends (3, Insightful)

Anonymous Coward | more than 8 years ago | (#12600714)

Depends on

(a) Your cost budget

(b) Your work requirement: A Search engine is different from a weather forecast center.

(c) Cost of ownership which includes maintenance etc

Re:Depends (0)

Anonymous Coward | more than 8 years ago | (#12600810)

That about sums it up.

Throw in Cost of Electricity in cost of ownership, and it becomes a math problem.

If they know if they need high cpu demands, memory demands or disk demands, they can design their system around those requirements.

Try a Green Solution (-1)

Anonymous Coward | more than 8 years ago | (#12600829)

The Silicon Valley Toxics Coalition [svtc.org] regularly assesses computer companies to determine whether they have good environment policies and respect for workers' rights. Chinese companies (including those in Taiwan) consistently fail on both counts.

Our conscience requires us to avoid buying Chinese computers and other Chinese products. As for the Linux clusters, here is one approach to creating a green clustered computer which is fast but may not be the fastest machine. We willingly sacrifice a little bit of speed in order to promote the environment and workers rights [phrusa.org].

Buy 16 Sony desktops, of which many are actually assembled in the United States of America. Alternatively, buy 16 Fujitsu or Toshiba desktops. Sony, Fujitsu, and Toshiba are committed to protecting the environment. Another alternative is to buy 16 Siemens-Nixdorf desktop.

Then, get a copy of Linux and Beowulf, and presto (!), you have a supercomputer in your office. The total cost should run you about $10,000.

Re:Try a Green Solution (0)

Anonymous Coward | more than 8 years ago | (#12600945)

Damn, how many times have you tried to karma-whore and link that phrusa.org site now? Don't you have a job or something?

Re:Try a Green Solution (0)

Anonymous Coward | more than 8 years ago | (#12601205)

CAN we just ban the parents IP??

please...

Gates Will Not Be Happy (1, Offtopic)

Master of Transhuman (597628) | more than 8 years ago | (#12600716)


when all his three-CPU XBoxes become Linux clusters!

Bwahahahahaha!!!

Re:Gates Will Not Be Happy (1)

FuturePastNow (836765) | more than 8 years ago | (#12600792)

Actually, he probably will. Wouldn't you want your console to be known as a supercomputer? It's all publicity. And a guranteed free ad on Slashdot.

Re:Gates Will Not Be Happy (0)

Anonymous Coward | more than 8 years ago | (#12600816)

Not if its serving porn..

Re:Gates Will Not Be Happy (1)

rpozz (249652) | more than 8 years ago | (#12600963)

Not if they're making a loss on it (which I imagine they will be). If the XBox 360 does get used by a few companies as a Linux cluster - which isn't out of the question, then it'll be at MS's expense.

And how was the grandparent post 'Offtopic'?

Re:Gates Will Not Be Happy (1)

Charles Jo (862028) | more than 8 years ago | (#12601083)

I believe that Gates will be happy if hackers repurposed Xbox360s for clustering for the following reasons: 1. Increase in unit sales for whatever reason will appear like Xbox360 is doing much better than otherwise. 2. MS can observe what all the interesting stuff that the FOSS community does with Xbox360 and then copy and replace the FOSS code with MS code, package it with expensive colorful labels and sell it. 3. As a previous poster noted, it's free advertisement for MS. CharlesJo360 http://www.charlesjo.com/newsletterissue?newslette rIssueEntityId=304 [charlesjo.com]

Re:Gates Will Not Be Happy (1)

rpozz (249652) | more than 8 years ago | (#12601434)

While you make some very good points there, I'd imagine that the idea of Linux (the most likely OS for the job) running on it would be quite embarrassing for them. I would doubt that it would be doing anything different to normal clustering software too - it would merely be a few changes in the back-end to make it 'appear' as a normal computer to applications.

Re:Gates Will Not Be Happy (0)

Anonymous Coward | more than 8 years ago | (#12601017)

Can you say propritary CPU - that means no published information on its internal workings no publicly available systems level compiler. This may well be the first computer that will have no Linux for it. That may well be Bills reason for designing this xbox around their own CPUs. Do not be too supprised if in the next year or so MicroSoft starts building its own computers on this or simular propritary CPU as Mr. Bill is all for as much lockin as he can get.

Personally (0)

EpsCylonB (307640) | more than 8 years ago | (#12600720)

I always choose the one with go faster stripes on the side.

Re:Personally (1)

cranos (592602) | more than 8 years ago | (#12600794)

Don't forget the fluffy dice and you might want to add some hydraulics to the case lid, so that it can bounce up and down when ever a "Hot Babe tm" walks past.

Re:Personally (0)

Anonymous Coward | more than 8 years ago | (#12600855)

I always choose the one with go faster stripes on the side.

You joke, but Akamai, before a round of VC funding did apparently get custom plastic with blue LEDs to make their system look extra special to the VCs.

XServe (1, Troll)

bonch (38532) | more than 8 years ago | (#12600727)

Probably a lot more expensive than you're considering, but at my new job, we have two XServes serving data in the back room. They were incredibly easy to set up and administer, and they are FAST. And best of all, they're UNIX.

However, we also have a low-end PC Linux Gentoo cluster for some extra low-end processing, mostly for when we get a special task and don't want to have the XServes do it. But I'm in love with the XServes. To those who say Apple isn't targeting the enterprise, look no further.

Re:XServe (-1, Troll)

Anonymous Coward | more than 8 years ago | (#12600786)

Yeah, with Linux on them, I bet they're not totally bad...

Re:XServe (0)

Anonymous Coward | more than 8 years ago | (#12600789)

MacOS X Server is not UNIX, it's based on a UNIX-like OS.

wow ... (-1, Offtopic)

Anonymous Coward | more than 8 years ago | (#12600817)

yuor gay....

Re:XServe (4, Interesting)

jschottm (317343) | more than 8 years ago | (#12600988)

To those who say Apple isn't targeting the enterprise, look no further.

Let me know when they stop trying to force their iPod updater (you know, the one that breaks Real's compatability DRM software) onto my servers. No matter how many times you put that update in the "Never update this" category, it shows back up the next time you run Software Update. Until they stop trying to play childish games on my production servers, I'll not consider them ready for the enterprise.

Re:XServe (3, Interesting)

Twid (67847) | more than 8 years ago | (#12601030)

Tiger Server lets you run your own Software Update Server [apple.com], which would solve this problem for you. You run a central update server, point all your servers and clients at that, and then you can approve or disapprove each update before it goes out.

Re:XServe (-1, Troll)

Anonymous Coward | more than 8 years ago | (#12601134)

Like most Apple zealots, you're completely missing the point here. Why should he have to run his own software update server to circumvent Apple's childishness? Quite a waste of resources, don't you think?

Re:XServe (3, Interesting)

NitsujTPU (19263) | more than 8 years ago | (#12601173)

Are you running iTunes on your production servers? Can't you just uninstall iTunes and be fine?

Why would a system configured as a fileserver have that software on it to begin with? Is Apple's apt tool so bad that it tries to patch software that hasn't been installed?

Green Solution Instead of the Fastest Solution (-1, Troll)

Anonymous Coward | more than 8 years ago | (#12600763)

The Silicon Valley Toxics Coalition [svtc.org] regularly assesses computer companies to determine whether they have good environment policies and respect for workers' rights. Chinese companies (including those in Taiwan) consistently fail on both counts.

Our conscience requires us to avoid buying Chinese computers and other Chinese products. As for the Linux clusters, here is one approach to creating a green clustered computer which is fast but may not be the fastest machine. We willingly sacrifice a little bit of speed in order to promote the environment and workers rights [phrusa.org].

Buy 16 Sony desktops, of which many are actually assembled in the United States of America. Alternatively, buy 16 Fujitsu or Toshiba desktops. Sony, Fujitsu, and Toshiba are committed to protecting the environment. Another alternative is to buy 16 Siemens-Nixdorf desktop.

Then, get a copy of Linux and Beowulf, and presto (!), you have a supercomputer in your office. The total cost should run you about $10,000.

Dual Opteron 1U rack units.... (5, Interesting)

Fallen Kell (165468) | more than 8 years ago | (#12600770)

For the size and performance, they are hard to beat. A dual opteron setup in a 1U rack case is a very powerful setup in and of itself. The bonus of using off the shelf components with no need for proprietary hardware or software also make them very affordable. The added bonus is that you can simply get the parts from regular retailers for replacement.

Re:Dual Opteron 1U rack units.... (4, Informative)

FuturePastNow (836765) | more than 8 years ago | (#12600822)

It's no dual Opteron server, but this Dan's Data [dansdata.com] article reviews what are probably the cheapest 1U servers you can buy. Definitely something to consider of you're going for cheap.

Re:Dual Opteron 1U rack units.... (-1, Troll)

Anonymous Coward | more than 8 years ago | (#12600837)

Don't buy that.

AMD cpus are made in china and have bugs!!!!

Re:Dual Opteron 1U rack units.... (2, Informative)

imsabbel (611519) | more than 8 years ago | (#12600894)

Not to mention that if you need more power, you can drop in the new dual core opterons without need for anything but eventually a bios upgrade.
(not to mention that the dual core opterons acutally consume less power than some of the early steppings of the single core ones)

Re:Dual Opteron 1U rack units.... (0)

Anonymous Coward | more than 8 years ago | (#12600921)

Price/performance I'll give you. But size/performance, no way.

IBM has some blades that can fit 14 dual opteron servers with a gig ethernet switch, redundant power, kvm in a 7U footprint.

A little pricy, but for easily deploying and managing large numbers of servers, it is hard to beat.

Ammasso (3, Interesting)

LordNimon (85072) | more than 8 years ago | (#12600796)

Ammasso [slashdot.org] is a startup that makes iWarp [wikipedia.org]-based RDMA [wikipedia.org] hardware that runs over gigabit ethernet. Their technology is like Infiband, but much cheaper and almost as fast. Their drivers and libraries also provide MPI [wikipedia.org] and DAPL [sourceforge.net] support. The only support Linux (all 2.4 and 2.6 kernels) and they're way ahead of their competition in terms of performance, product availability, and support. Once you've decided on the servers, I strongly recommend you use Ammasso's hardware for the interconnects. Your hardware vendor may even bundle it with their systems - be sure to ask about that.

Re:Ammasso (0)

Anonymous Coward | more than 8 years ago | (#12600872)

Ammasso's hardware IMHO is not yet ready for prime-time roll out of a large cluster from a maintenance perspective. In any case if you are willing to spend 400$ on a GigE Ammasso card why wouldn't you consider Infiniband.

Again, the most important point is to consider what is your application before you rollout a Linux Cluster. Main Point to consider:

* Requirement of HA
* Fault Tolerance
* Does your application require a large amount of IPC communication or is it an embarassingly parallel application
* Ability to support a large cluster. If you are google and can support open motherboards and set up the labs to do that more power to you. But most organizations can't do that. If you are anyone else buying a blade server or a 1U rackmounted chasis from a major vendor will reduce your longterm TCO. Google doesn't use MPI and have custom applications which have FT/HA features due to which they are able to support low cost devices and deal with failure. Most MPI apps can't deal with failure. Hardware with redundancy is usually a good idea.

Re:Ammasso (1)

jabuzz (182671) | more than 8 years ago | (#12601429)

Is it rubbish nearly as fast. The top speed of Infiband is 30 times faster than gigabit ethernet, and the lowest speed is 2.5 times faster. By the time you have paid for the Ammasso cards you might as well have gone with one of the other higher speed low latency interconnects.

Check out Xserve (5, Informative)

Twid (67847) | more than 8 years ago | (#12600798)

At Apple we sell the Xserve Cluster node [apple.com] which has been used for clusters as large as the 1,566 node COLSA [apple.com] cluster. We also sell it in small turn-key [apple.com] configurations.

Probably the most interesting news lately for OS X for HPC is the inclusion of Xgrid with Tiger. Xgrid is a low-end job manager that comes built-in to Tiger Client. Tiger Server can then control up to 128 nodes in a folding@home job management style. I've seen a lot of interest from customers in using this instead of tools like Sun Grid Engine for small clusters.

You can find some good technical info on running clustered code on OS X here [sourceforge.net].

The advantage of the Xserve is that it is cooler and uses less power than either Itanium or Xeon, and it's usually better than Opteron depending on the system. In my experience almost all C or Fortran code runs fine on OS X straight over from Linux with minimal tweaking. The disadvantage is that you only have one choice: a dual-CPU 1U box - no blades, no 8-CPU boxes, just the one server model. So if your clustered app needs lots of CPU power it might not be a good fit. For most sci-tech apps, though, it works fine.

If you're against OSX but still like the Xserve, Yellow Dog makes an HPC-specific [terrasoftsolutions.com] Linux distro for the Xserve.

Re:Check out Xserve (0, Interesting)

Anonymous Coward | more than 8 years ago | (#12600856)

Try getting parts for XServes after a couple of years. Want parts for a XServe G4 from yesteryear? Chances are you are gonna be fucked. We couldnt even source a drive caddy for one. Buy a proper server thats gonna be supported for longer than its trendyness.

Re:Check out Xserve (2, Informative)

Twid (67847) | more than 8 years ago | (#12600932)

That's simply not true. We stock parts for all Apple equipment for at least seven years. The drive caddys for the old Xserve G4 are exactly the same as what we ship in the current Xserve RAID, so not only is the drive caddy still available, it's also still in use [apple.com].

Actually, the cheapest way to go would be to buy the 250GB ADM, apple part number M9356G/A, and pull out the 250GB drive and use it somewhere else (or just use that drive). Service parts tend to be pricey (like everyone else).

Re:Check out Xserve (0)

Anonymous Coward | more than 8 years ago | (#12601148)

Wow, great service Apple!! You should be real freakin' proud. If Dell was doing this you'd be crying bloody murder.

Teraflopping Bogus Science (0)

Anonymous Coward | more than 8 years ago | (#12601089)

Star Wars redux: "Medeiros' team and RDECOM will also use the supercluster to improve the design of intercept missiles that defend against attack."

Re:Check out Xserve (0, Offtopic)

IANAAC (692242) | more than 8 years ago | (#12601636)

Good information, however, this statement:

We also sell it in small turn-key configurations.

only makes it look like an ad.

Granted, I don't begrudge you for trying to drum up Apple business in a Linux thread, but - and I supsect I'm not alone in thinking this - you're probably driving away some potential business using the aformentioned tactic.

Dual Core Opteron Blades (5, Insightful)

municio (465355) | more than 8 years ago | (#12600806)

At the current time I would choose blades based on dual core Opterons form many reasons. Some of the main ones are:

- Price
- Software availability
- Power consumption
- Density

Brand depends on what your company is confortable with. Some companies would want to have the backing of IBM, SUN or HP. Others will be quite satisfied with in house built blades. This days it's quite easy to build your own blade, some mother boards builders take care of almost all components and complexity (for example Tyan). But again, maybe the PHBs at your gig will run for the hills if you mention the word motherboard alone.

Re:Dual Core Opteron Blades (1)

PornMaster (749461) | more than 8 years ago | (#12601026)

Blades rarely make sense. Most blades available are not interconnected in any meaningful way, so you're really looking at independent systems, just of a small form factor. Why use blades? Well, they're space efficient. Great, unless you're actually using a lot of them, and don't have too much space. Because if you don't have too much space, you probably don't have too much cooling. If you don't have too much cooling, then you've totally killed your argument for blades based upon space. A rack full of HP BL40p blades puts out something on the order of 53k BTU. That's one rack. If you're hosting your stuff in a colo, then if the data center doesn't give a shit that you're pumping out that much heat from one rack, then they're not concerned about their other customers and the "commitment" they made to them. No data centers I know of are built to cool that much heat density... and if someone has the money to build space that can be cooled to that extent, then they can just make the damn room bigger and spread out the heat. Blades rarely make any sense.

Re:Dual Core Opteron Blades (0)

Anonymous Coward | more than 8 years ago | (#12601290)

The motherboards for dual core Opterons don't work yet, despite the vendors claims. The BIOS's aren't stable, and they're still trying to get the bugs out of their teeth from crossing the finish line first with no clothes on to reduce the weight on their vehicles.

The results for numerous vendors of the crash and burn as their unprotected naked bodies landed on the pavement because they also left the brakes off their motorcycles to save weight is left to the reader's imagination. But the resulting body bits are also scattered all over the poor choices and integrations of components on the first round of dual-core motherboards. Then have some twit stuff it with 16 Gig of RAM sold them by their cousin Guido on a street corner to save a few bucks, and watch the systems crash.

Technologies that new are basically using us consumers as beta testers: don't bother with dual core for at least six more months if you want your cluster to be stable.

Re:Dual Core Opteron Blades (1)

jcampbelly (885881) | more than 8 years ago | (#12601561)

Cost-effectiveness is often the biggest issue for a lot of companies who need the power but can't gain that TCO edge from buying in massive bulk. Many can't afford the convenient and complete offerings of an IBM/Intel or HP blade whose components alone can cost upwards of $10,000 with 5-10x that in software. A more practical solution would be an in-house project, but you're still paying off-the-shelf prices for something you may be buying in bulk (unless you've got some distributor channel to work with). In this case, the white-box manufacturer is what you need. There are cheap blade chassis that can easily be fitted with good hardware. I know of a few that won't get you the density of HP or Intel, but it's better than 1U's for cooling and for space. No matter how many systems you will be building, chances are (if you're not going the ol "massive bulk" route) you won't be buying enough hardware to get a decent price per component. That's where white-box companies come in, working with distributor channels to get profitable pricing competitive with solutions from larger vendors. At this point you're left with the warm-fuzzy of a smaller hardware OEM with kingly support (smaller companies often listen to customers more) and you're left to loose your own geeks on their new hardware. That is, of course, if you've got your own geeks. I'd recommend an OpenMosix Gentoo cluster for simpler job-processing work unless you've already got the software. Dual Opterons are the way to go as you can start out with 1 CPU, then in 3-4 years the same hardware should be able to run 8 cores -- that's real, present, available scalability with a side of cost-effectiveness. I work for a white-box custom integrator like this and we make such 1U systems which can also be put into a cheap blade form-factor. This is no advertisement so I won't give a name, but feel free to e-mail me if you think I've made a point or two.

Pictures? (1, Interesting)

Anonymous Coward | more than 8 years ago | (#12600809)

I would love to see a picture of the google hardware...

Re:Pictures? (3, Funny)

Anonymous Coward | more than 8 years ago | (#12600999)

Oh yeah, show me a picture of your rack baby!

IBM zSeries (1, Interesting)

Anonymous Coward | more than 8 years ago | (#12600814)

Definitely worth checking out. It's one bad-ass Linux server -- and probably the only one to offer instruction execution integrity. That's a fancy way of saying 2+2 will always equal 4 on zSeries -- because everything is executed twice and compared at the hardware level -- or it won't execute.

If you need this, you need it bad.

Re:IBM zSeries (0)

Anonymous Coward | more than 8 years ago | (#12600888)

Yeah right. Square peg, round hole.

Re:IBM zSeries (1)

Wesley Felter (138342) | more than 8 years ago | (#12600916)

If you need this, you need it bad.

Conversely, if you don't need it (because you've already decided on a cluster), then you really don't need it.

Read the Google paper ! (5, Insightful)

devitto (230479) | more than 8 years ago | (#12600818)

In the paper, it goes into tedious detail on the architecture and low-level operation of the application. Why do you think it does this? Because it is the application that *totally* depicts the solution, they chose lots of systems because of reliability, they made those systems "desktop class" because they didn't get much extra from using super-MP/MC systems.

It's a great article, I strongely suggest you read properly, and do what they said they did - evaluate need against what's available.

Re:Read the Google paper ! (0)

Anonymous Coward | more than 8 years ago | (#12600867)

Read *what* google paper. There are many. There are even a few discussing hardware architecture.

Quick FYI re: Google gear (0)

Anonymous Coward | more than 8 years ago | (#12601147)

off-the-shelf motherboards

Mostly true, though increasingly Google has commodity vendors producing custom hardware runs (ie - "Google" branded RAM, variant ASUS motherboards with a custom BIOS and no built-in video or sound, etc.).

screwed to aluminum shelves

Nope, no screws. The motherboards are attached to the trays by plastic pop-on/pop-off standoffs and everything else is held together by velcro straps; no screwdrivers or other tools are necessary when swapping parts. The trays can also hang from the front of the racks while parts are being swapped.

Re:Quick FYI re: Google gear (0)

Anonymous Coward | more than 8 years ago | (#12601622)

Does anyone sell such systems?

I've always thought it was silly to be paying for video chips and sound chips when the serial line and ethernet were the only way I interacted with the system.

well... (5, Informative)

croddy (659025) | more than 8 years ago | (#12600820)

at the moment we have a rack with Dell PowerEdge 1750's. They're very nice for our OpenSSI cluster, with the exception of the disk controller. Despite assurances by Dell that the MegaRAID unit is "linux supported", we're now stuck with what's got to be the worst SCSI RAID controller in the history of computing.

we're hoping that upgrading to OpenSSI 1.9 (which uses a 2.6 kernel instead of the 2.4 kernel in the current stable release) will show better disk performance... but... yeah.

It's not that the MegaRAID cards are bad... (3, Insightful)

Ayanami Rei (621112) | more than 8 years ago | (#12601472)

...it's the firmware Dell shoves on them.
It's only designed to hook up with Dell disc arrays and tape drives and everything else can shove it (from their point of view).
Do yourself a favor and skip 'em and just by the cards straight from LSI.

Re:It's not that the MegaRAID cards are bad... (1)

croddy (659025) | more than 8 years ago | (#12601515)

we're even using their disk arrays. i wasn't here when that purchasing decision was made, but man oh man, these things are crap.

Total Overkill (5, Funny)

Anonymous Coward | more than 8 years ago | (#12600838)

We can't help but notice that the Google gear in our co-lo appears to be off-the-shelf motherboards screwed to aluminum shelves.

That would be typical of a prima donna company like Google that's floating in cash from their IPO.

Around here, we don't waste money on fancy designer metals like aluminum. Salvaged wooden shipping palettes work just fine for us; they're free. And screws!? No need to waste resources on high-end fasteners when you can pick up surplus baling wire for less than a penny per foot. A couple of loops of wire and a few twists are all you need to assemble a working server.

The dotcom days are over. There's no reason to throw money around like there's no tomorrow.

Re:Total Overkill (1)

Sjobeck (518934) | more than 8 years ago | (#12601504)

Not to mention the fact that we are fiddling while the planet is about to burst in to flames! Seriously, people, wak up, stop entertaining yourselves to death, which is only a figure of speak, when there are actually people who are starvign to death, not a figure of speech. www.one.org www.data.org www.makepovertyhistory.org

Orion Desktop Clusters (0)

Anonymous Coward | more than 8 years ago | (#12600842)

How about Orion 's [orionmulti.com] Desktop and Deskside clusters? Where else can you get a cluster or 12 or 48-96 CPU's in one machine that you can plug into a normal power socket?

Depends on your CoLo's power & cooling. (0)

Anonymous Coward | more than 8 years ago | (#12600844)

Note that racks full of blades (and even dual-cpu-1U systems) can easily exceed the power and cooling capacity of many colocation center's racks. I'm not convinced blade systems are cost effective anyway; but just wanted to point out that the answer may depend on where you store the systems.

It depends what you are clustering for... (1)

Imposter_of_myself (636697) | more than 8 years ago | (#12600845)

Are you clustering for performance? Are you clustering for high availability? Are you clustering for fault tolerance? There are differnent types of clusters for different applications. We can't tell you what is "best" for clustering without knowing what you are trying to accomplish.

None of the above... (0)

Anonymous Coward | more than 8 years ago | (#12601507)

He, like so many others these days, is clustering for style. It's fashionable to do clusters right now and he doesn't want to be left out.

He's a slave to fashion and that means having the most kick-ass cluster that ever served a Word document! w00t! w00t! w00t!

minitowers!!!! (0)

Anonymous Coward | more than 8 years ago | (#12600887)

We have a farm of freebsd webservers in small tower cases on shelves. We have 10 of those. All servers were built from very cheap components that we got online. I think that each one is something like $300 - $400. The farm serves dynamic pages via php and apache and also does some image manipulation using netpbm tools. Servers talk to the outside via 2 redundant ballancers running freevrrpd. These have the same hardware as webservers. Our application stores a lot data so I have 2 (for redundancy) 2TB raid arrays served via nfs to the webservers. These are in a rack of course, as well as 2 (for redundancy again) Postgres servers. Rackmount servers are built from off the shelf components again. Works great.

No one size fits all answer but here is mine :) (5, Informative)

Anonymous Coward | more than 8 years ago | (#12600899)

My .02 cents worth ...

I build Linux and Apple clusters for biotech, pharma and academic clients. I needed to announce this because clusters designed for lifesci work tend to have different architecture priorities than say clusters used for CFD or weather prediction :) Suffice it to say that bioclusters are rate limited by file I/O issues and are tuned for compute farm style batch computing rather than full on beowulf style parallel processing.

I've used *many* different platforms to address different requirements, scale out plans and physical/environmental constraints.

The best whitebox vendor that I have used is Rackable Systems (http://www.rackable.com/ [rackable.com] . They truly understand cooling and airflow issues, have great 1U half-depth chassis that let you get near blade density with inexpensive mass market server mainboards and they have great DC power distribution kit for larger deployments.

For general purpose 1U "pizza box" style rackmounts I tend to use the Sun V20z's when Opterons are called for but IBM and HP both have great dual-Xeon and dual-AMD 1U platforms. For me the Sun Opterons have tended to have the best price/performance numbers from a "big name" vendor.

Two years ago I was building tons of clusters out of Dell hardware. Now nobody I know is even considering Dell. For me they are no longer on my radar -- their endless pretend games with "considering" AMD based solutions is getting tired and until they start shipping some Opteron based products they not going to be a player of any significant merit.

The best blade systems I have seen are no longer made -- they were the systems from RLX.

What you need to understand about blade servers is that the biggest real savings you get with the added price comes from the reduction in administrative burden and ease of operation. The physical form factor and environmental savings are nice but often not as important as the operational/admin/IT savings.

Because of this, people evaluating blade systems should place a huge priority on the quality of the management, monitoring and provisioning software provided by the blade vendor. This is why RLX blades were better than any other vendor even big players like HP, IBM and Dell.

That said though, the quality of whitebox blade systems is usually pretty bad -- especially concerning how they handle cooling and airflow. I've seen one bad deployment where the blade rack needed 12 inch ducting brought into the base just to force enough cool air into the rack to keep the mainboards from tripping their emergency temp shutdown probes. If forced to choose a blade solution I'd first grade on the quality of the management software and then on the quality of the vendor. I am very comfortable purchasing 1U rackmounts from whitebox vendors but I'd probably not purchase a blade system from one. Interestingly enough I just got a Penguin blade chasssis installed and will be playing with it next week to see how it does.

If you don't have a datacenter, special air conditioning or a dedicated IT staff then I highly recommend checking out OrionMultisystems. They sell 12-node desktop and 96-node deskside clusters that ship from the factory fully integrated and best of all they run off a single 110v electrical. They may not win on pure performance when going head to head against dedicated 1U servers but Orion by far wins the prize for "most amount of compute power you can squeeze out of a single electrical outlet..."

I've written a lot about clustering for bioinformatics and life science. All of my work can be seen online here: http://bioteam.net/dag/ [bioteam.net] -- apologies for the plug but I figure this is pretty darn on-topic.

-chris

Re:No one size fits all answer but here is mine :) (1)

dumbfounder (770681) | more than 8 years ago | (#12601097)

You can get a dell sc1425 with 2x2.8ghz xeons and 1GB ram for $1200, but the cheapest you can get a dual opteron sun for is $2444. Do you really think the opterons are more than twice as fast?

Re:No one size fits all answer but here is mine :) (3, Informative)

chris.dag (22141) | more than 8 years ago | (#12601209)


Depends on the specific code to meet your criteria of "twice as fast"...some apps will be more than twice as fast; some will be slightly faster, equal or in some cases slower.

For more general use cases (at least in my field) I can give a qualified answer of "dual Opteron currently represents the best price/performance ratio for small SMP (2-4 CPUs)".

I've also seen cheaper pricing from Sun than what you mentioned. You are right though in that there is a price difference between xeon vs opteron - whenever I consider a more expensive alternative I tend to have fresh app benchmark data handy to back up the justification.

-Chris

Re:No one size fits all answer but here is mine :) (1)

sysadmn (29788) | more than 8 years ago | (#12601539)

In our experience, the number of cpus per rack is limited by data center cooling. Blades are great to look at, and somewhat easier to provision, but having 60 - 80 cpus in 7U chassis isn't any better than having 60 - 80 dual cpu 1U pizza boxes. Our latest (commercial) cluster is 604 Dell boxes - PESC 1425 for the low-end jobs, 1850's for the heavy lifting (12 GB RAM).

Cheap isnt always the way to go (3, Insightful)

oh_the_humanity (883420) | more than 8 years ago | (#12600920)

When doing clkustering and super computer work. Cheap isnt always the best way to go , if you take into consideration that 5 - 10 % of nodes will either not be functioning correctly or will have some sort of hardware failure. The more you cluster the more man power it takes to repair these nodes. if you buy 1000 $499 colomachines , and 50 of them are failing at any given time, it becomes very time consuming and tedious to keep the cluster going. Spending the extra bucks on high quality hardware , will save you money and head ache in the end. I always use the analogy when talking to older folks who want to get started in computers. spend the extra bucks to get a new machine. The extra money you spend on buying new good equipment , will more than pay for itself in comparison , to the amount of frustration you get from buying old used slow computers. My $.2

Re:Cheap isnt always the way to go (0)

Anonymous Coward | more than 8 years ago | (#12601415)


I'll second that, but I've experienced up to over 30% failure rate.

I will never do whitebox stuff again.

FYI, the vendor was Microway [microway.com].

Re:Cheap isnt always the way to go (1, Informative)

Anonymous Coward | more than 8 years ago | (#12601523)

You are wrong. In *most* cases, cheap is the the way to go (but soo few people realize it).

Short argument: Google choosed the cheapest way, they are happy with it and don't plan to change their mind.

Long argument: What you don't realize is that cheap hardware is so DAMN cheap, that you CAN afford employing the manpower required to maintain your cluster. In other words: instead of spending a lot of money on expensive hardware, buy the cheapest gear (where the price/perf ratio is optimal), and then spend a fraction of the money you saved to employ technicians that will maintain/replace defectuous hardware. Moreover with such cheap hardware, you don't even care about the warranty/support, you could trash it directly without even bothering contacting the manufacturer to get replacement parts ! Nowadays $300 is all you need to get a mobo + AMD Sempron 2800 + 512 MB RAM + 80 GB harddisk + NIC 100 Mbps + PSU. So why spend $3000 on a dual-opteron server when you could buy 10, TEN, fscking white boxes for the same price. The only reason would be that the software cannot be parallelized, or that you have constraints on the space occupied by your clusters (that's why I said "in *most* cases cheap is the way to go"). But then even in thoses cases you could maybe spend the money to rent more racks/space, and then you would save money by buying those white boxes.

Re:Cheap isnt always the way to go (0)

Anonymous Coward | more than 8 years ago | (#12601586)

I forgot to say that, of course, one of the key point to successfully run a white box cluster is: design the software so that hardware failure is the norm instead of the exception. A cluster with 50% working nodes should run as well as a cluster with 100% working nodes.

One of the poster said "I will never do whitebox stuff again." I guess the reason is that the sofware he was running was not designed with hardware failure in mind...

Re:Cheap isnt always the way to go (1)

IANAAC (692242) | more than 8 years ago | (#12601689)

if you buy 1000 $499 colomachines , and 50 of them are failing at any given time, it becomes very time consuming and tedious to keep the cluster going.
actually, the same monitoring and swapout/repair of hardware is the same whether it's cheap hardware or expensive hardware. I've had a fair bit of hardware failures on an IBM cluster. Once monitoring catches it and notifies me, I get the new hardware in place and use xcat and kickstart to push everything back out to the replaced node(s). The most time consuming part is waiting for the hardware to arrive. Xcat/kickstart provide a decent toolset to automate everything after the hardware arrives.

We do nice house built clusters (1)

nikkoslack (739901) | more than 8 years ago | (#12601018)

We typically build clusters that are NOT destined for a fancy, high end data centers, so 110 volts and a less dense configuration is essential. We build them into the skinniest mid towers we can, with 2 80mm fans in front and two fans in back. we use the baker's rack method and a hot isle/cold isle arrangement of racks. It's hard to find desktop cases without crappy fans in the sides, which are no good to us. Currently, the ASUS A7V400-MX and SEMPRON 3000+, 1GB DDR400 ram, and 120 gig ide hd is the choice. OnBoard NIC and video are essential to clusters, IMO. We'll be moving to SATA drives and gb ethernet onboard soon, but IDE's are dirt cheap right now. Coincidently, this is the same white-box web server we usually build.

Obviously (4, Funny)

iamdrscience (541136) | more than 8 years ago | (#12601040)

Isn't it obvious that the best technology is blade servers? I mean, c'mon fucking BLADE servers! It's far and away got the coolest name of any of them. The only way you could beat them would be if some company came out with something cooler like ninja star servers, now that would be awesome.

Re:Obviously (0)

Anonymous Coward | more than 8 years ago | (#12601779)

and by awesome, you mean totally sweet.

frist psoT (-1, Redundant)

Anonymous Coward | more than 8 years ago | (#12601041)

is dyIng.Things a BSD box (a PIII Than this BSD box, 3ulogies to BSD's

It really depends on use. (1)

the_mutha (177709) | more than 8 years ago | (#12601063)

I think it really depends on use and in your facility. For example:
  • Do you host your servers at your own facility or do you host them in a data-center where rack space is a premium?
  • Is your application CPU intensive, disk intensive, memory intensive (i.e. scientific application, web servers, database servers, etc.)


We run a very large website. We have a 1U dual Athlon MP box for administration and log processing, and a 2U dual Xeon box with a 6 disk RAID 10 solution for apache + mysql. Its a great solution to us because we run things in a data center co-lo where space is a premium. Both servers use off-the-shelf components and are easy to upgrade and maintain.

If we were running some scientific application and had our own location and good network connection (for example, a university), then I suppose a cluster of standard hardware components like how Google does it would be perfect. If you were doing the same CPU-intensive stuff, but at a co-lo facility, then perhaps a blade solution would be best... it really all depends - you have to look at what your main needs and constraints are, and look at what is available and what will best suit your needs. There is no way to give a standard answer.

Re:It really depends on use. (1)

Sjobeck (518934) | more than 8 years ago | (#12601443)

Decent comments, but ... I do not think that two servers (one admin & one node) really constitutes a cluster, and if it did, it certaintly is not what the poster asked about.

Mobos on Ikea shelves (4, Interesting)

astrojetsonjr (601602) | more than 8 years ago | (#12601101)

Currently 65 (1 master, 64 nodes) of AMD Mobos on Ikea shelves. Cheap, easy to swap out, good air flow around the hardware. The shelves are wood, so everything just sits on them. It would be nice to find power supplies with extra connections to power more than one system.

What to look for? Thats simple: (0)

Anonymous Coward | more than 8 years ago | (#12601178)

Look for the "ULTRAsparc Driven" logo on the front .. and you've found an ideal base for your cluster...

-GenTimJS

Not everyone can be Google (2, Insightful)

Anthony Liguori (820979) | more than 8 years ago | (#12601191)

It's tempting to just go buy a bunch of motherboards on ebay and some bread racks to build your cluster. It's certainly the cheapest and most flexible approach.

However, it takes a special type of people to manage that kind of hardware. You have to deal with a high amount of failure, you have to be extra careful to avoid static problems, you've got to really think through how your going to wire things.

On the other hand, if you get something like a IBM BladeCenter, you have a very similar solution that may cost a little more but is significantly more reliable. More importantly, blades are just as robust as a normal server. You don't have to worry about your PHB not grounding himself properly when he goes to look at what you've setup.

I expect blades are going to be the standard form factor in the future. It just makes sense to centralize all of the cooling, power, and IO devices.

SunFire Servers (4, Informative)

PhunkySchtuff (208108) | more than 8 years ago | (#12601292)

SunFire v20z or v40z Servers.
http://www.sun.com/servers/entry/v20z/index.jsp [sun.com]
http://www.sun.com/servers/entry/v40z/index.jsp [sun.com]
They're the entry-level servers from Sun, so they have great support. They're on the WHQL List, so Windows XP, 2003 Server and the forthcoming 64-bit versions all run fine.
They also run Linux quite well, and as if that wasn't enough, they all scream along with Solaris installed.
The v20z is a 1 or 2 way Opteron box, in a 1RU case. the v40z is a two or for CPU box that is available with single or dual core Opterons.
Plus, they're one of the cheapest, if not the cheapest, Tier 1 Opteron servers on the market.

Re:SunFire Servers (3, Informative)

eryanv (719447) | more than 8 years ago | (#12601737)

The Sun 1U servers are great products for clusters. Last year our research group purchased around 72 v60x's (dual Xeons) and they've never given us a problem. Sun doesn't sell this model anymore as they've pretty much dropped Intel for AMD, but they run Linux with ClusterMatic just fine. Not only that, but Sun has the best techincal support I've ever encountered, I don't mind when we do need to call them up to fix something.
Besides that, we also have several clusters of midsized towers from a local company, which have the benefit of being able to be used as desktop machines when the cluster gets replaced.

Great story re w/ build your own IBM cluster (4, Interesting)

MilesParker (572840) | more than 8 years ago | (#12601729)

We wanted to set up a small 4-8 node cluster mostly for testing and as a compute resource. For various political reasons we were looking at an IBM solution. At my uirging we went for dual Opterons in the 1U format. And the price seemed right. Here's where it gets wierd *after* the OBM sales people step in. Going thourgh it peice by piece I thought I could put a decent system together - with our substantial IBM discount -- for $14k. By the time we got the quote with all of the crap they thought we needed it was 34k! Just to give the flavor, the rack and assorted pieces was 4k. But thats not the funny part. We were like, "well for this much money, we assume you are putting it together for us." "Um no...didn't you see the services quote that went along with this?" We hadn't -- with the services/support quote came in at $60k! So at this point we asked, can't we just buy the individual pieces we need and put it together ourselves. "Well, yes, but then it won't be an IBM e1350 cluster 'solution'..." "Yea, well, we don't really care what its called, it'll be just as fast and 75% cheaper..." At that time they were getting rid of their 325 servers for way cheap and we actually put that system together for as cheap as a whitebox and probably as cheap as if we'd tried to put it together ourselves. The moral I guess is that if you have to deal with the big vendors, have a very sharp pencil handy!

Clarification.. (2, Funny)

MilesParker (572840) | more than 8 years ago | (#12601811)

...after posting I realized that the 34 and 60k quotes were before discount (I think). SO the actual price descrepncy was "only" 62%.
Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...