Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: Clusters On the Cheap?

samzenpus posted about 3 years ago | from the discount-computing dept.

Hardware 264

First time accepted submitter serviscope_minor writes "A friend of mine has recently started a research group. As usual with these things, she is on a shoestring budget and has computational demands. The computational task is very parallel (but implementing it on GPUs is an open research problem and not the topic of research), and very CPU bound. Can slashdotters advise on a practical way of getting really high bang for buck? The budget is about £4000 (excluding VAT/sales tax), though it is likely that the system will be expanded later. The computers will probably end up running a boring Linux distro and Sun GridEngine to manage batch processing (with home directories shared over NFS)."

cancel ×


Sorry! There are no comments related to the filter you selected.

Beowulf cluster? (1)

Anonymous Coward | about 3 years ago | (#37406578)


Re:Beowulf cluster? (0)

2muchcoffeeman (573484) | about 3 years ago | (#37406896)

Of course this is the topic of the first reply. Of course it is.
Because if it were any other answer it would mean that /. had changed so completely that it had morphed into something unrecognizable as its former self.
Thank you for upholding my faith in humanity. Or, at least, in my fellow /.ers.
(Obvious and therefore obligatory follow-up questions: But does it run Linux? And how many Libraries Of Congress will it have in storage capacity? And can you imagine a Beowulf cluster of these clusters? Oh, wait ... )

Uhm AWS EC2 Cluster Compute (3, Informative)

Anonymous Coward | about 3 years ago | (#37406580)

Why waste money on building a cluster when you can rent the best in the world * by the hour * ?

This (0)

Anonymous Coward | about 3 years ago | (#37406598)

You have a limited budget, so it's more cost effective for you to lease time on someone else's equipment for now.

Re:Uhm AWS EC2 Cluster Compute (5, Informative)

jpedlow (1154099) | about 3 years ago | (#37406606)

AWS EC2 was my response aswell. :)

for raw horsepower on the short - medium term, use AWS []

ec2 should do well for this, imho :)

Re:Uhm AWS EC2 Cluster Compute (0)

Anonymous Coward | about 3 years ago | (#37407064)

The problem with EC2 is that its virtualised - my tests showed that simulations ran about 3 times slower than on bare (equivalent) hardware.

Re:Uhm AWS EC2 Cluster Compute (2)

KiloByte (825081) | about 3 years ago | (#37407144)

From a back-of-the-envelope estimate, I see that AWS gets even with buying your own hardware in three months. Except, you still get to own the gear.

Thus, if you need a week or maybe a month of computation, AWS might be a better option, but for anything above that, forget it. If your needs are more bursty, that shifts the balance towards AWS, but again, you need to estimate what you need.

Re:Uhm AWS EC2 Cluster Compute (0)

Anonymous Coward | about 3 years ago | (#37407268)

are you including electicity costs in that estimate, hardware/network maintenace, HVAC etc.

Re:Uhm AWS EC2 Cluster Compute (2)

Joce640k (829181) | about 3 years ago | (#37407382)

How about setup/installation time. Installing and configuring a whole bunch of machines takes a while.

Re:Uhm AWS EC2 Cluster Compute (1)

hardtofindanick (1105361) | about 3 years ago | (#37407164)

There must be a better way than ec2

Cluster GPU Quadruple Extra Large Instance
22 GB of memory
33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core “Nehalem” architecture)
2 x NVIDIA Tesla “Fermi” M2050 GPUs
1690 GB of instance storage

Personal experience: Disk IO: not dependable, network IO excellent.

4000 British pounds sterling = 6302.8000 US dollars
Assuming for each "experiment" you run 20 instances in parallel for 15 minutes (partial hours count as full hours, so remember to round up even if you use for a minute), you spend 40$ per session.

You get to make 160 experiments, and you are over budget.

Or if you adjust everything optimally, e.g., end the experiment at 59 minute mark, and assuming your "parallelization" uses 5 instances, then you get to make 640 experiments. But real life is far from optimal, especially with those pesky grad students.

Re:Uhm AWS EC2 Cluster Compute (1)

rioki (1328185) | about 3 years ago | (#37407326)

+1 Since this is probably a University, they probably already have a place to put the thing, that is powered and climate controlled. So there are actually "no running" costs. Yes, admins are "free" too. And looking at the average research project, there is always follow up research and if you need more money, you probably will get an answer along the lines of, "Whait? We what did you do with all that money we gave you?"

Re:Uhm AWS EC2 Cluster Compute (1)

toruonu (1696670) | about 3 years ago | (#37407428)

Yes, wanted to make the same comment that hardware purchases are usually coming from a different budget line than power/rackspace/cooling/admins so one cannot look at TCO in EDU case because individual research groups never add up to it. If one manages the whole university/insititute compute infrastructure, then yes TCO plays a role, but individual groups usually get a bag of cash for HW and never get any dough for support/electricity and usually don't have to pay it either, it's those pesky grads that do the admin work no matter how inefficiently...

trade-off (4, Interesting)

TheSHAD0W (258774) | about 3 years ago | (#37406616)

Actually, that's a good question... Assuming no time constraints, at what point does it make sense to buy hardware rather than use the cloud? Take that budget above (roughly US$6K) and the best hardware you can get for that price: How many months would you need to run it, flat out, to equal the number of floating-point ops EC2 would give you for that cost?

Re:trade-off (4, Insightful)

subreality (157447) | about 3 years ago | (#37406738)

Sometimes, never. Don't forget to add up power, cooling, sysadmin time... And that's before getting to intangibles like being able to spin up 400 cores for an hour and getting your result same-day instead of only owning 40 cores and having to wait until tomorrow.

Cloud computing really cleans up for batch computing jobs like this.

Re:trade-off (1)

hairyfeet (841228) | about 3 years ago | (#37406890)

Question: How much bandwidth would that run? Because I've never had the chance to set up a cluster (most of my customers are SMBs and SOHOs) so I have no idea how much bandwidth you'd need to feed something like the Amazon cloud. If he is like many of us they probably have bandwidth limits and/or have to share that bandwidth with other users so if it takes a big ass pipe I could see that possibly being a problem.

That said if the cloud was out of the question I'd snatch up plenty of cheap AMD boards along with some cheap triples from Starmicro [] and simply mount the boards tray style in a simple home made rack. Those triples are last gen but at $40 a pop dirt cheap, any cheap cooler will work on those with an open tray design, and geeks sells AMD boards that they'll lost the I/O shields on for something like $20. Throw in some cheap DDR 3 and some small SATA drives along with Linux and voila! Cheap cluster.

But I have to agree with you and everyone else that if they are that tight on money the cloud would give them the most bang for the buck, and cut out a lot of the hassles and upkeep as well.

Re:trade-off (1)

crutchy (1949900) | about 3 years ago | (#37407148)

one application for cloud-based supercomputing is things like FEA and CFD, and for those the inputs and outputs are relatively small (compared to the numbercrunching inbetween). Autodesk has an online FEA service as part of their Inventor Pro package that seems like it will bring them business for large models/meshes. sometimes the hardware cost isn't as much as the software cost. the op mentioned linux, which is free, but what good will that do on its own. big numbercrunching analysis packages like FEA usually aren't cheap, and the cost to develop your own software is always high.

Re:trade-off (1)

rioki (1328185) | about 3 years ago | (#37407362)

I am not so sure about CFD. We did 3D CFD with NaSt3DGPF at the German Federal Waterway Administration. The input data to the computation and result sets where along the lines of multiple GB. Pulling the data out of the Cluster over LAN was already trouble. The process was speed up substantially when input generation was done on the actual cluster. (Input came from a few input txt files and was compiled into the computation grid.) Just getting the data to and from AWS will be a major nightmare.

Re:Uhm AWS EC2 Cluster Compute (4, Informative)

subreality (157447) | about 3 years ago | (#37406710)

+1. It is very nice to be able to spin up 50 instances, run the hell out of your job, then delete them. It gets done faster, and you don't have to deal with maintenance, upgrades, and obsolescence. Realized you need more RAM? Just adjust it! And so on. It'll likely come out cheaper than owning your own after you add up all the hidden costs (power, cooling, space, time, etc).

The only downside is there are no GPUs. But that's not really a downside: if you do end up developing a GPU version, your cluster configuration would completely change (1x2 cores per box, 3-4U boxes with many PCI-E slots, instead of 2x8 cores or however many you can economically cram into a 1-2U pizza box), so the investment you'd make now would be completely wrong for that future development. With cloud servers you minimize sunk costs.

I use Rackspace Cloud [] and it performs as promised. It's definitely worth a look.

Re:Uhm AWS EC2 Cluster Compute (1)

jpedlow (1154099) | about 3 years ago | (#37406754) []

amazon has gpu instances :)

Re:Uhm AWS EC2 Cluster Compute (1)

subreality (157447) | about 3 years ago | (#37407370)


Re:Uhm AWS EC2 Cluster Compute (2)

GeorgeK (642310) | about 3 years ago | (#37406756)

Actually, Amazon now offers instances with GPUs. See their page on High Performance Computing [] for more details.

Re:Uhm AWS EC2 Cluster Compute (1)

justforgetme (1814588) | about 3 years ago | (#37407022)

Have used rackspace cloud on some occasions. Runs very smooth indeed. Actually I stilll have some web apps over there. But is it just me or did they introduce a new low tier? I can remember running 512MB instances for about 12€/month about a year ago...
I have started experimenting with scale engine [] lately. They could actually be better for the cluster thing the author wants since they can deliver more computing power/$
I don't know how easy it will be to setup the distributed computing on them but it could be a good solution.

EC2 is expensive (0)

Anonymous Coward | about 3 years ago | (#37406720)

EC2 is really expensive, brah.

Re:EC2 is expensive (1)

Gaygirlie (1657131) | about 3 years ago | (#37407034)

EC2 is really expensive, brah.

Once you count all the costs of running your own cluster, ie. electricity, cooling, man-hours spent on configuring, installing and maintaining them, repairing broken parts etc. suddenly EC2 is likely cheaper than your own cluster, not to mention you can scale up on-demand if your requirements suddenly require such.

MB stacks (1)

Max Romantschuk (132276) | about 3 years ago | (#37406608)

I've seen quite a few projects where people have stacked motherboards with spacers, using booting over Ethernet and a single power supply for multiple MBs. Google should be of use here, I'm trying to get my offspring to school so I'm cheating and not providing any links...

But the idea is that skipping the case and other components makes things cheaper. Leaving the rig exposed without a case also eliminates the need for most cooling.

Re:MB stacks (0)

Anonymous Coward | about 3 years ago | (#37406662)

I've seen quite a few projects where people have stacked motherboards with spacers, using booting over Ethernet and a single power supply for multiple MBs. Google should be of use here, I'm trying to get my offspring to school so I'm cheating and not providing any links...

But the idea is that skipping the case and other components makes things cheaper. Leaving the rig exposed without a case also eliminates the need for most cooling.

Infectedtech are doing something exactly like this: stacks of motherboards for multi multi cpu on the extremely cheap! maybe they could help?

Re:MB stacks (1)

Required Snark (1702878) | about 3 years ago | (#37406708)

In what plane are the motherboards stacked? Are they in the horizontal or vertical plane? Vertical stacking allows the hot air to exit the top, while horizontal implies that external airflow must be provided to get the hot air out. Also, if you have multiple layers of vertical stacks then the top boards are getting the hot air from the lower boards.

A random suggestion: Have the motherboards all parallel at a 45 degree angle. This could provide passive heat driven air flow. The cool air enters at the lower edge and exits at the higher edge, so one side of the stack is the cool side and the opposite side is the warm side, I would think that you want the CPU fan near the upper edge of the board.

Re:MB stacks (1)

JorDan Clock (664877) | about 3 years ago | (#37406726)

Does it matter which way they're stacked? If it's horizontal, you could just flip it on its side if heat becomes an issue...

Re:MB stacks (1)

hedwards (940851) | about 3 years ago | (#37406792)

You're better off doing it from the start rather than waiting for it to be a problem. One of the things I remember from college was that if you had a DEC Multia, you had best be standing it up on its side, as they would have some serious problems very quickly if you lay them on their side.

Best thing is to avoid the possibility and the head aches of reorienting however many motherboards after it becomes a problem. Chances are you'll know it's a problem because they're unstable and possibly damaged.

Re:MB stacks (1)

mpetch (692893) | about 3 years ago | (#37406796)

Good idea, we could recommend an Apple Crate II [] which previously appeared on /.

Caseless Example: Ikea Cabinet Cluster (1)

cmholm (69081) | about 3 years ago | (#37406714)

An near-example of what Max is talking about can be found at the Home Linux Render Cluster [] . The builder threw six dual cpu motherboards into a small, gutted filing cabinet and Gig-e. Cheap, expandable.

However, if your friend hasn't got a very good idea how much mmmph she needs, the AWS EC2 rental idea has merit.

Re:MB stacks (1)

tempest69 (572798) | about 3 years ago | (#37406758)

I did this back in '03 (ok, I had discrete power, but a diskless boot) there is a project called warewulf that was pretty decent. The pxe boot was a little odd with the hardware at hand, so make sure the MB supports that sort of thing should you go this route. If you have a small enough data requirement (or fast enough broadband), a web service might be the way to go. Uploading/Downloading terabytes of data is a horrible thing over a low grade connection, and certainly isnt pretty over 100mbps lines. good luck, you're gonna need it.

Well, Infiniband is out (0)

Anonymous Coward | about 3 years ago | (#37406618)

Infiniband is out in that budget. But you could see how far you could get buying some cheap quad cores and interconnecting them with GbE. You can take a look at TomsHardware cpu charts (e.g. for 3dsmax rendering since this is a similar task:,2420.html [] ) and get the most bang for your buck.

Amazon EC2 (0)

Anonymous Coward | about 3 years ago | (#37406636)

why not try it?

Has she investigated existing clusters? (4, Informative)

Goonie (8651) | about 3 years ago | (#37406638)

Many universities/consortia have supercomputers available on which researchers can apply for (or buy) time. For example, my university is a member of VPAC [] , which has a big-arse cluster shared between a number of institutions. She might get much better bang for buck if she uses the money for that, rather than splashing out for dedicated hardware.

Re:Has she investigated existing clusters? (1)

RuBLed (995686) | about 3 years ago | (#37406882)

Also this guy have some and he's not sure what to do with it. They could also have brunch or something. []

Re:Has she investigated existing clusters? (0)

Anonymous Coward | about 3 years ago | (#37407178)

I was gonna suggest she spend the 4k on professional thieves to lift his shipment when it comes in, fence a few boxes to cover GbE, and build it herself.

But brunch is nice, too.

Re:Has she investigated existing clusters? (0)

Anonymous Coward | about 3 years ago | (#37407248)

What is this? Slashlist? Craigsdot? Anyone?


Re:Has she investigated existing clusters? (0)

Anonymous Coward | about 3 years ago | (#37406942)

Many universities/consortia have supercomputers available on which researchers can apply for (or buy) time. For example, my university is a member of VPAC [] , which has a big-arse cluster shared between a number of institutions.

She might get much better bang for buck if she uses the money for that, rather than splashing out for dedicated hardware.

i agreen with you ,good idea ! women’s handbags womens bags womens designer handbag womens designer handbags womens designer purse womens designer purses womens handbags womens purses designer handbags for men designer handbags for sale

Re:Has she investigated existing clusters? (1)

serviscope_minor (664417) | about 3 years ago | (#37407452)

Thanks for the suggestion about university facilities. This university is not great in this regard. Also, the cluster is used to process experimantal data, and can do either one large dataset or many tiny ones within 6 to 12 hours. The advantage of having a personal cluster is that the latency is low: you can tell by tomorrow if some of today's experiment worked, which is necessary to have a decent turn around on the experiments.

Yahoo M45? (0)

Anonymous Coward | about 3 years ago | (#37406652)

Is this program still around?

Whitebox 1U rackmounts (0)

mcrbids (148650) | about 3 years ago | (#37406654)

You don't specify whether or not your friend would be working out of a colo. If so, space will be at a premium.

My needs are high reliability, low cost, and high density. (colo)

I've been providing an excellent bang/buck ratio using whitebox 1U rackmounts made by SuperMicro. For about $1,000 I can get a late model CPU with a decent amount of quality ECC RAM, dual Gbit Ethernet ports, SCSI / SATA3 interfaces with a chipset highly compatible with CentOS Linux. (my distro of choice)

This is server-grade equipment, optimized for I/O throughput and reliability over raw processing power. You may be looking for raw computational power with a higher tolerance of downtime, in which case you'd want to try something else.

Benchmark then go by price-performance (0)

Anonymous Coward | about 3 years ago | (#37406670)

First thing is to run some benchmarks to find out which architecture is best. Next figure out how much memory per core and buy the systems with highest $ per performance. We recently went through this and for our highly-parallel workload. We found that the Intel Nehalem processors were faster by a factor of 2 or so than Opteron 6100 cpus. However, the Opterons were cheaper. We ended up getting boxes with 4 Opteron 6100 cpus that have 12 cores each. We need about 1 GB per core, so ended up with 32 GB of memory. We have an existing cluster with disk servers, etc, so went with 1 U boxes since we are somewhat space-constrained. Our communications need are modest so we are simply using gigabit Ethernet, though we are experimenting with channel bonding. We have found that it is worth buying name brand systems, though not by top-tier manufacturers. Our systems cost about $7k apiece as I recall. For us one of the benefits of buying more expensive systems was that they are considered equipment so we can use funds originally budgeted for facilities (overhead) expenses in our grant to pay for the computers. Without this the calculation might have been different. In the past the sweet spot for price/performance has been 1U boxes with 2 CPUs. We run CentOS on our cluster and like the stability combined with security fixes. We buy our systems with 3 year warranties and do not pay for service contracts. By the end of 3 years we have usually added some new systems and if the old ones die it isn't a big deal. The whole cluster thing works very well for us since we can add computing power in relatively small increments as funds are available.

Re:Benchmark then go by price-performance (0)

Anonymous Coward | about 3 years ago | (#37407166)

... buy the systems with highest $ per performance.

That sounds expensive. Wouldn't it be smarter to go for the best performance per $?

Imagine (1)

Tablizer (95088) | about 3 years ago | (#37406672)

Imagine a cluster of cheapness!

FPGA (0)

Anonymous Coward | about 3 years ago | (#37406674)

Check out a FPGA based solution such as NI's FlexRIO. Moving computation over to hardware makes things much much faster.

starcluster may be the answer (0)

Anonymous Coward | about 3 years ago | (#37406676)

It runs sun grid engine with a NFS master on EC2.

Use the Existing Grid (3, Insightful)

Roger W Moore (538166) | about 3 years ago | (#37406682)

Why buy your own when you can use existing GRID infrastructure? For 4k you can't do much more that get a few decent desktops for yourself and a few grad students and/or postdocs. Rather than blow it on a massively underpowered cluster use the grid. I know the UK has massive clusters available to researchers so find out how to get an account and resources on them and use those. For test jobs, interactive analysis and other low latency tasks use your desktop.

Not LoseThos (-1)

Anonymous Coward | about 3 years ago | (#37406686)

Use Bare Metal: Good luck. LoseThos is not for high performance computing -- it's for cheap thrills for kids and stuff.

God says...

to have been £ 2:0:6 10/32, which is about ten shillings and sixpence,
or more than five-and-twenty percent. cheaper than it had been during
the sixty-four last years of the last century; and about nine shillings
and sixpence cheaper than it had been during the sixteen years preceding
1636, when the discovery of the abundant mines of America may be
supposed to have produced its full effect; and about one shilling
cheaper than it had been in the twenty-six years preceding 1620, before
that disco

buy one Opteron 6100-based box (5, Informative)

Chalex (71702) | about 3 years ago | (#37406702)

You can get a SuperMicro reseller to sell you one workstation with 4 sockets of CPUs and a bunch of RAM. UK£ 4000 = 6 299.2 U.S. dollars

That buys you a box with 4 x Opteron 6134 (32 cores) and 128GB RAM (32 x 4GB sticks). And some hard disks.

Re:buy one Opteron 6100-based box (2, Informative)

Anonymous Coward | about 3 years ago | (#37406842)

parent is correct.

or for some more get the the 4x6168 (48 cores for about 770$ each ~ 3000, MB is around 800$, supermicro cabinet - 800$ OR use a chenbro fileserver for 300$) and 8GB sticks are pretty cheap these days and almost proportional in price compared to the 4GB - get the KVR1333D3Q8R9S/8G for about 90$ or about 90*8 = 720$ for 64GB. Around 5000 in total, though there are some additional costs like coolers etc but still wont break budget.

i got a similar config for doing something that is cpu intensive (branches, fp and mem intensive) and not easily portable to gpus.

the resultant machine hardly cracks 600w, and is 4 times as fast as an i7 960 on my application.

Re:buy one Opteron 6100-based box (4, Informative)

toruonu (1696670) | about 3 years ago | (#37407416)

Yes, my recommendation would be also, we do loads of LHC data analysis and simulations and have found that for real science real cores outweigh hyberthreaded ones so we run Opteron 6172 x2 in supermicro chassis that fits 4 servers into 2RU. The cost of such a box of course is ca 11keur, but it gives 96 cores 192GB ram. Now she can get for half the money that she has about half of that so 48 cores 96GB ram should be doable using SM boxes and you can scale up/down with CPU frequency to adjust the cost and maybe adjust total RAM alongside to fit in the budget. If she plans to later expand she may actually want to spend the money to get the 2U chassis with only 2 of the 4 machines present and later add one/two more by just buying the board with cpu/ram.

Consider other options (1)

msobkow (48369) | about 3 years ago | (#37406716)

You can't really get a cluster for that kind of money. You can barely get one decent box.

But you shoud be able to rent a lot of computer time in the cloud for that kind of money, or use it to buy time on someone else's cluster.

Commercial cloud computing clusters? (0)

Anonymous Coward | about 3 years ago | (#37406732)

Have you considered commercial services like Amazon? I believe some are pay as you use.

Theoretical analysis (3, Informative)

Warlord88 (1065794) | about 3 years ago | (#37406742)

OP hasn't mentioned a lot except budget. Since you are on such a tight budget, I would highly recommend doing some theoretical analysis first. Do you have a serial code? How much parallelism exists in the code? You say the task is 'very parallel', but Amdahl's law [] (which is really common sense) will tell you that even for small amounts of serial sections of code, your speedup will be limited. You should also consider the amount of time the code actually runs. Achieving a speedup of 2 for a serial code that runs for one minute is near worthless.

After you estimate speedup, do some rough calculations on the basis of average cost of a processor and the the number of processors required. This should give you an estimate of the hardware cost required. Compare that with the cost of CPU cycles per dollar you get using a cloud service such as Amazon.

Re:Theoretical analysis (0)

Anonymous Coward | about 3 years ago | (#37406878)

When a scientific computing task is described as 'very parallel', I think you can assume that it's more-or-less perfectly parallel. There are lots of scientific problems that are embarrassingly parallel. My own work, for example, could be parallelised 100,000-fold or so without using more than a few percent of overhead.

Amazon AWS. (4, Informative)

Haven (34895) | about 3 years ago | (#37406744)

$1.60 / hour for the largest non-GPU cluster instance. This also provides you with rather fast interconnects and scalability with multiple instances.

Only £4,000 in hardware would be a waste of money. You wouldn't have all that much computing power, and it would be obsolete immediately.

Re:Amazon AWS. (0)

Anonymous Coward | about 3 years ago | (#37406970)

You have a point here, with this price of 1.6$ / hour she can pay for 5 and a half month of 24x7 computing processing time on cutting edge (virtualizaed) hardware. If I were she, I'll make my mind in favor of AWS (of course if this 5.5month is enough for the project)

Re:Amazon AWS. (1)

outsider007 (115534) | about 3 years ago | (#37407194)

That's the on-demand price. She would be paying the spot price which is roughly half of that.

Buy the cycles, not the machines (0)

Anonymous Coward | about 3 years ago | (#37406752)

It will cost her more that 4000GBP in grad student time to configure/manage the cluster. (Even using a turnkey installation system like ROCKs.) She should use the money instead to buy time on a national/university cluster system.

Your best bet (1)

eclectro (227083) | about 3 years ago | (#37406768)

Study the design of the "microwulf" [] and it's relatives. Considering that hardware prices has dropped since 2009, your task might be achievable.

HP (1)

guttentag (313541) | about 3 years ago | (#37406770)

HP? Is that you?

Not doable (1)

darkjedi521 (744526) | about 3 years ago | (#37406800)

Assuming a 1.5 to 1 correspondence with the USD, you're either getting a decent cpu box and no storage, or a reasonable amount of storage and no CPU. I build/run supercomputing clusters for molecular dynamics simulations at an university in upstate New York, and I wouldn't even consider attempting a cluster for less than $25,000.

Since the OP didn't specify if this was massively parallel or not, I'm going to assume this is so I can use AMD chips for cheapness.

First off, storage. Computational output adds up quick. You're looking at $7,000 USD for 24TB raw storage from the likes of IBM or HP or Dell. Yes, you can whitebox it for cheaper, but considering if you lose this box, nothing else matters (And I doubt you have the funds for proper backups), it pays to get hardware that's been tested and is from a vendor you can scream at when it breaks.

Second, interconnect. A cheap netgear will work, but reasonable internode communication is not cheap, especially if moving largish amounts of data. This could run $1000 to $3000

Finally, the compute hardware itself. A decent node will run $3000 to $5000 depending on the core count, socket count, GHz, and to a lesser extent RAM.

Assuming you want 128 cores, you're looking at 8 machines for compute ($32,000 right there assuming $4K/node, and dual 8 core chips), plus another $7K for the file server/landing pad, and finally add $1500 for a decent switch that can let those nodes talk to each other at line speed and allow room for future growth. Total cost: $40,500 USD or 27,000 pounds assuming the 1 pound:1.5 USD ratio.

Re:Not doable (1)

cowboy76Spain (815442) | about 3 years ago | (#37407270)

First off, storage. Computational output adds up quick. You're looking at $7,000 USD for 24TB raw storage from the likes of IBM or HP or Dell. Yes, you can whitebox it for cheaper, but considering if you lose this box, nothing else matters (And I doubt you have the funds for proper backups), it pays to get hardware that's been tested and is from a vendor you can scream at when it breaks.

Having someone to scream off may help ease some pressure, but if you do not have proper backups you are fubar no matter who you scream to.

computer recyclers (1)

thedarknite (1031380) | about 3 years ago | (#37406804)

Try a computer recycling centre, most tend to be short on storage and are happy to sell a large number of desktop machines at a lower than normal price per unit. Community operated ones tend to be more helpful than business ones though.

Buy a small chunk (1)

pigwiggle (882643) | about 3 years ago | (#37406814)

Buy a small chunk of something that looks like the big machines she will be using. As others have said, with that little money you aren't going to get legitimate computational resources. But she will certainly qualify - or already has - on some of the larger public machines. In my experience, it is really nice to have a small, i.e two or three nodes, cluster to test and benchmark code. You can look at things like parallel performance on a single node versus across nodes. If the code plays well with shared memory. Can the code reasonably mix shared and non-shared parallelization schemes. And so forth.

Re:Buy a small chunk (0)

Anonymous Coward | about 3 years ago | (#37406844)

great advice. buy 3 or 4 of the cheapest you can get at the time, get the infrastructure going and get some results.

when you're really finished, the market will have changed. reevaluate your options then

Um, hello? Botnet? (0)

Anonymous Coward | about 3 years ago | (#37406838)

They are going for like $3 an hour.

Ps3 (0)

Anonymous Coward | about 3 years ago | (#37406866)

If your friend doesn't mind tinkering with the OS, it might be worth buying a bunch of ps3's. They disabled the old other os feature, but id imagine its not too hard to mod them , or do something along those lines.

Your best choice (-1)

Anonymous Coward | about 3 years ago | (#37406868)

It will be your best chice on our website ! wholesale purses wholesale purses cheap wholesale purses suppliers wholesalehandbags wholesalers designer handbags women designer women designer bags women designer handbags women designer purses women s designer handbags

Raspberry Pi, Model B...... (1)

Anonymous Coward | about 3 years ago | (#37406874)

$35/element, runs a boring Linux distro, runs very cool, low power consumption (less than 1w), onboard Ethernet.


Raspberry Pi []

BOINC Project? (4, Interesting)

bradley13 (1118935) | about 3 years ago | (#37406904)

She could also consider creating a BOINC project [] . She could then do some publicity locally and on forums, to get people to choose her project. I've never tried creating a BOINC project, so I don't know how hard this is. However, I do run the client as a background task, and I imagine many other people do as well.

Don't spend it on hardware. (1)

Jane Q. Public (1010737) | about 3 years ago | (#37406914)

Spend the money on a programmer to parallelize the algorithm on standard CPUs, and put it out on BOINC. People volunteer their spare cycles for BOINC projects that are barely more interesting than the chemistry of aardvark snot. She would likely get volunteers if there's anything of even passing interest in her research.

Boxes on shelves (1)

stevelinton (4044) | about 3 years ago | (#37406930)

If your friend doesn't want to do a lot of engineering work, then for this price I would just buy 10 or so PCs (depending on memory/CPU tradefoffs) from wherever has a special offer, plus a gigabit switch and put them on shelves. If you need a lot of memory, or can usefully share memory then that would be a bit different, but you can buy a usable headless PC for £300-£400. This will also not be terribly power efficient, nor will components like motherboards be of the highest quality, but you get more bang for the buck that way than almost anything else except second-hand. At the other extreme, you could probably buy a single 24-core AMD box for the money with quite a lot of RAM and just run a lot of processes on it.

Talking of second-hand, the other thing to do is to see if anyone has a cluster they can't feed (ie power) any more. Our aplied maths dept is about to shut down a 3 year old 1000-core cluster because they can't afford the power to run it and their newer 2000 core cluster. A slice of that would be great and someone locally might be able to help you in a similar way.

Re:Boxes on shelves (1)

stevelinton (4044) | about 3 years ago | (#37406966)

Just did a bit of checking. For £249 including VAT you can get a mini-tower, dual core midrange CPU and 2GB RAM. A dozen or so of these and a switch looks very appealing if there is space. 300W PSU, so cluster should be under 5kW.

Re:Boxes on shelves (1)

Vectormatic (1759674) | about 3 years ago | (#37407330)

if you are open to custom building yourself, you can dispense with the optical drive and other crap you dont need (pick smaller drives if storage isnt a concern etc.. avoid windows tax, how ever small it may be on a 250 quid box) and dump that into stronger hardware

Late last year we had a crunch-intensive problem at work, and the internal IT department wouldnt even give us a price quote, just said they could do it (a problem which required six octo-core xeon machines, in a world wide company with 90k in people, go figure), so i drafted a small proposal saying i could build ten hex-core AMD boxes with 4 GB ram each for 5k in euros. Off course it wasnt take seriously, because it wasnt enterprise, but there you go, custom bare bones build with a beefy cpu

Re:Boxes on shelves (0)

Anonymous Coward | about 3 years ago | (#37406986)

You are charged for electricity in the UK? I've never heard of that in the US. The university pays for the electricity, presumably with some of the money they took out of your grants for "overhead." The departments and individual research labs aren't charged for power. Do you pay less in overhead in the UK? Do you turn stuff off at night? Because we usually leave the lights on all night.

Microwulf (1)

tqk (413719) | about 3 years ago | (#37406940)

Microwulf [] .

Who is paying the electricity bill? (1)

Michael Woodhams (112247) | about 3 years ago | (#37406950)

And how much space and air conditioning do you have? Depending on the answers do these questions, the optimal* solution might be 'get a bunch of 5 year old computers nearly for free.'

* Optimal for your friend, not for her university.

Re:Who is paying the electricity bill? (1)

Memroid (898199) | about 3 years ago | (#37407012)

Free? where?

Re:Who is paying the electricity bill? (0)

Anonymous Coward | about 3 years ago | (#37407162)

Around here you can by Core 2 Duo + 2GB Lenovo desktops for 35€. They even have linux installed. I would call that almost free considering the processing power.

Many of those kind of places don't advertise, but I'm guessing most decent size cities have several places selling used hw.

Re:Who is paying the electricity bill? (0)

Anonymous Coward | about 3 years ago | (#37407298)

At your university. Computers are replaced all the time, and you can't throw out the old ones. there are unwanted computers all over the university. just ask for one. but they aren't going to be good enough to do supercomputer calculations.

best dre beats (-1)

Anonymous Coward | about 3 years ago | (#37406992)

For songs lovers who would opt for the popular audio dre beats [] phones, due to the fact the created-in MP3 features has been confirmed to completely exchange the MP3 player abilities, the initial wife arrives along with dre beats diddy [] cell phone headset, ear products mainly because suppliers do not have the advantage , With each other with the consideration in the finished solution, so it will not use more upscale goods, these beats pro by dre [] headphones naturally does not meet the ears of music lovers, they will also purchase Sennheiser, Audio-Technica beats solo hd by dre [] headphones and other skilled vendors to change.And the marketplace beats dre solo [] headset quantity of versions, there are ear, head, ear, and so on., the consumer demands according to their preferred beats studio [] headphones with the no cost solution, then Xiaobian identified a amount of specialist brand of beats tour [] headphone items, quality and cost are very good.

Had similar situation Dell410s + Condor; no cloud (1)

Oori (827315) | about 3 years ago | (#37406994)

I was in a similar situation setting up a research group. Wanted an expandable setup for a research group, that would meet approval of local IT sysadmins (some remote management opts, vendor support). Per 2.6K pounds a pop I got a Dell poweredge T410 server with 2 6-core CPUs and 24GB RAM. I'm never one to push a Dell (been purchasing IBM/HP for years) but this is a decent machine for a decent price. I tried various cloud solutions using virtual machines on Amazon and similar frameworks, but for the kind of work we do (frequent software updates, massive amounts of data that need to be stored locally and can't be transferred easily), those don't scale. We use Condor as a job submission engine. Not that we don't like SGE but with Oracle's plans ( one can never tell. PS: remind you friend to invest in a QNAP NAS or similar for backups / disaster recovery.

Don't use EC2 (1)

hawguy (1600213) | about 3 years ago | (#37407010)

I don't think any of the posters recommending EC2 have ever looked at the economics of EC2 versus self-hosting.

If you have long-term compute needs (as opposed to needing to throw lots of cores at a problem to get fast results in a short time), you're better off buying a Dell.

An EC2 Quadruple Extra Large EC2 instance is $1.60/hour. You have around $6500 USD, so you could buy 169 days of computer time at EC2 (ignoring the cost of I/O and network bandwidth).

This instance has 23GB of RAM and is equivalent to 2 x Intel Xeon X5570 [] CPU's.

For around $5000, you could buy a Dell R710 with dual X5647's, 32GB RAM, RAID-1 1TB SATA drives (depending on your storage needs, you might want to move to faster SAS disks). As long as you have a suitable office to host the server, your only recurring hosting cost is electricity (around $70/month) and maybe you'll need to spend $500 on a UPS. If you need to pay for hosting/colocation somewhere, that will definitely change the economics.

So, with your budget, you get one node + UPS + electricity for a year. All for the price of around 5 months of EC2 time.

You come out ahead even if you want to throw away the server every 6 months and start fresh.

You can save a few bucks by building your own (or going to a custom whitebox builder), but the Dell comes with 3 years of next business day support. Last time I priced out a whitebox builder, they beat Dell's best discounted price by about 10% and only offered a 1 year warranty.

Re:Don't use EC2 (1)

goarilla (908067) | about 3 years ago | (#37407156)

Dell support isn't ( always ) heaven on earth !

Re:Don't use EC2 (1)

outsider007 (115534) | about 3 years ago | (#37407260)

I think you're overestimating the cost of ec2 by 2x and underestimating the cost of electricity by 3x

Re:Don't use EC2 (1)

PhrstBrn (751463) | about 3 years ago | (#37407280)

You can save a few bucks by building your own (or going to a custom whitebox builder), but the Dell comes with 3 years of next business day support. Last time I priced out a whitebox builder, they beat Dell's best discounted price by about 10% and only offered a 1 year warranty.


  • SYS-6026T-URF - $1200
  • Xeon X5647 - $850 each
  • 4GB ECC Registered - $50 each
  • 1TB SATA 7200RPM - $120 each (WD RE4 or equiv)

Chassis + 2xXeon + 24GB RAM + 2x SATA drives =~ $3500. Drop the chassis down to 1U and you're looking at ~$3200. I'm assuming the $5k Dell is an H200 controller, and not a H700 card.

These prices are retail prices, not wholesale, and most are on the conservative side, so a system builder is going to have better wholesale prices on parts than what I quoted even. Unless your system builder is making 50%+ margins an their hardware, your system builder is trying to rip you off, or you're not comparing apples to apples.

Clusters on the cheap (1)

kawabago (551139) | about 3 years ago | (#37407070)

Mums are nice.

Re:Clusters on the cheap (1)

John_Booty (149925) | about 3 years ago | (#37407188)

Yer mum's pretty nice, mate. Winkwink nudgenudge.

Just pay to use someone else's HPC cluster (0)

Anonymous Coward | about 3 years ago | (#37407206)

You're probably excited about being able to help her burn some money on BUILDING some ultra-cheap Beowulf-style HPC cluster but your friend is probably more interested in just actually USING a working production HPC cluster to solve her very parallel, very CPU-bound scientific/mathematical problem ASAP, and not so much into having to setup and administrate the HPC cluster.

Based on the little bits of info you've provided, I'm going to guess your friend is something like a new assistant professor/lecturer at some small research university? Maybe located in the UK or some other European country that prefers to use pounds sterling instead of euros?

Let's see here...4K in GBP is about 6.3K in USD right now. For that amount of money, I would check if the university or a larger affiliated academic institution might have some sort of HPC cluster that you can just pay for compute time on. Best bet is if she can talk to her colleagues involved in her specific research area who might already be aware of what's available. A Google search for "UK high performance computing" comes up with some possible good hits.

In the U.S., there are organizations like XSEDE ( which has a system where academic researchers can apply for an allocation of compute time to use on various HPC clusters at institutions affiliated with XSEDE. Some of these institutions will also separately independently rent out compute time (sometimes it's easier to just wave some money at some supercomputing center than deal with the hassle for applying for formal allocation based on the proposed merits of your project hoping that somebody will think it's worthy). See the Triton Resource ( at SDSC ( as an example.

When your friend is actually comfortable just using a HPC cluster to run her compute/simulation jobs, publishes her results, gets noticed for her work by more senior peers and funding agencies, applies for more grant funding, and when serious amounts of money start rolling in ($50K-$1M+), then it's time to start building her own (which is another long story).

It all depends on structure of tasks to be run (0)

Anonymous Coward | about 3 years ago | (#37407240)

Main question is how specialized is software to run. If You are sure what kind of microkernels are required and its fixed over cluster lifetime, make it fpga/custom made cpu with IP logic extension. The best for buck if possible and allowing moderate variation in calc kernels are dsps, also ones becoming multicore lately (for exmple TI). If previous fail: biggest raw power but with high energy costs are x86 variations. If task is easily separable into tinytasks and energy costs do matter - hundreds/thousands of separate tiny arm cores of A8 or A9 grade (A9 planned and starting production for multicore cpus) on gigabit ETH link might suffice.

Do you count manpower and infrastructure costs? (1)

grid17 (2461914) | about 3 years ago | (#37407242)

If you are extremely data heavy, the cloud becomes quickly much more expensive than buying your own. The Broad Institute made some recent experiments on Amazon analyzing genome experiments, and they said Amazon was 4x more expensive.

But for her cpu-heavy workloads the cloud would work perfectly.

more things to consider:

If she buys her own hardware there are a lot of extra costs to the raw hardware:
1 someone needs to set the thing up, administer it, and support it with patches, etc. even for a small cluster this is a good percentage of a person. if she has a slave student doing that, great! although if the guy leaves there will be an issue. if she needs to spend money on a person, then amazon will be muchmuch cheaper.
2 there is an electricity bill and you need space, probably cooled space. if it is available, great! otherwise it can be a showstopper (fire hazard in a lab)
These costs are included when people talk about total cost of ownership. If you factor them in, the cloud suddenly becomes veeery interesting. Btw, EC2 is not the only player on the market now, there is Azure and also the IBM SmartCloud, with competitive pricing.

For normal cluster computing you can go to a number of startups that will build an on-demand cluster for you based on amazon, my favorite in the research domain is cloudbroker - [] - who will actually render the software you need into a SaaS based on EC2 or the IBM cloud. you just launch your hpc cloud app, with your uploaded data and you pay for the workload you did. the bill includes just amazon or ibm costs, licenses if any and a surcharge by cloudbroker which is totally worth the money because now you do not even need to set up the software and the virtual machines anymore.

World Community Grid: Free if You Qualify (1)

BBCWatcher (900486) | about 3 years ago | (#37407244)

Somebody upthread mentioned BOINC [] , which is a great idea for many parallel-oriented compute-bound problems. However, while making your project compatible with BOINC is necessary, it's usually not sufficient. The problem is marketing, to convince enough people to run your work. World Community Grid [] , sponsored by IBM, is free and is an excellent way to solve that problem. You can submit a proposal [] , and if approved you'll quickly have lots of BOINC-powered computing working on your problem.

Buy time on HECToR (1)

CSMoran (1577071) | about 3 years ago | (#37407312)

At least you'll be running on the bare metal, not some virtualized piece of cloud. []

AWS, Grid, Owned hardware, many options (1)

gef7 (1789448) | about 3 years ago | (#37407368)

In short:
  • AWS could be good value for money, if you DON'T have data-intensive tasks (otherwise it gets expensive quickly)
  • Grid could be even better value for money, assuming you can get the service AND assistance in your lab, for low or zero cost (eg. in US/EU this should be fine)
  • Owning hardware is a MUST if you have sensitive data (medical, financial etc) or just need to build local expertise (more input needed here)

Assuming the later, check among Supermicro & Dell servers. Last time I needed to setup a cluster, the Dell R610s were a good pick, giving great manageability over the LAN, low volume and decent features (balanced storage space along with cpu capacity, around 8 cores + 8 TBs per 1u blade). Don't rule out also options like Shuttle XPCs, they are damned robust in thermal aspects (hey, you'll be running these continuously, won't you?). Finally, don't underestimate the need for local sysadmining; you will likely need to setup a queueing system (Torque, *PBS*, SLURM, SGE, LSF, NQS, Condor) and manage the whole thing. This won't happen automatically, take a note on that. If you run something of the pbs or sge family I can happily help with setting up a tool called qtop []

Phone big companies (0)

Anonymous Coward | about 3 years ago | (#37407404)

Call the IT depts in some big companies. They may be throwing out old desktops. If you are an institution then you can honestly turn what would have been a scrappage deal (which tends to be regulated and cost money) into a £1 sale from one institution to another.

Just beware the liability for scrapping the computers after the project. But hey, that probably comes out of someone elses budget.

Use idle lab machines (1)

axedog (991609) | about 3 years ago | (#37407414)

Is the research group in a university? Most universities have a lot of computing power that sits idle for large proportions of the time in their undergraduate computing laboratories. There's a significant resource that could be exploited simply by deploying jobs to idle machines.

Use the Grid (1)

nextgens (1575645) | about 3 years ago | (#37407440)

In the UK there are academic grids that research groups can use like the ngs or gridpp (for free or next to nothing) [] I used to work at the Center for Parallel Computing where I am sure some people would talk to her. []
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?