Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Linux Clustering

CmdrTaco posted more than 9 years ago | from the old-memes-resurrected dept.

Linux 154

An anonymous reader writes "Beowulf clustering turns 10 years old, and, in this interview, creator Donald Becker talks about how Beowulf can handle high-end computing on a par with supercomputers."

cancel ×

154 comments

Sorry! There are no comments related to the filter you selected.

Can you imagine... (5, Funny)

FortKnox (169099) | more than 9 years ago | (#10238959)

... the amount of replies that will start with the same subject header as mine and not be funny at all?

I sure can!

Re:Can you imagine... (2, Funny)

Anonymous Coward | more than 9 years ago | (#10239012)

Not really (well apart from your post of course), tell me, how would it go?

"Hey, can you imagine a beowulf cluster of er... beowulf cluters?"

Doesn't quite work, you see.

it will only rival... (-1, Troll)

Anonymous Coward | more than 9 years ago | (#10239018)

the number of horrendously written network drivers spewed by becker, himself.

Imagine All... (1, Funny)

grunt107 (739510) | more than 9 years ago | (#10239047)

In honor of the Beatles/Apple topic a while ago, I was inspired to alter the heading slightly (sorry).

Imagine all the clusters
It's easy if you try
With Linux on them
Computing Particles in Sky

Imagine all Beowulf
Crunching in harmony (AhIahahah...)

MOD PARENT DOWN (-1, Flamebait)

Anonymous Coward | more than 9 years ago | (#10239061)

jees, his post sucks. +5 insightful? my fucking ass.
try "+5, thinks he's funny", or how about "+5, got modded cos he posted early, although his content is absolute shite".

Re:MOD PARENT DOWN (-1, Offtopic)

Anonymous Coward | more than 9 years ago | (#10239132)

Looks like someone is jealous I got first post, and he didn't.

I only have 3 words for you:
YOU FAIL IT!!!!!

With Love,
FK

re: can you imagine... (2, Funny)

ed.han (444783) | more than 9 years ago | (#10239074)

i want you to know that i seriously thought about doing that and having had the same thought, decided against it.

now, imagine a beowulf cluster of slashdotters having that same thought... :>

ed

Yes I can! (0)

shfted! (600189) | more than 9 years ago | (#10239083)

And I can already see the beowulf cluster of responses like yours ;)

Re:Can you imagine... (-1, Offtopic)

Anonymous Coward | more than 9 years ago | (#10239274)

I'd just like to publicly announce that 20721 has thoroughly owned mercatur.net [mercatur.net] . Truely a camwhore legend. She will be missed.

A few Slashdot clichés: (5, Funny)

Sean Johnson (66456) | more than 9 years ago | (#10240210)

- Can you imagine a Beowulf cluster of these?
- How long until the RIAA sues them into oblivion once they find out how may MP3's you can put on one?
- "Can you put Linux on it?" or "Yes, but will it run Linux?"
- "Yeah, but does it run Doom3?" or "And it still won't run Doom3."
- Any comment regarding "Duke Nukem Forever" taking literally 'forever' and being termed 'vaporware'.
- I am not buying one until they support ".ogg".
- I for one welcome our new (insert name of company mentioned in post or story) overlords.
- "George Lucas raped my childhood" or "Greedo shoots first" comments on any story incorporating the Star Wars franchise.
- A comment including these 3 components in any order: Natalie Portman, naked and petrified, hot grits, one's pants.
- Microsoft = Evil, MPAA = Evil, RIAA = Evil; with anything else incorporated to try and fit those equations into the topic at hand
- Some type of reference to the size of one's ProN collection, the amount of ProN that can be stored on the gadget or technology in question, or the ProN industry itself being the first to make good sue of the new technology or gadget in question (ergo: the ProN industry drives technology)
- The posted cliché being self-described as an "obligatory" post in the heading area if that particular cliché had not been addressed yet by previous slashdotters. (e.g. "obligatory Beowulf cluster comment")
- Post revealing the fact that the story's homepage had been slashdotted already, culminating towards another post later on with the homepage story itself being copied & pasted verbatim (often with a subsequent post purporting that this is karma whoring, even though the poster admits it is indeed helpful anyways.)
- Remark on the size of some new storage advancement about how many LOC's (Library of Congresses) can fit on it, or any other remark noting how this can be an actual valid unit of data storage measurement.
- A variation of the Zero Wing video game intro dialogue regarding it's broken English translation: "Someone set up us the base....we have every ZIG, make your time".....blah, blah, blah.
- Very soon lists such as this will be clichés as well.
- Similarly noted and additional clichés may be found here: http://en.wikipedia.org/wiki/Slashdot_subculture

Not first post (-1, Offtopic)

Anonymous Coward | more than 9 years ago | (#10238961)

this is not the first post so you may continue reading on

news? (5, Insightful)

dan2550 (663103) | more than 9 years ago | (#10238962)

i dont mean to sound like a troll or anything, but is this really news. over the last year or so, (nearly) all of the articles on /. about fast computers have been clusters.

I second your observation: +1, Interesting (-1, Troll)

Anonymous Coward | more than 9 years ago | (#10239064)



With all of the actions by the delusional, dangerous, and inarticulate U.S. "President" George W. Bush [whitehouse.org] to burn the U.S. consitution, there are many alternatives to Beowulf stories.

Thank you and have an Ashcroft_Bush_Cheney_Powell_Ridge_Rice_free_day,
Kilgore Trout

Re:news? (3, Funny)

strictfoo (805322) | more than 9 years ago | (#10239076)

Great news too: creator says creation is really good!

In other news: "Ford says their cars are just as good as BMW's and Emachine states their computers rival Apple's"

Re:news? (0, Offtopic)

EvilAlien (133134) | more than 9 years ago | (#10240393)

I'm sure Emachines do rival Apple, at least in terms of desktop penetration.

I'm also sure that Ford kicks BMW's ass in terms of popularity, at least in the USA.

I'm going to have to work something about the prevalence of bad taste into my Stupid People Theory.

oh great... (0)

Anonymous Coward | more than 9 years ago | (#10239184)

a story on how linux clustering is good.

I'd predict this thread will degenerate, but where would it go from here? :)

How do you mean, fast? (2, Funny)

raehl (609729) | more than 9 years ago | (#10239197)

Clusters are the slowest computers available...

If your metric is moving around data, as opposed to how many no-ops you can do a second while waiting for your data to get there.

Re:How do you mean, fast? (1)

Mark of THE CITY (97325) | more than 9 years ago | (#10239558)

It's fast only if the problem being solved is, as they say on comp.parallel, embarassingly parallel. IBM mainframes are likely to still be the I/O champs.

Re:news? (2)

fymidos (512362) | more than 9 years ago | (#10239500)

Clusters is power for the people.
It's something that simple people, without a white form, a microscope, and 500 million budget, can work on, and make better.
Undoubtfully many advances that have been made for clustering, will be used in many other aspects of computing. Even supercomputer will benefit from them.

Re:news? (0)

Anonymous Coward | more than 9 years ago | (#10239791)

Interesting, so you are saying that one day soon we will see a Beowulf cluster of Cray's?

Obligatory... (-1, Redundant)

LighthouseJ (453757) | more than 9 years ago | (#10238969)

I know... I know... Imagine a Beowulf cluster of these!

Re:Obligatory... (1, Funny)

grape jelly (193168) | more than 9 years ago | (#10239013)

Wouldn't a beowulf cluster of beowulf clusters in fact just be a beowulf cluster? Hmm....

Oh yeah and *groan* with regard to that joke! =-P

Re:Obligatory... (1, Redundant)

saudadelinux (574392) | more than 9 years ago | (#10239030)

I don'wanna! Won't an infinite number of /. readers imagining an infinite number of variations on Beowulf cluster jokes topple all the turtles?

Re:Obligatory... (3, Funny)

maxwell demon (590494) | more than 9 years ago | (#10239517)

I know... I know... Imagine a Beowulf cluster of these!

Ok ... but does it run Linux?

imagine (3, Funny)

Anonymous Coward | more than 9 years ago | (#10238972)

awww, fuggit

Great! (-1, Offtopic)

Anonymous Coward | more than 9 years ago | (#10238978)

I [apple.com] most certainly [oldlutheran.com] like [likeitis.org.uk] how [howstuffworks.com] slashdot [slashdot.org] provides [provide.net] links [dlink.com] in [in.gov] its [techtarget.com] articles [findarticles.com]

Candle Cluster (3, Funny)

KrackHouse (628313) | more than 9 years ago | (#10238983)

I'm picturing the ten candles on the Wolf-cake in close proximity with frosting interconnects and one big flame in the middle.

You knew it was coming (-1, Redundant)

Anonymous Coward | more than 9 years ago | (#10238991)

Imagine a Beowulf clust-.... oh...

Cliche.. (-1, Redundant)

Anonymous Coward | more than 9 years ago | (#10238993)

Imagine a beowulf cluster of... oh, wait!

(-1 redundant, I can't be the first person to think of this)

imgaine a beowulf cluster... (0, Redundant)

herbert_axelrod (554087) | more than 9 years ago | (#10238995)

el oh el

Happy Anniversary... (5, Funny)

crawdaddy (344241) | more than 9 years ago | (#10238996)

Happy Anniversary to the most over-used joke on Slashdot. I'll be wearing my tin-foil hat all day to commemorate it. (The 10th anniversary is the aluminum/tin anniversary)

actually... (2, Interesting)

ed.han (444783) | more than 9 years ago | (#10239163)

what is the most over-used joke on slashdot? options:

1. step 1: [topic], step 2: ?, step 3: profit!
2. natalie portman/hot grits
3. in soviet russia, [inversion of topic] you!
4. cowboy neal [action]
5. beowulf cluster
6. goatse guy
7. [technology/entity] is dying!

just curious...

ed

Re:actually... (3, Funny)

XaviorPenguin (789745) | more than 9 years ago | (#10239229)

8. ???
9. Profit!!

Re:Happy Anniversary... (1)

Stillman (185591) | more than 9 years ago | (#10239504)

o/~ And the word that you wanted was Aluminium.

;)

Damn merkins.

premature (1)

commodoresloat (172735) | more than 9 years ago | (#10240182)

The beowulf cluster itself has the 10 year anniversary today. The over-used joke doesn't have its anniversary until ...

(checking calendar)

tomorrow.

sure (-1, Offtopic)

CrazyJim1 (809850) | more than 9 years ago | (#10239015)

In soviet russia, computers make a beowulf cluster of people for work. www.geocities.com/James_Sager_PA

I'm imagining... (-1, Redundant)

Oxy the moron (770724) | more than 9 years ago | (#10239016)

... a Beowulf cluster (or two, or three, or forty) of posts that are all roughly the same...

BlueGene (4, Interesting)

a3217055 (768293) | more than 9 years ago | (#10239021)

All this sounds good and Interesting, and Becker did a tremendous ammount of development in this field. But I was just wondering, what about supercomputers like BlueGene/L which have very fast interconnects. Many supercomputers/distributed systems run MPI based programmes and such programmes need a high interprocess commmunication does anyone one know how good these are in a Bewoulf Cluster? thanks a3217055 They said that of all the kings upon the earth he was the man most gracious and fair-minded, kindest to his people and keenest to win fame. :-The Geats' tribute to Beowulf after his death.

Re:BlueGene (5, Insightful)

jamesdood (468240) | more than 9 years ago | (#10239303)

Since I administer a fairly large cluster, I can say that the answer is "It depends" (Of course that is ALWAYS the answer!). It depends on the codes being run, it depends upon the interconnect optimization.(yes myrinet is fast, but the real key is that it has much lower latency and this has to be engineered carefully if using more than one switch) My cluster runs both myrinet and Gig/E, some codes run well on the the ethernet interfaces (take codes like mpiblast for instance) while others (NAMD comes to mind) run faster on the myrinet. However this machine may be fast, but I have some large SMP boxes (IBM P-series) that cycle for cycle SMOKE the performance of the x86 boxes. But you have to remember that the cluster computers cost about $3000 /node while the SMP boxes with a similar config cost about $13,000 apiece, and even more if you want a box that supports more than 8 CPUs (think 1 million and up)
So once again, it comes down to the types of jobs, and how much you are willing to pay to get those jobs done in a hurry! A Cluster is still great, I have just completed some jobs that consumed over 12 years of CPU time, in 1 week of wall-clock time!

Re:BlueGene (0)

Anonymous Coward | more than 9 years ago | (#10239449)

12 years=52*12 weeks=624 weeks. That's 624 nodes, assuming 1:1 parallelisation (which is ridiculous). Using your quote of $3K apiece, that's almost two fragging million dollars! Probably closer to 1K nodes, which gives a nice round $3*10^6.

Fairly large, indeed!

mpiblast... NIH? Celera?

Re:BlueGene (1)

psavo (162634) | more than 9 years ago | (#10239660)

I doubt it's UP, most probably 2xSMP, so 312 nodes and about $1M. Which sounds reasonable for 12 years worth of computing in two weeks.

Re:BlueGene (1)

fitten (521191) | more than 9 years ago | (#10240442)

I'd mod the parent up if it weren't at 5 already :)

He's pretty much dead on the money there. "Beowulf" in the strictest sense doesn't have Myrinet though, only commodity parts like 100 or 1000 Mb Ethernet. In these configurations, any latency bound application will be horrible (typically fine grained parallelism, lots of messages, typically small, being transferred). The latency of 1GbE vs 100MbE isn't that much different and both are an order of magnitude or more slower than Myrinet or any of the high performance interconnects.

Having ported MPICH to the T3D and the T3E in the past, as well as ported MPICH to a number of other platforms and written an MPI (and MPI-2_ implementation from scratch and ported it to many different platforms, I can say that clusters do some things well (as the above poster) such as the somewhat trivial benchmarks of the Top500.org lists and some applications that are similar to that in communication needs. However, there are many problems that run much better on Myrinet and even much better on higher speed interconnects.

You still gotta have some love for something like 6.5usec latency for small messages and 325MB/sec (yes, megaBYTES per sec) bandwidth, even on a supercomputer that is 10 years old (Cray T3E). And there are four such links off of each Cray T3E node. Source [sinistershane.net] and shameless plug.

Re:BlueGene (2, Insightful)

Christopher Thomas (11717) | more than 9 years ago | (#10239374)

All this sounds good and Interesting, and Becker did a tremendous ammount of development in this field. But I was just wondering, what about supercomputers like BlueGene/L which have very fast interconnects. Many supercomputers/distributed systems run MPI based programmes and such programmes need a high interprocess commmunication does anyone one know how good these are in a Bewoulf Cluster?

Anywhere from "terrible" to "almost not bad", depending on how much you're willing to pay for the interconnect network. The point of Beowulf-style clustering is low cost/node, allowing scientific computing to be done with commodity hardware (unheard-of at the time). While using something like Myrinet instead of Ethernet, and careful topology layout, can bring you to the "almost doesn't suck" stage, you'll still suffer heavily in communications-bound problems.

Fortunately, there are many interesting problems with low enough communications load to make commodity technology based clusters very, very useful.

Re:BlueGene (1)

flaming-opus (8186) | more than 9 years ago | (#10239821)

Blue Gene is an extremely clever design in that it uses several interconnect networks all at once. The main memory-memory interconnect is a packetized load-store interconnect arranged in a 3D mesh. Each node also has an ethernet tap for the management network, and a very wide tree network for all-reduce calls. They built their networks with MPI in mind.

The difference between a commodity cluster and something like blue-gene is only a half-step. The codes that run well on blue-gene are MORE like the codes run on clusters than those on a traditional vector super. The CPUs, memory controllers, etc, etc. are commodity parts from the microcontroller world, and they work on high compute, low memory-bandwidth tasks with moderate inter-node communication needs. Blue Gene will likely come in at costs somewhere between those of clusters and those of vector supers, or even traditional MPPs like the altix.

You did not take this thought to the end... (1)

PaulBu (473180) | more than 9 years ago | (#10240558)

The codes that run well on blue-gene are MORE like the codes run on clusters than those on a traditional vector super.

And if you code your application for MPI you can debug/test/optimize it on a cheap cluster. THEN when you start running into communication latency and problems too large to be solved on commodity hardware you can recompile your code on big(ger) iron, like Blue Gene/L.

Paul B.

Especially the version in the Ninnle distro (-1, Offtopic)

Anonymous Coward | more than 9 years ago | (#10239026)

Ninnle and Beowulf...clustering at its finest!

Beowulf seems older than that (4, Funny)

Weaselmancer (533834) | more than 9 years ago | (#10239028)

...doesn't it to you? I mean how long have you been sick of the "imagine a beowulf cluster of those" comments? Doesn't seem like only 10 years would make me that sick of it.

Re:Beowulf seems older than that (2, Funny)

lpangelrob2 (721920) | more than 9 years ago | (#10239390)

I mean how long have you been sick of the "imagine a beowulf cluster of those" comments? Doesn't seem like only 10 years would make me that sick of it.

Depends... it only took 4 weeks for Floridians to get real sick of hurricanes. :-)

Passé? (2, Interesting)

Kurt Wall (677000) | more than 9 years ago | (#10239035)

Could it be that Beowulf clusters, however cost-effective and powerful they have become, are passé now that most universities and research institutions have some sort of COTS-based high-performance computing solutions? Not that Beowulf isn't cool - it is - it just doesn't seem as cool as it used to.

Re:Passé? (1)

maxwell demon (590494) | more than 9 years ago | (#10239661)

You mean, Beowulf is dying?

The choice between Beowulf and Big Iron... (4, Insightful)

corvair2k1 (658439) | more than 9 years ago | (#10239051)

...can be simple. The more complex a problem gets, the more likely you need one supercomputer as opposed to a cluster. It's not elitism, it's just that the problem will probably require a lot of communication between processors.

Any kind of networking solution between computers will never be as fast as a hard-wired bus can be. If a lot of communication between nodes is required, you will spend more time waiting than computing, which shoots efficiency to hell.

Re:The choice between Beowulf and Big Iron... (5, Insightful)

monoi (811392) | more than 9 years ago | (#10239193)

The more complex a problem gets, the more likely you need one supercomputer as opposed to a cluster.

I'm not sure it is that simple. For some problems (e.g. Monte Carlo [wikipedia.org] simulations), a more complex problem means more individual nodes are required, with very little inter-node communication. For other kinds of problem (finite element methods, maybe?), you're probably right.

In other words, the physical structure of the solution depends on the kinds of algorithms that you intend to run: there's not just one `correct' answer.

Re:The choice between Beowulf and Big Iron... (-1, Redundant)

ssbljk (450611) | more than 9 years ago | (#10239218)

yes, yes...
but... anyway, can you imagine beowulf cluster of cray supercomputers?

Re:The choice between Beowulf and Big Iron... (0, Redundant)

Psycho77 (695148) | more than 9 years ago | (#10239627)

The more complex a problem gets, the more likely you need one supercomputer as opposed to a cluster.

Yes, but imagine a Beowulf cluster of supercomputer ! =P

imagine (5, Funny)

justforaday (560408) | more than 9 years ago | (#10239059)

imagine a lone computer sitting by itself not connected to anything...

Re:imagine (1)

shfted! (600189) | more than 9 years ago | (#10239156)

Hey, don't go making fun of my Apple IIe!

Re:imagine (1)

djdavetrouble (442175) | more than 9 years ago | (#10239486)

Hey, don't you go awakening my Apple ][e lust! Now I am doomed to spend an hour on ebay drooling over them.

Re:imagine (1)

blindbat (189141) | more than 9 years ago | (#10239396)

... and I'll show you a safe Windows computer

Re:imagine (1)

caluml (551744) | more than 9 years ago | (#10239457)

One of the funnier posts I've seen on here... :)

Re:imagine (2, Funny)

UnknowingFool (672806) | more than 9 years ago | (#10239876)

imagine a lone computer sitting by itself not connected to anything...

With a little editing that could describe most /. readers:

imagine a lone geek sitting by himself not talking to anyone...

;)

Winterware (2, Interesting)

Eberlin (570874) | more than 9 years ago | (#10239078)

If you could imagine a...ok, well maybe after ten years, we all could. The horse has been so beaten and tenderized that even takko vell wants a piece of the action.

I've never seen a beowulf cluster personally. I've never run anything on one. However I do know that it made "supercomputing" more affordable. That in itself is a feat -- and a primary goal of most Open Source software. A proverbial "Hats off" to the open source hackeres out there. Thanks...and keep hacking.

Now if I can gather enough old 486 machines, will I be able to run Doom III? With WINE? At least it'll keep me warm during the winter months.

Re:Winterware (1)

oberondarksoul (723118) | more than 9 years ago | (#10239963)

While with a lot of 486s you would have a fair amount of processing power, if a program attempts to use instructions introduced into later CPUs, chances are it'll not work.

In Soviet Russia... (2, Funny)

halivar (535827) | more than 9 years ago | (#10239082)

In Soviet Rus-- oh damn. I pulled out the wrong dead horse.

Re:In Soviet Russia... (-1, Troll)

Anonymous Coward | more than 9 years ago | (#10239231)

Okay, I'll finish it, then:

In Soviet Russia, a Beowulf Cluster imagines you.

Re:In Soviet Russia... (-1)

CdBee (742846) | more than 9 years ago | (#10239337)

lucky stop - that joke ALWAYS gets modded down to -1

Re:In Soviet Russia... (-1, Troll)

Anonymous Coward | more than 9 years ago | (#10240011)

In Soviet Russia, the wrong dead horse pulls out YOU.

(AC!)

On par? Yes and no (5, Informative)

grape jelly (193168) | more than 9 years ago | (#10239099)

Beowulf clusters have never been the fix-all solution to pricey supercomputer needs. Traditional mainframe supercomputers will forever have their niche in computing that can't just be muscled through sheer volume of vector processes (i.e., processes in which good latency is essential). Even the creator of the Beowulf cluster agrees:

Quote from the article: *snip!*
Supercomputer vendor Cray has created a new product that is designed to compete with some Linux clusters. Cray Canada CTO Paul Terry said that Linux clusters really can't compare to a supercomputer. What is your take on Cray's moves against Linux?

Becker: They are simultaneously saying that Linux clusters are not high-performance computing systems while introducing a product to compete with Linux clusters. They clearly saw that a large part of their customer base was moving toward commodity clusters, Beowulf-class clusters, to do high-end computing.

Clusters can't replace all of the workload being done by supercomputers today, but it can replace the bulk of the traditional vector supercomputers. There is always that 10% of the market that won't run well on clusters, and that is the market that Cray is in. We are happy to solve most of the problems of the world and run most of the applications and play in our part of the marketplace.

Re:On par? Yes and no (2, Informative)

EnderWiggnz (39214) | more than 9 years ago | (#10239155)

yeh - sometimes, you actually need the pure performace that a vector processor will give you, without the initial overhead of paralelizing a process to run loosely-coupled on a beowulf.

loaded question (1)

flaming-opus (8186) | more than 9 years ago | (#10239881)

Cray has sold linux clusters before, and now have 2 products that use linux in some way. (Red Storm and the XD1) They even have done some experiments runnint linux on the X1 vector supercomputer. Cray certaintly isn't making moves against linux. They would just prefer you to run linux on their mpp box, rather than a rack full of Dells.

Mr. Becker has an interest in you using a penguin computing setup, rather than either Dell, or Cray. I must, however, admire the way he didn't get sucked into the interviewers desceptive question.

On Par Or No, Vector CPUs Got Cluster-fucked (1)

cmholm (69081) | more than 9 years ago | (#10240230)

For around about the last five years, it didn't matter whether clusters were the best or even a reasonable solution to a number of problems, which I'm not sure was only 10% of the then market.

AFAIK, the cluster proponents sold the NSF and the DoD's HPC office on the idea that they would solve the limitations of "pile of PC" systems in software, the result being that both organizations basically mandated clusters for all new projects. Imagine the CIO of an aerospace firm requiring WinNT henceforth for any application of computing resources and you get the idea.

The net result was that advances in vector computing stopped in the US. SGI sold a handful of existing Crays, but that was it. Otherwise, you needed to go to NEC, Fujitsu, or Siemens for iron. The only reason Cray has even a prayer is because there are one or two agencies of the US Federal government that need up to date vector processing, and ain't gonna buy from a foreign supplier that may either cut 'em off or screw with the systems for political reasons.

So, while on one hand Becker and his buddies provided a great service to the difusion of parallel applications to a wider market, their politicking to be the high end solution end-all be-all moved the development and manufacture of high-end systems offshore. Becker is the Sam Walton of parallel computing. Good job, brother.

Hail Clusters! (0, Redundant)

olclops (591840) | more than 9 years ago | (#10239111)

I for one welcome our new beowulf cluster of these linux-running overlords.

Just out of curiosity... (2, Interesting)

Krypto420 (652140) | more than 9 years ago | (#10239154)

I was wondering if it is possible to make some sort of cluster out of old computers I have lying around? Nothing spectacular, just hooking up 3-4 old P2's to make a game server or something of the sort. Is there software out there to do this?

Has anyone had any experience with this?

Just a thought...

Re:Just out of curiosity... (0)

Anonymous Coward | more than 9 years ago | (#10239404)

No. That's absolutely impossible. We'll need a major breakthrough in our understanding of physics and the universe before you're able to do what you are asking about.

Re:Just out of curiosity... (1, Informative)

Anonymous Coward | more than 9 years ago | (#10239612)

ClusterKnoppix [bofh.be] may be just what you're looking for...

Hmmm Can you say HAL (0, Offtopic)

mortation (740766) | more than 9 years ago | (#10239186)

H.A.L.

It's not just about speed and massively parallel (5, Informative)

Wizzy Wig (618399) | more than 9 years ago | (#10239217)

processing...

To be considered a "supercomputer," it also needs enough CONTIGUOUS MEMORY SPACE to hold the massive amounts of data associated with true "supercomputing." So far, no cluster has met that requirement.

help me out (0)

Anonymous Coward | more than 9 years ago | (#10239594)

Can't you just put a big mothertrucking hard disk on one of them and NFS mount it?

Re:It's not just about speed and massively paralle (2, Informative)

vsack (558342) | more than 9 years ago | (#10239724)

There are certain classes of problems that clusters don't map to well. Applications with a very high cost of inter-processor comminucation or that demand a huge piece of contiguous memory are probably always going to be outside the realm of clusters.

However, problems that are embaressingly parallel can be handled by a cluster very adequately for a fraction of the cost of a traditional supercomputer. I don't know that you can ignore this class of problems and say that clusters aren't "true 'supercomputing'".

Re:It's not just about speed and massively paralle (1)

joib (70841) | more than 9 years ago | (#10239812)


To be considered a "supercomputer," it also needs enough CONTIGUOUS MEMORY SPACE to hold the massive amounts of data associated with true "supercomputing."


Well, that's one way of seeing it I guess. A way not shared by most people in supercomputing, I might add.

Re:It's not just about speed and massively paralle (1)

hackstraw (262471) | more than 9 years ago | (#10240129)

Huh? Contiguous in what sense? Attached to the same motherboard? In one DIMM? Addressable in one chunk by the OS?

I've only been to one supercomputing conference, but when I was there most all of the people there ran clusters and the top500 [top500.org] site (although this list is produced by the same supercomputing conference people) lists many clusters there.

In other words, where does this contiguous memory requirement come from?

FUD, FUD, and wrong. Crey stockholder eh? (4, Informative)

Rhys (96510) | more than 9 years ago | (#10240247)

Sad to see this little knowledge about parallel computing on slashdot: blatantly wrong information marked as informative. +5 no less.

Let's address this first: there are two common memory architectures, distributed memory (a cluster) and shared memory (a 'traditional' supercomputer). Each can emulate the other. Saying a cluster doesn't have enough memory, presumably at each node, is really saying: "I don't really understand message passing."

This would be more important if datasets were actually large. Unfortunatly for your argument they aren't. A handfull of nodes and they'll hold the whole simulation easily in memory (albeit it'd take years to run because there's so few CPUs at work.)

How would I know? Well, I work with the Center for Simulation of Advanced Rockets aka CSAR at UIUC, one of five DoE ACSI sites in the country. I manage their supercomputer, which is getting upgraded from 200 P3-class dual proc PCs to 640 dual proc Xserve G5s. Before that I was a grad student working with them, albeit not on the CSAR simulation but instead on a related grant, the CPSD.

Now, there are computing problems which clusters aren't good at (or at least that's the traditional claim. My master's thesis and advisor would seem to dispute that this is actually the case.) However, most problems as the interview says, run just fine on clusters. Physical simulations (which covers CSAR's rockets to the national labs nuclear weapon research to hurricane/weather simulation, all the way down to protein folding and atomic and sub-atomic scale crystal formation simulation) need to know about what's in the area you're working on, and what's in nearby areas.

Occasionally you'll find an oddball like galactic simulation (or molecular dynamics) that needs to compute gravity across the whole universe. Fortunatly we have multigrid methods and a friendly gravity equation to solve this problem: get real data from those near you. Average those far from you and use that instead.

Then of course there's the idea that even "traditional" supercomputer problems that don't run well on clusters can be run efficiently on clusters IF you move beyond 1 process per CPU. Load up 10, 20, 100, 1000 little workers on a processor. Get fast context switching between them (not OS level!). Use message passing rather than shared memory (locking, ick!) to communicate. One worker blocked waiting for network data? Process the next one! If you've tuned things right you'll find you always have work to do.

Sounds crazy? Supercomputing '02 didn't think so: http://charm.cs.uiuc.edu/research/moldyn/

How fast will it run Doom3? (2, Interesting)

ARRRLovin (807926) | more than 9 years ago | (#10239247)

That is what I want to know!

In all seriousness though, what is the ratio of cluster to big iron in supercomputing nowadays? I know a clusters can scale out to a lot of FLOPS, but what is the highest FLOPS processor available?

Re:How fast will it run Doom3? (0)

Anonymous Coward | more than 9 years ago | (#10239472)

I know a clusters can scale out to a lot of FLOPS, but what is the highest FLOPS processor available?

I hear the Itanium processor is one massive FLOP. A cluster of them...

IN SOVIET RUSSIA... (-1, Troll)

oldosadmin (759103) | more than 9 years ago | (#10239264)

Beowolf isn't a cluster, he's a mythological hero, you clod. So I guess... Beowolf would CLUSTER YOU!

New coolness (1)

gowen (141411) | more than 9 years ago | (#10239301)

"Imagine a Beowulf cluster..." jokes are passe. The really hip respond to mentions of Donald Becker's name with oblique references to Steely Dan [wikipedia.org] records.

Re:New coolness (0)

Anonymous Coward | more than 9 years ago | (#10239445)

Actually, the two guys who are Steely Dan are DONALD Fagen and Walter BECKER....

"...you crossed the diamond with the pearl, you turned it on the world, and then you turned the world around..." or something like that...

Imagine.... (5, Funny)

drkich (305460) | more than 9 years ago | (#10239313)

Imagine there's no cluster,
It's easy if you try,
No adapter below us,
Above us only loopback,
Imagine all the computers
computing for themselves...

Imagine there's no internet,
It isnt hard to do,
Nothing to download or upload for,
No porn too,
Imagine all the computers
computing pi in peace...

Imagine no tokens,
I wonder if you can,
No need for ethernet or tcpip,
A brotherhood of computer,
Imagine all the computers
Sharing nothing at all...

You may say I'm a dreamer,
but I'm the only one,
I hope some day you'll leave us,
And the computers will computer alone.

copyright violation (1)

commodoresloat (172735) | more than 9 years ago | (#10240240)

I'm pretty sure Steve Jobs owns that song.

But can you imagine (-1)

Anonymous Coward | more than 9 years ago | (#10239339)

A beowulf cluster of a beowulf cluster of supercomputers? That'd be something!

Cluster Schedulers (2, Informative)

Anonymous Coward | more than 9 years ago | (#10239365)

And GridEngine is free and opensource:

http://gridengine.sunsource.net/

Otherwise known for... (2, Informative)

jaylee7877 (665673) | more than 9 years ago | (#10239421)

Donald Becker also has done a large amount of work on Linux Network drivers. Grep through linux/drivers/networking and you'll find he's done work on Intel NICs, Realtek 8139s, even the ne2000 (I think he said he puked a few time while working on that one). Thanks for all your hard work Donald!

newbie question re: ease of management (2, Interesting)

SABME (524360) | more than 9 years ago | (#10239516)

A question for beowulf-savvy folks:

At the end of the article, the comment is made that one reason for setting up a cluster is ease of management (for updates, applications, etc.). Can anyone with experience comment on whether this is true or not, with the way clustering exists today? I have no experience at all with cluster, and I'm wondering if this is something I should look into to ease administrative burdens?

Imagine... (0, Redundant)

FusionDragon2099 (799857) | more than 9 years ago | (#10239535)

A Beowulf cluster of Beowulf clusters!

Lets imagine the oposite (1)

genner (694963) | more than 9 years ago | (#10239635)

Imagine all the pathetic individual computers you'd have if you take apart a bewoulf cluster.
I think they did this in soviet russia.

openMosix (2, Informative)

241comp (535228) | more than 9 years ago | (#10239677)

Beowulf isn't the only game in town folks. A much easier to maintain and balance cluster can be built using openMosix [openmosix.org] . openMosix is a single-image-cluster extension for Linux.

Re:openMosix (1)

wertarbyte (811674) | more than 9 years ago | (#10239852)

OpenMosix is quite cool, I tried it once and it really impressed me. Although there are no paches for 2.6, does anyone know where the development of this great project is heading?

Re:openMosix (1)

skelley (526008) | more than 9 years ago | (#10240046)

Don't forget OpenSSI [openssi.org] as well. This is a single-system-image clustering product with a long lifespan and great support within HP.

Re:openMosix (0)

Anonymous Coward | more than 9 years ago | (#10240512)

openmosix is nice, but the shared-memory facilities are lacking (and still considered experimental patches? I should check but I'm so lazy).

openssi is an contradictio in terminus: it claims to aim for single-system image clustering by adding even more bloat to the horrible linux kernel... go figure.

what is seriously lacking in both beowulf and openmosix are serious queueing and batch control mechanisms, you only find those for mpi/pvm.

Thank you Donald (1)

martser (716335) | more than 9 years ago | (#10239873)

He has been such a huge contributer to the Linux platform.

Thanks,
Eric

Why Beowulf? (2, Insightful)

Trogre (513942) | more than 9 years ago | (#10240022)

If you maintain a group of networked but otherwise independent computers for example a student lab or office farm, consider deploying something like PVM or MPI. It's a great way to get some use out of those idle cycles.

PVM at least scales incredibly well: 25 machines rendering a povray scene take just a fraction over 1/25 the time taken to render it on one machine. I haven't tested MPI yet.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>