Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Building Your First Cluster?

Cliff posted more than 8 years ago | from the it's-been-a-while dept.


An anonymous reader asks: "I'm interested in building a DIY cluster using Linux and will be using conventional Linux software. However, the number of possible ways to do this is huge. Aside from Beowulf, there's Mosix, OpenMosix, Kerrighed, Score, OpenSSI and countless others. Therein lies the problem. There are so many ways of clustering, development seems to be in fits and starts, most won't work on recent Linux kernels and there's no obvious way to mix-and-match. What have other people used? How good are the solutions out there?"

cancel ×


Sorry! There are no comments related to the filter you selected.

Don't sweat it ... (5, Funny)

tomhudson (43916) | more than 8 years ago | (#15788001)

Try them all. After all, you just KNOW your first one's going to be a clusterf*ck.

Seriously, if you're going to take that route, you really should be prepared to invest the time in test-driving several different solutions.

Re:Don't sweat it ... (1)

megaditto (982598) | more than 8 years ago | (#15788126)

a clusterf*ck.
Or a Cluster bomb []

Re:Don't sweat it ... (0)

Anonymous Coward | more than 8 years ago | (#15789429)

for a *really* low cost version look at clustercompute []

first post (-1, Offtopic)

Anonymous Coward | more than 8 years ago | (#15788003)


Well, uh... (1)

keesh (202812) | more than 8 years ago | (#15788010)

It kinda depends upon what you want to do. What are the requirements for your cluster? Have you considered going massively SMP instead? Or are you just after a cluster for the sake of having one?

Re:Well, uh... (1)

MyDixieWrecked (548719) | more than 8 years ago | (#15788041)

I've found that most people I know that wanted to build a cluster, when asked why, replied something to the effect of "for the coolness factor... and to compile things faster."

I tried to set up 5 node (7 processor) distributed compile farm for a while which let me build gentoo packages with blazing speed. unfortunately, I couldn't get cross compiling to work, nor could I get XCode integration working, so in the end, I had a 400mhz G3 and a 800mhz G4 doing distributed builds with distcc [] and a 1ghz and a dual 600mhz Pentium3 set up with distcc as well.

my G5 was stuck in OSX and xcode, but it's fast enough, I guess. =P

Re:Well, uh... (-1, Flamebait)

Anonymous Coward | more than 8 years ago | (#15788117)

I'm absolutely amazed that you took time out of your busy schedule to tell everyone how much you suck at cross-compiling. That must be extremely humbling.

Re:Well, uh... (4, Insightful)

kimvette (919543) | more than 8 years ago | (#15788091)

"massively SMP" does not provide fault tolerance and does not eliminate certain bottlenecks such as disk I/O and network throughput, so if it's for an extremely high volume/high availability fileserver, mail server, or web server, massive SMP isn't going to cut it.

Also, I'd go a render farm (if that's the task) if I had to choose between clustering and SMP, because if one node dies (depending on the managing application) the job just continues, whereas if it's on one single monster machine with no fault tolerance, if the job dies you often have to start rendering again from the beginning. Not fun.

So let's back up and ask:

  1. What problem are you trying to solve?

  2. If it's a learning experience, try them all, take notes on which suit you best for tasks a, b, and c,

  3. What are your priorities

Rocks (5, Informative)

corvair2k1 (658439) | more than 8 years ago | (#15788013)

Rocks [] is a great tool to build a cluster. It includes lots of monitoring tools and such, so you can see the status of nodes, etc. However, I'm not quite sure how large you're planning on going... May be overkill for a 4-noder. =)

Re:Rocks (4, Informative)

kst (168867) | more than 8 years ago | (#15789752)

No, Rocks isn't overkill for a 4-node cluster; I'm running three 4-node Rocks clusters myself.

I'm not familar with other solutions, but Rocks is remarkably easy to install.

Re:Rocks (2, Funny)

alc6379 (832389) | more than 8 years ago | (#15792525)

So would you say that Rocks Rocks?

Thanks... I'll be here all week!

Find the problem before trying to solve it (4, Informative)

linuxkrn (635044) | more than 8 years ago | (#15788038)

Sounds like you're trying to solve a problem that you don't have.

Cluster need special software to take advantage of the disturbed computing. They are built with a specific task in mind. Or do you already have a need and just failed to tell us?

For me, I run my network with distcc ( So all of my Gentoo boxes can compile using shared computing power. It cut a typical 33Min app down to less then 2 mins doing this. And works wonders for my slower laptop.

With distcc, all you need to do is have the same tool chains. (glibc, gcc, coreutils, etc) You can even specify how many threads per box you want running to fine tune your network.

On the other hand, if you just want to learn, then you should try them all. The all suit different needs.

Re:Find the problem before trying to solve it (3, Funny)

Daxster (854610) | more than 8 years ago | (#15788062)

Cluster need special software to take advantage of the disturbed computing.

Well, indeed, clusterf*cks might turn your distributed computing project into a disturbed one..

Re:Find the problem before trying to solve it (1)

BobNET (119675) | more than 8 years ago | (#15788196)

For me, I run my network with distcc ( So all of my Gentoo boxes can compile using shared computing power. It cut a typical 33Min app down to less then 2 mins doing this. And works wonders for my slower laptop.

And don't forget, ccache [] can work with distcc [] , for an even bigger speedup...

Re:Find the problem before trying to solve it (3, Insightful)

Fishbulb (32296) | more than 8 years ago | (#15788243)

Cluster need special software to take advantage of the disturbed computing. They are built with a specific task in mind. Or do you already have a need and just failed to tell us?

And specifically, is this a processing cluster or a failsafe cluster? I kind of assume a processing cluster, since that's what most people on slashdot refer to as a cluster, but in my experience most of the clusters out there are failsafe clusters ("5 9's" of service versus raw horsepower). Two rather different applications of clustering, requiring different design philosophy and sometimes different clustering applications.

OB Mony Python ref:
"I don't know what you mean, an african cluster or a euopean cluster?"

Re:Find the problem before trying to solve it (1)

eflanery (210795) | more than 8 years ago | (#15788447)

I.e. do you want a cluster that acts like a mainframe, or one that acts more like a supercomputer?

You will need to answer "why?", before anyone can really help you with "what?" or "how?".

Re:Find the problem before trying to solve it (2, Informative)

Sithgunner (529690) | more than 8 years ago | (#15788650)

He's apparently not asking for compile farm...

Author did fail to say what the purpose was, but here are some good starts.

Apache cluster []
MySQL cluster [] (should also refer to resources)
Ultra monkey [] , heartbeat and the like [] can make cluster as well.

Re:Find the problem before trying to solve it (1)

LWATCDR (28044) | more than 8 years ago | (#15790511)

Well honestly the problem could be he wants to learn about clusters. Not a bad ambition but also kind of very broad.
If learning is his goal he could use qemu Xen or VMWare and create a virtual cluster on one really fast Linux box with a good amount of ram. He could also try out different clusters all with out hooking up five or six old boxes.
Just to add to the confusion I would like to mention the could use Plan 9 for his cluster since it is distributed by nature.
I would love to set up an OpenMosix Cluster with LTS but I lack the time, space, and spare machines to do it.

Re:Find the problem before trying to solve it (1)

musselm (209468) | more than 8 years ago | (#15792268)

Cluster need special software to take advantage of the disturbed computing.

Nicely put; sometimes I feel like my computing is disturbed, too.

Re:Find the problem before trying to solve it (1)

kimvette (919543) | more than 8 years ago | (#15794883)

How to build a disturbed cluster:

  1. Cluster unpatched Windows 2000
  2. Install spyware
  3. Install SQL Server (unpatched)

There's your disturbed cluster.

Or, for another form of disturbed:

  1. Move to Sv^Hweden
  2. Start legal torrent site based in Sweden
  3. Wait for our govermnent (Bush Administration) to coerce the Swedish government into breaking the law by illegally siezing your servers. That'll be a disturbed cluster!

Xboxen (1, Funny)

Anonymous Coward | more than 8 years ago | (#15788040)

hella X-Boxes...

Easy decision (0)

Anonymous Coward | more than 8 years ago | (#15788081)

Beowulf... cluster. Must... be...

BEOWULF!!! book is a good reference (2, Informative)

Anonymous Coward | more than 8 years ago | (#15788114)

You may want to look at this online book (free): []

At least get to know various approaches at a high level before proceeding...

Whatever happened to... (3, Interesting)

techno-vampire (666512) | more than 8 years ago | (#15788116)

I remember in the '80s, back at JPL, they were talking about connecting a number of computers into a hypercube. One node would be for I/O, and it would communicate with the others along the edges of a hypercube.

Two computers make a 1 dimensional "cube." Four, in a round-robin make a square. Six, properly connected, make a regular cube and so-on. Does anybody out there know if they still connect clusters this way and if not, why?

Re:Whatever happened to... (5, Informative)

sophanes (837600) | more than 8 years ago | (#15788312)

While a few early parallel computers used hypercube based interconnects (eg, CM-1), there hasn't been a lot of interest in hypercubes since then. The advantage of hypercubes is that their diameter only increases logarithmically with the number of nodes in the network. Their disadvantage is that the node degree increases with the dimension, meaning that each node must be configured with sufficient ports to support the maximum dimensionality of the network -- making hypercube-based networks either expensive or non growable. (This can be solved to an extent by using cube-connected cycles.)

Re:Whatever happened to... (0)

Anonymous Coward | more than 8 years ago | (#15788483)

Or a hypercube of hypercubes (of hypercubes (of hypercubes (of hypercubes)))

Lispification G

Re:Whatever happened to... (3, Informative)

SirLoadALot (991302) | more than 8 years ago | (#15788326)

No, cluster network links are not nearly as elaborate as that, for the most part. The advent of high-speed crossbar links means that only the largest clusters need to even consider the network structure. You can get a 288 port Infiniband switch that provides full crossbar connectivity across all data paths. Only if you outgrow the largest available switch do you need to consider how to best link the multiple switches together. Generally this would involve minimizing the amount of IO that would need to be performed between the switches, which isn't always easy, or even possible.

Re:Whatever happened to... (1)

andrewman327 (635952) | more than 8 years ago | (#15790468)

Very true. When you think about distributed computing, you realize that people's (comparatively) slow Internet connections can still prove useful to clusters.

Re:Whatever happened to... (1)

0racle (667029) | more than 8 years ago | (#15788443)

Two computers make a 1 dimensional "cube."
This is more commonly known as a line.

Re:Whatever happened to... (2, Informative)

BigFootApe (264256) | more than 8 years ago | (#15788907)

Some high-speed interconnects like SCI and Dolphin are designed to be deployed in ring based structures (hypercube is based on a bus). The multidimensional analog to the ring is the hypertorus, and many clusters based on SCI and Dolphin use a hypertorus topology.

For instance: []

Re:Whatever happened to... (1)

sophanes (837600) | more than 8 years ago | (#15789475)

Hmm? Hypercubes are generally constructed using point-to-point links, not buses. Unless by bus you mean bidirectional links, but toroids often use bidirectional links as well.

But you're right, toroids constitute the most popular k-ary n-cube architecture today. Folded 3D toroids (eg. by Avici) are especially sweet (uniform, short interconnects, low diameter, huge path diversity).

Re:Whatever happened to... (1)

BigFootApe (264256) | more than 8 years ago | (#15794479)

Unless by bus you mean bidirectional links, but toroids often use bidirectional links as well.

Poor choice of word. I mean a bi-directional, linear arrangement of nodes. :)

It doesn't have to be p2p, I believe. One could implement a hypercube using ethernet switches.

Re:Whatever happened to... (2, Funny)

N. P. Coward (953833) | more than 8 years ago | (#15790002)

It all came to a halt when they had 27 boxes [] nicely stacked up... Some joker came along and painted primary colors on all the outside faces and this was irresistable to some other jokers who began moving the boxes around, making quite a tangle of all the cords.

build a beowulf ... (1)

hanulec (37611) | more than 8 years ago | (#15788130)

beowulf: [] or []

i'd recommend looking at the ease of building diskless clusters w/ warewulf.

Re:build a beowulf ... (1)

boyter (964910) | more than 8 years ago | (#15788219)

Since I cant figure out how to reply to the article I will chuck something here (perhaps someone can help a slashdot noob here). 10 [] []

Re:build a beowulf ... (1)

thrillseeker (518224) | more than 8 years ago | (#15788524)

Since I cant figure out how to reply to the article I will chuck something here (perhaps someone can help a slashdot noob here).

Just underneath the article at the top of the page is a line where you can change the threshold you read the articles at. There is a "change" button to the right of that line, that submits the changed threashold. To the right of that is a "reply" button - that is to post a new subject in the article thread. It took me a while to find it too.

My Experence (3, Informative)

PAPPP (546666) | more than 8 years ago | (#15788183)

I have a stack of five origional Pentium boxes with 32mb of RAM and 2gb harddrives (except for one, with a larger drive for a software repository). Origionally built it to experiment with AFAPI [] based clustering, but since AFAPI is a reasonably non-invasive setup, it works well for trying other techniques too, everthing from simply running distcc [] on the nodes to speed up i586 software builds to briefly fiddling about with some of the other clustering options mentioned. Fiddling around with options on a real cluster (running cluster software on a single node really isn't a good impression) that could be reinstalled from scratch in a few hours, and the machines aren't worth enough to matter if it is physically damanged is a great way to learn.

I am in the exact same position. (1, Interesting)

B5_geek (638928) | more than 8 years ago | (#15788226)

I have 5 boxes of somewhat random configuration that I am looking to do the same thing as the OP. My goal for this cluster would be high-speed video processing (using tovid).

sweating it out.. (4, Interesting)

tempest69 (572798) | more than 8 years ago | (#15788245)

ok a few caveats (I cant spell)

1. I do beowulf, other clusters arent my thing. 2. I can handle C and C++, but I'm not a guru. 3. I can fumble my way through unix-linux but I get cranky with new versions (command/ flag changes in utilities). 4. I have 6 lazyish years as a unix sysadmin.

getting prepped,

You want to make sure that the boxes that are talking to eachother are very secure from the rest of the world. Most of the concepts on a cluster are about trashing the security of the machines in question. There are ways to make a secureish cluster, but a good firewall is a better way to go. let the firewall talk to your "head node", and preferably locks the "body nodes" from seeing the outside world. There are a ton of ways to get this done. on the cheap have all the body nodes have a non-existant gateway ie, set the firewall as (forget dhcp) and let the head node point to the,, and have the firewall route services (ssh, ftp, telnet (ok not ftp or telnet)) to the head node.

getting started

1. Load all the boxes with the same OS. (the same way) (DONT select SELINUX or you will cry) 2. build a hosts file (names for machines) /etc/hosts 3. build a hosts.allow , hosts.equiv (still in /etc) 4. add in some entries into securetty for your rlogin rcp rsh.. 5. youll probably have kerberos(weakly secured) rlogin rcp rsh... you want to rename those and replace them with the non secure versions, there are other ways, but this saves a bunch of hassle. 6. pop into /etc/pam.d and adjust the rlogin rcp rsh.. (this may not be needed in some cases). 7. add in a + + in the .rhosts file of each cluster user.

after you have pulled your hair out decyphering my glossed over instructions, you should be able to type: rsh node002 and be at the prompt for node002 with no password asked, and no silly kerberose failed: trying /usr/bin/rsh message given.

At this point then you can configure LAM (you may nee to download it and get it installed on your boxes)

basically it needs an arbitrary file Mynodes.txt that will contain the list of nodes you wish to launch. you type in lamboot Mynodes.txt and then it will kick back some silly error 99% of the time because something small was forgotten. you fight through those errors until it finally gives you a sucess.

Now your golden, then its just a matter of figuring out how to compile and run MPI programs with the mpiCC and mpirun. But if you got through the first gloss over then the rest is a snap.

Remember if these machines see the outside world they are naked, defenseless, and totally exploitable..

Be aware that these instructions can cause all sorts of havok and any reasonable person would just hire an expert.

Honestly I hope that this gives you a starting point. You'll still need a bunch of time with google.



oops glossovers.. (1)

tempest69 (572798) | more than 8 years ago | (#15789459)

Some other silly parts that if your doing a pure DIY beowulf that might be important.

set up an NFS share on the head node, it will simplify a whole bunch of data collection. oh, dont overwrite the home directory as your share, it will cause weird issues, make the directory something like "clustershare" it will make sure that you launch the exact same executable on all the boxes.

The LAMRSH variable needs to be set to "rsh" otherwise you get an error message.


Most of this guide is if your looking to build a cluster to understand how they work, quite a bit of the software out there can make things quite a bit easier to initially set up, easier to maintain, and simpler to diagnose an issue.

I'm a fan of Warewulf, it has a reasonable memory footprint, a nice node management system. As far as diskless boxes go I've found it to be really doable. And learning how to build a bootable disk image is really gratifying (but not manditory). I havent played with "Rocks", but it seems to get rave reviews.

As far as MPI programming goes, it provides control over when and how you communicate, without totally leaving you at a bare metal interface. Systems that attempt to hide those layers may be nice, but leaves you wonderin if something wonky is making the system slowdown.


Re:sweating it out.. (0)

Anonymous Coward | more than 8 years ago | (#15790529)

Not enabling SELinux is a poor choice. The security benefits it provides far outweigh the rumored complexity. Running the default targeted policy on RHEL systems is a cakewalk. Only about 10 daemons are "confined" and any additional applications you install are "unconfined". I think what happened a long time ago was some poor bastard tried to use SELinux, couldn't figure out how to use the /selinux/booleans directory, and didn't know how to run the security level configuration tool, and found that by default you can't use Apache to serve files out of user's home directories. One tick in the gui, or just using setsebool could have fixed it in a snap. But nooooooo, the person gave up immediately, started crying on a mailing list, and soon it became trendy to disable the best security to hit the server and desktop environment...

Re:sweating it out.. (0)

Anonymous Coward | more than 8 years ago | (#15791318)

Not enabling SELinux is a poor choice.
For a desktop or an internet facing server maybe, but did you read the rest of his instructions? The whole point is to make it as simple and painless as possible for the clustered machines to talk amongst themselves and it's up to you to keep them safe and isolated from the rest of the world.

Re:sweating it out.. (0)

Anonymous Coward | more than 8 years ago | (#15791413)

Security should be provided even in a environment isolated from the Internet. Your argument would suggest that you should just "chmod -R a+rwx /". But as you said... security such as users/groups and discretionary access controls aren't needed when you're not on the Internet ;)

Re:sweating it out.. (1)

kimvette (919543) | more than 8 years ago | (#15794965)

I've known people to insist on chmod -R 777 / on OS X - in a production environment.


Because it doesn't work like Mac OS. Mac OS let you share out an entire volume and drop anything, anywhere, any time. Learning is hard!

Rocks Rocks! (4, Informative)

Frumious Wombat (845680) | more than 8 years ago | (#15788339)

No, seriously, if you're setting up a cluster where your work can be batch-queued, or intend to run MPI, then Rocks [] is the way to go. It also comes with tools such as SGE (Sun Grid Engine) or OpenPBS pre-configured, Intel compilers and libraries ready for you to drop a license onto (but of course the entire GNU suite is there as well, including Ada), has more monitoring tools (plus some nice web-based interfaces) than you can shake a stick at, and runs on IA-32/AMD-64/IA-64 (Itanium). It also has a Roll to help build a tiled display wall, which would be a really cool use of a small cluster.

They're also really great guys.

On the other hand, Oscar is supposed to be good, and if you're not into the whole batch-mode thing, you can get OpenMosix up and running using [] ClusterKnoppix, and just fire jobs off into space and let them find their own unburdened node.

But still, Rocks is really an elegant and clean way to go, plus it will scale up in case you're going to deploy a huge one of these for real after you get your feet wet.

Re:Rocks Rocks! (1)

fbartho (840012) | more than 8 years ago | (#15789408)

Hey! Do you have any more details for that tiled display wall and where to look for that sort of thing? I have a bunch of simple machines and associated monitors. I don't have hefty computing needs, but a tiled display wall would be just the thing... Got any pointers?

Re:Rocks Rocks! (1)

Frumious Wombat (845680) | more than 8 years ago | (#15790779) z/4.1/ []

I'll admit, I've never tried this because I couldn't get anyone to spring for the displays, or the space to stack several monitors, but just follow their instructions, and you're off and running.

What about? (-1, Troll)

Anonymous Coward | more than 8 years ago | (#15788354)

Just to give you an insight []

"cluster" means lot of different things (2, Insightful)

tcpipgeek (991310) | more than 8 years ago | (#15788414)

I am pretty much summarizing what has been already mentioned. Cluster is somewhat of an ambigous term which means tons of different things. You really need to specify what you want to achieve before any meaningful suggestion can be provided.

This is a good reference: []

java (1, Informative)

Anonymous Coward | more than 8 years ago | (#15788444)

Do a Google search on "Java heterogenous cluster". I have been playing around with some of them.. Nice things - you can use any machine. No lengthy build out process. Downsides - slower than dedicated, not as flexible.

Re:java (0)

Anonymous Coward | more than 8 years ago | (#15788537)

It really doesn't need to be Java for that. You can do the same thing with any interpreted / byte-coded language, even functional ones like Lisp.

Slower, sure. But not as flexible?, abstraction and code reuse promote flexibility! Getting bogged down in technical platform-specific minutia is what leads to rigidity, it's what OOP (even Java-style) is all about.

Depends on what you're doing (3, Informative)

mrsbrisby (60242) | more than 8 years ago | (#15788563)

Clustering means too many different things these days. I operate several clusters- but they're all so different that I can't say that any of them are the same.

I run ClamAV and Spamassassin- both very slow programs- with cexec [] which simply lets me farm regular unix tools across multiple (lots) of CPU servers. This lets me replace the clamscan and spamc programs with "wrappers" that use my farm. I like cexec because it doesn't make me create lists of clients and servers, but automatically load balances and fails out very nicely.

For my frontend web servers, I use fake/heartbeat and some custom proxy software for routing frontend requests to backend farms.

I haven't found a real reliable replicated directory- with one, I could use cexec as a filesystem... Maybe some day...

Re:Depends on what you're doing (2, Insightful)

TheRaven64 (641858) | more than 8 years ago | (#15790022)

The article submitter seems to have decided two things:

  1. They want to use Linux.
  2. They want a cluster.
In general, these are the two things that should be decided last. Other posters have addressed the 'why do you actually think you need a cluster' issue, so I will take a look at the 'why do you want to run Linux' bit.

If what you want is reliability, then nothing beats OpenVMS. You have to pay a premium for hardware that can run it (VAX, Alpha or Itanium only), but if you really need that much reliability then it might well be worth it.

If you want a compute cluster, then Solaris might be a better bet. Most of the Linux clustering distributions still use (heavily patched) 2.4-series kernels, which leaves you without things like AIO. The Solaris cluster management tools have been progressively refined for well over a decade, while the Linux equivalents are in their infancy.

Finally, if what you really want is a big computer, then take a look at DragonFly BSD in the next few months. Their aim is to build a system designed for Single System Image clustering (i.e. pretending you have a big NUMA computer, rather than a lot of little nodes) and they have made a lot of progress recently.

Building a cluster is not too hard. Maintaining one is. Our OpenMOSIX Linux cluster really needs a full time technician to dedicate 40-60% of their time to it.

Diskless OpenMosix (3, Informative)

rwa2 (4391) | more than 8 years ago | (#15788582)

I worked for a linux supercomputing startup way back when. The easiest time I had was by separating the components : one big machine for storage, and lots of little diskless machines for computation.

So I'm a Debian fan, so that involves just creating one large computer (or two with redundancy using linux-ha) with a good RAID as a shared home directory. Then just install the "diskless" package. This will allow you to spawn off as many hosts that mount root off of NFS as you like. All you have to do is get the compute nodes to boot a kernel that supports nfs root (I used a single floppy, but you can do bootcds or net-boot if you're more sophisticated).

I used a Mosix kernel at the time, but I suppose OpenMosix is a better bet today. Mosix pretty much makes the entire system look like a massive SMP, so you just launch a whole lot of batch scripts on one computer, and it automatically distributes the load out to idle machines, and ships the results back to the one you started on. You just choose a node to become the master diskless-image, and then use the diskless scripts to clone it as many times as you like.

The compute nodes could have a local drive, but I just used them for swap and maybe local /tmp. It's damn convenient to be able to configure all the nodes from one place whether they were online or not.

The other nice thing about OpenMosix is that it's architecture agnostic, so you could conceivable join and remove nodes that were all different speeds, AMDs or Intels or maybe even 64-bit platforms in any combination. The faster processors would get the heaviest loads first, etc.

After you have this system up and running, you might start playing with more sophisticated stuff, like parallel distributed global filesystems and the like. But before that you could certainly make your NFS root server scale up by splitting it up across multiple machines (for /home , /usr, etc.). You'd be amazed at the performance you can get with a well tuned NFS share... since one server can cache most of the disk access, it can even dish out files from one big high performance RAID faster than if you had bothered to give all the nodes their own drive or two.

Anyway, it's the systems management that will get you... so I recommend using Debian, getting real cozy with aptitude, and searching the apt repository for all of the little command and monitoring thingies that will help you, like clusterssh and cfengine and nagios2 and stuff.

Burning a bunch of ClusterKnoppix CDs will pretty much get you on track with most of this, I'd imagine. Also check out the "KNOPPIX Remastering" howto so you can customize your own livecds, should you choose that path insteads of diskless nfsroot.

So that's a software approach, the hard part is really selecting, testing, and optimizing all of the hardware. The slowest component is always going to be storage (invest in lots of separate SATA cards using the Linux software RAID5 or RAID10 - reconfigure and test lots with hdparm -t and bonnie++ and format reiserfs), followed by network (gigabit NICs are cheap - you could afford separate ones for the NFS and the "real" network, though gigabit switches are still up there - Linksys and D-Link make some good 16-port ones for ~$300).

Um, if you're looking for parallel applications, povray is fun. And for a time we could sort of measure how many nodes I had up and running by monitoring my stats at distributed-net . But with OpenMosix, just about anything with lots of CPU-intensive parallel batch processing is fair game and works out of the box.

Have fun!

OSCAR (3, Informative)

Odocoileus (802272) | more than 8 years ago | (#15788667)

Last year I built a cluster using OSCAR []
I haven't tried any others, but OSCAR installs pretty easy. Just run the installer on the head node, and when it is done it feeds an image to each of the other computers that are a part of the cluster. It includes the ganglia monitoring tools and the apache server so you can view it.

better reason (2, Insightful)

tezbobobo (879983) | more than 8 years ago | (#15788721)

Common people. It really saddens me that te only reason people can think for doing this is rendering, compiling, and coolness. Maybe, and I'm wishing more than expecting, the guy is compiling a new breed of kernel for super gaming. I think the most fun thing to do now is assume he is doing it from a gaming point of view and move into fun, spectulative hypothesising. If it doesn't help the poor guy, then at least it may give him some muc cooler ideas.

Imagine a... (1, Funny)

Anonymous Coward | more than 8 years ago | (#15788881)

oh, wait, never mind.

warewulf-cluster (2, Informative)

Imp- (40963) | more than 8 years ago | (#15789194)

I have used [] for my Opteron cluster. Works with new kernels and with many different distros. Seems to be under good devopelment.

OpenMosix LiveCD (3, Interesting)

dargaud (518470) | more than 8 years ago | (#15789278)

First, here are some notes about the first cluster I build [] 3 years ago, and most of it is still relevant.

This being said, for an instant trial, there are some OpenMosix LiveCDs, such as Quantian or other variants of Knoppix. Put the Quantian DVD in the 1st PC, boot, enable the remote boot option, boot the other computers over the network. Here: you have an operational cluster.

I think there may also be Rocks LiceCDs but haven't tried them. And remember your electricity bill when playing with clusters !

Beowulf! (2, Insightful)

sp1nm0nkey (869235) | more than 8 years ago | (#15789425)

What do you want to DO with it? To get one thing straight, beowulf is a distribution, and bproc/mosix/lam/mpich are ways of getting apps to communicate over a cluster. What technology you use is going to depend on the app. If the app is written for mpich, you have to use mpich. If it's written for bproc, you need to use bproc. If you're writing it, look around at the various technologies and see which one you like the most. MPI is a smallish layer above sockets that allows you to explicitly pass messages from cluster node to cluster node. Bproc allows a program to fork across the nodes of a cluster and then join back together. For just getting started (and I'm probably biased since I work for scyld), Beowulf is awesome! The latest distro is about to go beta fairly soon. It installs on top of redhat, and right after you install, you can power up the nodes and if they're set to pxe, all the nodes will come up as compute nodes. It comes with MPICH and Bproc and a few interesting demos (tachyon, a raytracer; and a fractals program) and linpack. The only bad thing about Bproc is that it has to patch the kernel. However, it works very well. I've heard bad things about OpenMosix, it does some fairly bad things like migrating file descriptors and some other silly things. The main thing is it's pretty much a hack to get threadded applications not intended to be clustered to distribute across nodes, which is just... not a good thing. Applications should be written to work in a clustered environment. Anyway, have fun!

Try some LiveCDs (0)

Anonymous Coward | more than 8 years ago | (#15789515)

There are plenty of clustering LiveCDs out there. Hacking Knoppix* has a chapter covering the installation and use of ParallelKnoppix. PK is astonishingly easy to set up yet pleasantly powerful. I'd definitely recommend playing with a few of these CDs/DVDs. Which one is most appropriate depends on the problem you're trying to solve. There's also the slight detail of just how god-awful complicated designing and writing clustered code is, but that comes after some really fun demos. My favorite is, by far, the MPI Mandelbrot viewer covered in that chapter. It's just so damned pretty to watch.

* You probably don't need to buy the book for the content in the chapter, though.

Disclaimer: I wrote the chapter, but I don't make any money if you buy the book. So, by all means, skim it at Borders, and write a nice review of that chapter specifically on Amazon so it makes my resume look better =)

Microsoft (0)

Anonymous Coward | more than 8 years ago | (#15789572)

Just use Microsoft's new HPC solution, no really.

OpenMosix (1)

gweihir (88907) | more than 8 years ago | (#15789678)

I tried it on a cluster with 22 nodes. After 2 years I threw it out. One node had one weak bit and it was impossible to find with OpenMosix, as the errors would just seemingly happen on all nodes.

Re:OpenMosix (1)

Slashcrap (869349) | more than 8 years ago | (#15789708)

I tried it on a cluster with 22 nodes. After 2 years I threw it out. One node had one weak bit and it was impossible to find with OpenMosix, as the errors would just seemingly happen on all nodes.

I would say that if you're not smart enough to do either of the following :

a) Run Memtest.

b) Remove one node at a time and test.

Then you don't need a cluster anyway. So it sounds like it turned out alright in the end.

Re:OpenMosix (1)

Omeger (939765) | more than 8 years ago | (#15806520)

You are a GODDAMN NIGGER FUCKTARD, go suck a donkey cock and fucking die you nigger kike!

Perfect example of when to use VMware (1)

zerus (108592) | more than 8 years ago | (#15790294)

For learning purposes, you can save a lot of headaches by setting up a VWware linux setup and do snapshots so you can roll back when you do something that screws up the system (had a guy try to set up an NFS server and ended up formatting the /usr directory, how that happened, I don't know). It's also quite nice as a testing platform since the vmplayer is free. What that means, is if you have a room full of computers that never get used (as I do in my undergraduate student's computer lab), I can run the vmplayer over windows as my user and let the students login separately. This is also kinda nice so that I can do a scripted active hosts file that updates the cluster setup so that I can have 2 or 20 machines without touching a conf file myself. Surprisingly, there isn't much cpu power lost when running the VM over windows, just network lag since I do the NAT option so I don't have to register new IP's. I run MPI to drive a distributed version of MCNP (neutron transport code) on my personal cluster (Rocketcalc with extra OTS PC's as extra nodes) and do the VM for fun/testing. Other than that, look at the Gentoo HPC howto's, MPI/PVM/LAM manuals, and try to plan out your cluster before starting. It's much easier to get things to work when you know what you're trying to do beforehand than to bumble through it and say "wouldn't it be cool if..."

Warewulf (1, Insightful)

Anonymous Coward | more than 8 years ago | (#15790553)

So far, no one has mentioned. Warewulf [] .

I have built three Warewulf clusters in the past year. I like how light weight and customizable WW is. It consists of a bunch of scripts that netboot/etherboot/PXE boot a custom RAM disk as your root file system from a tftp server (in my case the head node). (The smallest RAM disk we have built is around 10 MB. Everything else can be NFS mounted so each of the nodes has the capabilities of a standalong workstation.) From there you can configure it to do whatever you want. (Corollary: if you want to do it all, it may be better to start with one of the heavier cluster distributions like Rocks. I prefer my systems to be lean and mean.) By default it assumes that you are going to run "diskless" [1] but that can be overridden. I use the local disks as swap and /scratch.

[1] One of the biggest headaches in running a cluster is ensuring that each of the nodes are the same. Configuration creep is one of the banes of administering a cluster. Running diskless is a major advantage because it is easy to bring a node up to date...just reboot. It only takes 30 seconds for my nodes to reboot and be back up. (It would be about 10 seconds shorter if the BIOS would allow me to turn off the checks for Promise SATA RAID that I am not using.) Another advantage is that there is no local state so I can swap out a failed node fast with no hassle.

WW currently works best on RedHat systems, but ports for Debian and other Linux distributions are in the works. (My last two have been Debian based, but I had to do a lot of tweaking to get it right.) The community is very active and the principles are very involved in helping you be successful.

Give it a try.

my first cluster setup (4, Interesting)

Wornstrom (920197) | more than 8 years ago | (#15792206)

For my first diskless cluster, I used the warewulf cluster solution [] to see one up and running. Then, I wiped the master node's disk and built one with the openmosix kernel patch etc, and used the Linux terminal services project [] which was really cool. the ltsp stuff made the node filesystem stuff easy to build onto. I am waiting for the openmosix team to finish up work and release the userland tools for the 2.6 kernel for my next build. here is a good how-to on LTSP+Openmosix - []

Once you get that going, you might look at PVFS2 [] Parallel Virtual File System. "PVFS2 stripes file data across multiple disks in different nodes in a cluster. By spreading out file data in this manner, larger files can be created, potential bandwidth is increased, and network bottlenecks are minimized."

Good Luck!!

Bullet Proof Linux is the way to go (1)

Henkc (991475) | more than 8 years ago | (#15792476)

Do yourself a favour and contact the team at Bullet Proof Linux [] .

We needed a cluster for load balancing a typical web application, with OpenSSI [] being the chosen SSI system. Sadly, OpenSSI is far from a "working" solution, and needs quite a bit of massaging - especially if you have newer equipment.

The guys at Bullet Proof Linux have been professional, helpful and incredibly patient in their efforts to get us going. Worth every cent.

Any they did it all remotely via ssh from the other side of the planet...

sol or xgrid (1)

capsteve (4595) | more than 8 years ago | (#15795729)

if you really want grid computing, use an out-of-the-box solution like xgrid, or a sun based solution. why linux? if not for some special OS-specific reason, sun or OSX will privide better success in the cluster realm. if you want HA, look into everything fiber with FC subsystems and brocade and heartbeat. please, this is not flame bait, just tired of the "linux everywhere" mentality. using the right tool for the job is more important...
Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?