Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

A Diagnosis of Self-Healing Systems

michael posted more than 9 years ago | from the heal-thyself dept.

Operating Systems 149

gManZboy writes "We've been hearing about self-healing systems for a while, but (as is usual), so far it's more hype than reality. Well it looks like Mike Shapiro (from Sun's Solaris Kernel group) has been doing a little actual work in this direction. His prognosis is that there's a long way to go before we get fully self-healing systems. In this article he talks a little bit about what he's done, points out some alternative approaches to his own, as well as what's left to do."

cancel ×

149 comments

Sorry! There are no comments related to the filter you selected.

The challenge of a truly self-healing system (3, Funny)

IO ERROR (128968) | more than 9 years ago | (#11153985)

Your operating system provides threads as a programming primitive that permits applications to scale transparently and perform better as multiple processors, multiple cores per die, or more hardware threads per core are added. Your operating system also provides virtual memory as a programming abstraction that allows applications to scale transparently with available physical memory resources. Now we need our operating systems to provide the new abstractions that will enable self-healing activities or graceful degradation in service without requiring developers to rewrite applications or administrators to purchase expensive hardware that tries to work around the operating system instead of with it.

Neither the applications nor the OS should depend on the other providing any failover or self-healing services; they should always be prepared to go it alone if necessary (as it might be the failover system). Services that crash should restart themselves, etc. This part is pretty well done by most enterprise-grade server software. It's the operating systems we're waiting to play catch-up.

And I'm still waiting to see any box that can replace its own power supply after someone flips the 115/230 switch. Once we get that, then we'll have truly self-healing systems. And all you BOFH's out there might be looking for a new career...

Re:The challenge of a truly self-healing system (4, Informative)

grahamsz (150076) | more than 9 years ago | (#11154052)

Plenty of Sun's boxes have redundant power supplies.

If something goes wrong with one, the system should detect either too little or too much DC voltage or current coming from it, and switch to it's backup.

Your suggestion doesn't make much sense. Should mozilla know what to do if a usb mouse fails or is removed unexpectedly? Of course not, the mozilla developers expect that this will be taken care of.

Likewise when an correctably memory or disk error occurs... The memory controller or disk firmware should deal with it and the application should be none-the-wiser.

Re:The challenge of a truly self-healing system (0)

Anonymous Coward | more than 9 years ago | (#11154122)

Missed Joke... nice job.

Joke Spoiler (2)

IO ERROR (128968) | more than 9 years ago | (#11154318)

Plenty of Sun's boxes have redundant power supplies.

Click here to ruin the joke. [ntk.net]

Re:Joke Spoiler (-1)

Anonymous Coward | more than 9 years ago | (#11154725)

Ah, the old BOFH rubbish -- not a funny joke then, you sad nerd.

Re:Joke Spoiler (-1)

Anonymous Coward | more than 9 years ago | (#11154973)

A nerd? On SLASHDOT? Whoever would have guessed????

Back in the cave, troll.

Re:The challenge of a truly self-healing system (1)

MooseGuy529 (578473) | more than 9 years ago | (#11154729)

Should mozilla know what to do if a usb mouse fails or is removed unexpectedly? Of course not, the mozilla developers expect that this will be taken care of.

Of course not... the point is not that each layer (peripheral, BIOS, kernel, application) can handle errors in all other layers. The point is that Mozilla should be designed to be able to recover from crashes without help from the kernel, BIOS, or anything else. Likewise, if a USB mouse somehow gets "confused" (protocol-wise) it should take the initiative to re-register itself with the system; if the BIOS notices that the system is frozen (through a watchdog), it should reboot it; if the kernel notices that some part of the BIOS has stopped responding (e.g. IDE bus won't work) it should reboot. The idea is for every layer to be able to recover from errors that spring from its own responsibilities--in other words, MySQL is responsible for keeping a database, so it should be able to do that relatively robustly, but if the underlying media is corrupted, it should not take it upon itself to rebuild the RAID array!

Self-healing == human brain (0)

Anonymous Coward | more than 9 years ago | (#11154067)

"And I'm still waiting to see any box that can replace its own power supply after someone flips the 115/230 switch. Once we get that, then we'll have truly self-healing systems"

You may know of the JTAG boundary scan originally implemented as a form of system test to go beyond the limits of 'nail bed' hardware testing. Developing on this circa 1993 were several ideas that extended the hardware self-test model to include self-heal redundancy such that components that failed the test could switch out and switch in a secondary circuit. What I am reading here is encouraging, instead of having a separate 'test and monitoring' circuit for the 'main' device the device itself (and its software) becomes a coarse grain distributed system in which each node can both opearate as part of the devices global behaviour or monitor (test) and switch other nodes within the system. There is one other device that uses this highly distributed and non-specialised approach to fault tolerance, we call it the brain.

Re:The challenge of a truly self-healing system (1)

isurge (574887) | more than 9 years ago | (#11155028)

.... my thoughts have always been the computer is a neuron .. so we designed our system (server farm) so that neurons are backed up and if one fails we simply restore it to a fresh server (fully automated -- simply follows the logic path we humans follow) we use DCH or Distributed Cheap Hardware -- aka we go to a local computer store/online and buy the biggest bang for buck stuff(linux only) we can find average server life time is about 2.5 years or about $10 a month --- our only missing part is an automatic purchasing system ... and that is because our suppliers ordering systems vary so much ... my 2 cents .... validation: our servers presently average about 60 connections per second and have went as high as 1400 connections per second (Nov 3rd 2004) - no it is not Google size but that is nor our goal.

As a Tech... (1)

ZSpade (812879) | more than 9 years ago | (#11154004)

So in the future, instead of out-sourcing our tech jobs to India, they'll simply in-source it to the computer it'self..

What'do I care, I'm out a job either way.

Re:As a Tech... (2, Insightful)

Rew190 (138940) | more than 9 years ago | (#11154172)

If your future depended on merely fixing computers, it was a bad one in the first place.

Re:As a Tech... (1)

ZSpade (812879) | more than 9 years ago | (#11155203)

True, Glorified service men, that's what we are. We all have dreams, I wanna be a writer, I'm just pointing out that one of my day jobs might be dissapearing in the not so distant future.

Re:As a Tech... (1)

StikyPad (445176) | more than 9 years ago | (#11155551)

Don't be such a pious jackass. Aside from your questionable use of the past tense in reference to the future.. Simply because someone picks a different career path or has a different vision of what they want out of life than your own, doesn't make their future bad. Not everyone has the luxury of picking a trade with future job security -- some people just need jobs, now. If people weren't teaching your children, paving your roads, constructing your buildings, fixing your car, or whatever other occupation you consider to lead to a "bad future," you'd be doing these things yourself or going without.

Had this 3 years ago (4, Interesting)

shoppa (464619) | more than 9 years ago | (#11154012)

According to a documentary movie from 3 years ago, we already had this. HAL 9000 sent an astronaout out to help repair the antenna azimuth control board.

Which turned out not to be faulty... hmmm...

Some IBM mainframes are already at this level of self-diagnosis. Where I work, IBM repairmen show up with spare drives for the RAID array when they fail and the array phones IBM to report the fault. We don't know that a drive failed until the field service tech shows up!

Re:Had this 3 years ago (0)

Anonymous Coward | more than 9 years ago | (#11154034)

IBM ~ HAL

UH OH!

Re:Had this 3 years ago (0)

Anonymous Coward | more than 9 years ago | (#11154269)

It's a conspiracy I tells ya. I mean we all know the IBM shifted right thing, and that Aurthur C. Clarke says that it is just coincidence, but come on [internetnews.com] , and now this article.

*puts tinfoil hat on*

Re:Had this 3 years ago (0)

Anonymous Coward | more than 9 years ago | (#11154077)

According to a documentary movie from 3 years ago, we already had this. HAL 9000 sent an astronaout out to help repair the antenna azimuth control board.

Which turned out not to be faulty... hmmm...


Thus proving it's at least as easy to program artificial hypochondria as it is artificial intelligence.

"I'm sorry, I can't do that, Dave. It hurts too much. I think I'm coming down with a virus."

Remote Monitoring (1)

grahamsz (150076) | more than 9 years ago | (#11154085)

Is a little bit different from self-healing, but they are in the same vein.

I believe Sun are working on systems that will attempt to spot failure trends, so they can proactively identify other customers who may run into similar problems and then either have the system fix itself or send someone out to deal with it.

The other mindset i've seen with RAID disks, is why bother replacing them. Disks are getting to the point that it's probably cheaper just to leave the dead one in there and power up a spare than to dispatch someone to install a new one.

Re:Remote Monitoring (1)

gomoX (618462) | more than 9 years ago | (#11154910)

Actually, decent RAID systems have hot spares installed. When a disk dies, the hot spare takes it's place - you just remove the dead drive in order for it to become a new hot spare, and not to get the array to it's original state.
You can't just "not replace dead drives" unless you have like 400 SCSI controllers on your machine, therefore providing an insane amount of hot spares for future failures.

Re:Had this 3 years ago (3, Interesting)

jomas1 (696853) | more than 9 years ago | (#11154168)

Some IBM mainframes are already at this level of self-diagnosis. Where I work, IBM repairmen show up with spare drives for the RAID array when they fail and the array phones IBM to report the fault. We don't know that a drive failed until the field service tech shows up!

Interesting. Where I work this happens too except instead of IBM techs we get sent techs who work for the city and instead of finding out that they were sent for some good reason, 90% of the time it turns out that the techs were sent for no reason. The techs usually don't even know that a machine called in a service request and waste a lot of time asking me why they were called.

If the future holds more of this I hope I die soon.

Similar to IBM's Autonomic Computing (2, Informative)

bhadreshl (841411) | more than 9 years ago | (#11154266)

Well this seems like where computing services are heading as IBM is doing extensive research on Self-Configuring, Self-Healing, Self-Optimizing, and Self-Protecting computing systems called 'Autonomic'

Check out: Autonomic Computing [ibm.com]

Re:Had this 3 years ago (1)

Aardpig (622459) | more than 9 years ago | (#11154465)

HAL 9000 sent an astronaout out to help repair the antenna azimuth control board.

Unfortunately, the astronaut (one 'Dave') wasn't able to comply, because HAL refused to open the pod-bay door.

Re:Had this 3 years ago (1)

NeuralAbyss (12335) | more than 9 years ago | (#11155465)

HAL9000? Bah.. they should've used a Mac.. it would have been compatible with the iPod bay door..

Synonym for this (0)

Anonymous Coward | more than 9 years ago | (#11154746)

Extortion.

One system this will never work on (1, Insightful)

mboverload (657893) | more than 9 years ago | (#11154029)

This will never work on Windows. With all the registry crap it has, I don't see anything like this working. The registry is a nightmare to fix if anything goes wrong, it is ALWAYS easier to reinstall. In fact, I'm reinstalling XP tomorrow because of all the crap and bugs it has accumulated. I do this at least twice a year, and its a shame.

Re:One system this will never work on (0)

Anonymous Coward | more than 9 years ago | (#11154057)

sounds like an end user problem

Re:One system this will never work on (1, Funny)

Anonymous Coward | more than 9 years ago | (#11154203)

Perhaps that's a feature Micro$oft are going to provide in Longhorn: self-reinstalation. :p

Re:One system this will never work on (1)

jproudfo (311134) | more than 9 years ago | (#11154656)

They already do this. Check out the OS deployment feature pack for SMS 2003. :)

http://www.microsoft.com/downloads/details.aspx? Fa milyId=3E51FD48-C412-48C9-942D-648914C2759E&displa ylang=en

Actually... (0)

Anonymous Coward | more than 9 years ago | (#11154255)

I recently bought a refurbished amd64 HP computer on the cheap.

I was surprised and almost impressed by the "self-healing" nature of the pre-installation.

When booting the computer, you can press F10 to tell the BIOS to get the customized OS files on the hidden partition of the hard drive and overwrite the existing files on C:. This can also be done right from Windows by clicking an icon. Note that it could well be a chemotherapy type of self-healing (with the registry especially)!

I promptly installed Gentoo though, so I wonder if I can make pressing F10 do emerge sync; emerge world automatically!

HP computers are probably a good purchase for regular Windows home users.

Re:One system this will never work on (0)

Anonymous Coward | more than 9 years ago | (#11154504)

Um..

Use XP's System Restore. You can roll back to a previous working state.

This will fix any registery issues you have.

Personaly I'm a linux fan. I like it a lot. But you can tweak a windows platform [2000 or XP] to be pretty darn stable if you've a mind to. In my own experience, I find I have to reinstall Mac OS X far more often than Windows. Simply because of the kinds of software I run on it (poisined and co.) Tends to be the same thing for windows.

I've had plenty of luck with self healing (1)

Neil Blender (555885) | more than 9 years ago | (#11154050)

I use a little method I like to call the crontab coupled with shell scripts.

Self-Healing Systems (1)

MonkeyCookie (657433) | more than 9 years ago | (#11154053)

It looks like the T1000 won't be appearing any time soon: at least not until Skynet comes online.

TiVo (3, Insightful)

Radak (126696) | more than 9 years ago | (#11154071)

TiVo has had self-healing Linux systems out there for five years now. There are virtually no complaints of TiVo software failure (hard drives certainly go bad from time to time, but very rarely does the OS get itself into a state it can't fix), so the notion that self-healing systems are still years off is silly. They may not be extremely advanced yet, but they're certainly out there.

Re:TiVo (1)

daeg (828071) | more than 9 years ago | (#11154113)

Yes, self healing and reactive systems exist for specific implementations, but not as a whole. If the kernel or some low level drivers in a TiVo went bad somehow, I doubt the TiVo could repair that. I doubt we will see self-healing desktop boxes or general purpose servers for at least a few more years. I think that is what the article is trying to drive home.

Not really (2, Insightful)

grahamsz (150076) | more than 9 years ago | (#11154128)

It's very easy to make a system self-healing when you are running in a completely controlled evironment.

Indeed my TiVo very rarely crashes and always recovers, but the same is also true of every embedded system i've used - be it a cellphone, weather station or alarm system.

Now if i screw around modding my tivo then it's entirely possible to crash it and it doesn't recover very well from that...

Re:TiVo (1)

Daniel Ellard (799842) | more than 9 years ago | (#11154189)

That's a nice accomplishment, but it's not the same thing. TiVo has complete control over the software they run on their boxes, and I'm sure they test it quite carefully before shipping. This isn't unique to TiVo; you could say the same thing about the software that runs in your cell phone, DVD player, etc.

These people are in a different domain; they don't know what apps their system will run, or what mistakes the sysadmin will make, or what worms someone will write next month -- they're preparing a reactive defense against the unknown.

Re:TiVo (0)

Anonymous Coward | more than 9 years ago | (#11154305)

Someone at work gave me a TiVo that, after an "upgrade" would no longer boot (green screen of death). I attempted to fix it by putting in a different disk and loading a fresh image on it. It worked for a while, but eventually gave the green screen of death again. So much for self-healing.

Re:TiVo (1)

sysadmn (29788) | more than 9 years ago | (#11154878)

There's a big, big difference between designing a system that can do one thing well, and a system that can do many things well. Worst case with TiVO, it restarts itself and you wonder why a few minutes of your program didn't record. Worst case with Solaris, your system restarts and everyone wonders where eBay went :-0

Re:TiVo (1)

rainwalker (174354) | more than 9 years ago | (#11154971)

Must be nice...my DishPVR 501 crashes all the time. Normally if you're careful you can avoid it, but woe to the user that tries to change timer settings while it's recording something, or elects to stop watching something as it's being recorded and goes to the PVR menu. I can crash it about 30% of the time doing either of these common activities. Stupid thing takes almost 4 minutes to reboot, too, so you miss a big chunk of whatever you were trying to watch, plus it commonly crashes again because it gets confused about being in the middle of a timer.

Of course, I have no idea what OS it uses, but I suspect it's unpatched Win95 from its behavior.

GNU Hurd (0)

Anonymous Coward | more than 9 years ago | (#11154081)

This seems to be the closest current system design that could possibly allow a self healing system when the hurd is setup as a cluster of servers.

the real question is (0)

Anonymous Coward | more than 9 years ago | (#11154082)

if someone wrote a virus that exploited a vulnerability in KDE, would they call it "the Klap?"

My fav self-healing method (1)

xv4n (639231) | more than 9 years ago | (#11154108)

INT 19h [xaff.org]

Re:My fav self-healing method (1)

CmdrObvious (680619) | more than 9 years ago | (#11154126)

My favorite self-healing method is to take to asprin, and call myself in the morning. it works every time, and saves myself thousands in doctor bills. ...wait, that is self-diagnostic too! I am saving even more!!

Re:My fav self-healing method (1)

ThisNukes4u (752508) | more than 9 years ago | (#11154193)

Sorry, but that only works with windows.

The blue button (1)

Otter (3800) | more than 9 years ago | (#11154121)

Networks and servers, they tell me, can self-defend, self-diagnose, self-heal, and even have enough computing power left over from all this introspection to perform their owner-assigned tasks.

After repeated viewing of those Thinkpad commercials where the techs tell the hysterical PHB to press the blue button on startup and thereby enable IBM to magically resurrect his hard drive, I summoned up the courage to try it. (The curly haired guy in those ads is also in one of my favorite commercials ("Please stay on the line...Playing with the queen of hearts..." so I figured I owed it to him.)

I gingerly pressed the button on my T40 and -- nothing happened. Maybe it's the firewall.

Yikes, now all the people who flipped out last month when I said I prefer my TiBook to the T40 are going to go even more nuts! "You ignorant asshole troll, don't you know the only thing that matters on a laptop is a magic blue button...!"

Re:The blue button (1)

0racle (667029) | more than 9 years ago | (#11154190)

Apples do have a magic button to access the HDD if the OS can not boot.

Re:The blue button (0)

Anonymous Coward | more than 9 years ago | (#11154549)

open apple - S

and then somone screams "ARRRRRRRRRRRRRRRRG"

because single usermode is scary.

Re:The blue button (1)

0racle (667029) | more than 9 years ago | (#11154832)

Single User Mode exists in any Unix/Unix like OS. I was talking about when the OS would refuse to start at all, no single user, no login, nothing. The the magic button I was thinking of is the 't' key, which enters target disk mode, you would also need a firewire cable and another Mac.

Re:The blue button (0)

Anonymous Coward | more than 9 years ago | (#11154338)

You need to have the IBM support software installed. It's not very good though.

if (2, Insightful)

Kanasta (70274) | more than 9 years ago | (#11154143)

if self healing = ms office keeps putting another icon in my start menu whenever I start word, then I don't want self healing.

How many times do I have to move their icons to a submenu before they realise I don't want my root menu cluttered up with crap?

Re:if (0)

Anonymous Coward | more than 9 years ago | (#11154198)

Well if we are talking about a self-healing Windows OS, then yes. If we are talking about a self-healing computer, then I imagine the first thing it will do is wipe-out the Windows partition and install Linux.

Re:if (0, Troll)

elid (672471) | more than 9 years ago | (#11154253)

Does that make Windows self-destructive?

Re:if (1)

upsidedown_duck (788782) | more than 9 years ago | (#11154736)

if self healing = ms office keeps putting another icon in my start menu whenever I start word

It's much better than that. Self-healing means that disks in a RAID array can detect corrupted blocks of data using checksums and correct them from good mirrors on-the-fly. With multiple mirrors with checksums proving whether there is a problem or not, corrupt data files should be a thing of the past (on systems with RAID). It seems failing drives would be detected sooner, also.

Re:if (0)

Anonymous Coward | more than 9 years ago | (#11155380)

I think I speak for all of us when I say:

What the hell are you talking about?

How about systems that I can manually heal first? (3, Insightful)

grumbel (592662) | more than 9 years ago | (#11154161)

While a self healing system sounds nifty, todays systems aren't even good enough to be healed manually.

Uninstalling applications is often not handled by the OS and has to be done by application itself, resulting in incomplete installations, config files and registiry entries that havn't been properly cleaned up and whatever.

Files arn't versioned, so every change done to a file will simply erase the former content forever, not so good if the former content might have been important.

Undelete? Nope, we don't have that either, we have this hack of a Trashcan, but that won't help you much if some programm deleted the file.

Check of integritiy of an installed piece of software isn't possible either, sure there are third-party solutions, but again that should be something that the OS provides at default

Well, there are millons of more issues why todays system suck and why it is often easier to simply reinstall from scratch then to try to actually fix the mess, and yep, that is true for both Linux, Windows and MacOS, sure for some more then for the others, but thats it.

Re:How about systems that I can manually heal firs (1)

upsidedown_duck (788782) | more than 9 years ago | (#11154753)


Mostly, that's because Windows is a piece of shit.

Re:How about systems that I can manually heal firs (1)

upsidedown_duck (788782) | more than 9 years ago | (#11154818)


Sorry about that, I just said the first thing that came to my mind.

Re:How about systems that I can manually heal firs (-1, Flamebait)

Anonymous Coward | more than 9 years ago | (#11154890)

Moron.

Re:How about systems that I can manually heal firs (0)

Anonymous Coward | more than 9 years ago | (#11154850)

Files arn't versioned, so every change done to a file will simply erase the former content forever, not so good if the former content might have been important.

VMS had this years ago.

Windows XP also has the 'restore' utility (which I've had to use a couple of times when one of their updates hosed the system). This is one feature though that is both useful and worked when I neeeded it to.

Re:How about systems that I can manually heal firs (2, Insightful)

Qzukk (229616) | more than 9 years ago | (#11154877)

Files arn't versioned

Undelete?

Check of integritiy of an installed piece of software

During the desktop's formative years, the raw drive space needed to actually implement these kinds of things just wasn't available. This is why things like file versioning (popular on large systems like VMS, where the universities/companies running it had the money for the storage requirements) and permanent storage of "unwanted" files just didn't appear.

The third problem is a bit tougher without some extra metadata and hardcore discussions on exactly what should be monitored/done/etc (personally, I don't think this is a kernel-level operation). Something must be stored somewhere so that the system can identify a modified binary. At some time (before change, in which case the operation is stopped? After change? Monthly?) someone (root? file owner? script kiddie currently logged in as root?) has to be notified (syslog? message to terminal? email?) that something (virus? script kiddie? make install? dpkg? rpm?) has altered the (executable? configuration? library? manpage?). As you can see, its one thing to say "oh yeah the OS should do this" and another entirely to define what this is.

The second problem is tough as well, but there are patches to libc's unlink() function (either as a patch or as an LD_PRELOAD library to override libc's function) that move the files to a pre-defined trashcan, and that every dynamically linked application will use.

The first problem is mostly just a lack of demand. Nobody cares, so nobody made a filesystem that can do it. Both ext*fs and reiserfs are extendable (with optional options. Reiserfs moreso than ext), so if you care, do it yourself, but again there's questions you'll have to be prepared to answer (and since you insist on doing this at the kernel level, you have to have THE answer): If a program writes 1MB to a file 1 byte at a time, is that one million revisions? If you're writing a document and you hit save after every paragraph, is that a revision? How are you going to tell this apart at the kernel level?

Re:How about systems that I can manually heal firs (1)

grumbel (592662) | more than 9 years ago | (#11155487)

### Something must be stored somewhere so that the system can identify a modified binary.

Well, the system ultimativly knows when something changes, since it is the one who changes it. You are right that one needs some metadata, those however in most cases already comes with the packages (deb/rpm) one installs, there just isn't a standard way to automatically check these changes. However this problem can be solved completly in userspace with a cronjob, would just be nice to have a standard way to do it.

### During the desktop's formative years, the raw drive space needed to actually implement these kinds of things just wasn't available.

While that might be true, that was at least 10 years ago. Since we count in GigaByte we have more space then needed and in most cases more then we ever can legally fill. And even if the harddisk runs full from time to time it isn't much of a problem, a versioning filesystem should simply scale back and automatically discard older versions, but as long as there is still free space on the harddisk I see little reason to not use it for something usefull.

### The second problem is tough as well, but there are patches to libc's unlink() function

The throuble with that patch is the same as with the trashcans, they only get what you actually delete, not what you overwrite, fill with 0 bytes or destroy by other operations. I don't have any hard numbers, but in my experience the deleted files don't happen that often compared to the overwritten ones.

### If a program writes 1MB to a file 1 byte at a time, is that one million revisions?

If it does 'fopen(.., "a"); write() close();' then yep, one million revisions. A new version should be created once the filehandled is opened for writing, something that is hard to catch from userspace. Storing of the new version could then happen in some Copy-on-write style manner, thats why the kernel need to play at least some role.

### How are you going to tell this apart at the kernel level?

Since you only need to listen to open() and close(), how many bytes are written inbetween that is not much of an issue. And about the "nobody cares" part, well, the programmer might not, the users who yet again have lost data however might a lot, but as with all things that are only usefull if things go wrong it of course isn't much good for marketing.

Get a *real* OS (0)

Anonymous Coward | more than 9 years ago | (#11155222)

While a self healing system sounds nifty, todays systems aren't even good enough to be healed manually.

Only when you limit your OS choices to turds. See below.

Uninstalling applications is often not handled by the OS and has to be done by application itself, resulting in incomplete installations, config files and registiry entries that havn't been properly cleaned up and whatever.

Why in hell does the OS have to be involved in an application install?!?!?!

Oh, right. Because there this single-point-of-failure, dumbest-idea-ever-to-fly-out-of-a-monkey's-butt registry where all applications need to put there configuration data.

Why!?!?!?

What "problem" did this solve?

Files arn't versioned, so every change done to a file will simply erase the former content forever, not so good if the former content might have been important.

VMS. Old news. Also lots of disk space needed


Undelete? Nope, we don't have that either, we have this hack of a Trashcan, but that won't help you much if some programm deleted the file.

And why not a Trashcan inside the trashcan. Even better, a popup that will ask if I'm really sure I want to stop using this wonderful Fisher-Price GUI.

Check of integritiy of an installed piece of software isn't possible either, sure there are third-party solutions, but again that should be something that the OS provides at default

Just how in hell is the OS supposed to determine that a third-party application's file hasn't had it's "integrity" destroyed? How on God's good Earth would the OS even define integrity?

Well, there are millons of more issues why todays system suck and why it is often easier to simply reinstall from scratch then to try to actually fix the mess, and yep, that is true for both Linux, Windows and MacOS, sure for some more then for the others, but thats it.

Yeah, but it's really only Windows that sucks. Real apps on real computers don't need a dumbass "registry" so the "OS" can tell them where to find their files and data (but only on that one computer....), they smart enough to do it themselves (it's actually really easy, because an app always knows where it's located, and can use that information to find other data in a completely relocatable manner that doesn't depend on anything other than itself. Cool idea: limit your app's impact on the box it's installed on.....)

Re:Get a *real* OS (1)

grumbel (592662) | more than 9 years ago | (#11155563)

### Only when you limit your OS choices to turds. See below.

Feel free to suggest one, neither Windows, MacOSX, Linux (any distri) nor FreeBSD get the job done properlly.

### Why in hell does the OS have to be involved in an application install?!?!?!

If not the OS, then who else should take care of it? If the application are free to dump themself anywhere there won't without anybody taking care of them doing it properly its just a matter of time since one app will to bad things.

### VMS. Old news. Also lots of disk space needed

VMS, I now, current OS however are still behind that. About diskspace, yep, plenty, my HD and those of basically EVERYBODY else has at least a few hundred megabytes free, often that would be more than enough for recovery of a accidently overwriten file.

### And why not a Trashcan inside the trashcan.

Trashcan only works if the application does use it, most however don't and never will. Beside from that it can't catch overwritten files only deleted ones.

### Just how in hell is the OS supposed to determine that a third-party application's file hasn't had it's "integrity" destroyed? How on God's good Earth would the OS even define integrity?

Application comes with a bunch of MD5 or whatever checksums which the OS can check again.

### Yeah, but it's really only Windows that sucks.

No, all OS do, some just more then others.

### it's actually really easy, because an app always knows where it's located

No, the app does not know where it is located. The only way to do that portably on Unix today are hardcoded path inside the binary, which simply causes endless throuble in case one wants to relocate a application. Registry is of course not the best solution out there, but hardcoding configuration values into a binary for sure is just an ugly hack.

Sod the stupid machines (1)

Timesprout (579035) | more than 9 years ago | (#11154213)

If they break they can be fixed or replaced. I want selfhealing (prefereably wolverine type) for me.

Reset button (2, Interesting)

mboverload (657893) | more than 9 years ago | (#11154229)

I don't know why windows doesn't just have a reset button for all the settings to return it to it's original condition. It's a bitch to reinstall it twice a year, you know.

Re:Reset button (0)

Anonymous Coward | more than 9 years ago | (#11154601)

XP has system restore which does exactly that.

I'm confused (1)

SpeedBump0619 (324581) | more than 9 years ago | (#11154260)

Just out of curosity, can anyone define what the precise difference is between "Fault Tolerance" and "Self Healing"?

Explain to me how any of the failure responses I see discussed in the article or in these discussions qualifies as "Healing"? Almost all fault tolerant systems isolate failing components or programs from the rest of the system (killing rouge processes counts as isolation). Quarantine is not an attempt to heal, it is an attempt to tolerate. Are there actually any non-quarantine "self healing" systems out there today?

Re:I'm confused (0)

Anonymous Coward | more than 9 years ago | (#11154879)

Just out of curosity, can anyone define what the precise difference is between "Fault Tolerance" and "Self Healing"?

I would consider "fault tolerance" is that if something breaks the system keeps running -- but the component remains broken. So if you're considering software, you have databases running on two hosts and if one segfaults all requests go to the other one. The first is still down until and admin has to restart it.

Self healing is that the database sefaults, the system notices it (perhaps through a monitoring processes), and restarts it (if it's determined that doing so won't corrupt data).

"Fault tolerance" is have redundancy, "self healing" means that the system takes care of itself so that an autoside entity (like an admin) doesn't have to.

At least, this is how I understand these terms.

Re:I'm confused (1)

MickLinux (579158) | more than 9 years ago | (#11155182)

I think that when most people talk about self-healing, they mean fault tolerant. An example is the Tandem systems mentioned a little below this. Yet I also think that self-healing and fault tolerance are a bit the same.

However, if you want to understand what self-healing really means (and does not mean), consider that our DNA are self-healing.

Now, I do not claim to understand the mechanisms whereby the DNA is self healing. I am aware that there is a recent article that points out how the DNA breaks get healed after damaging radiation.

http://story.news.yahoo.com/news?tmpl=story&cid=57 1&ncid=751&e=1&u=/nm/20041220/hl_nm/tech_mobilepho ne_health_dc [yahoo.com] )

I might imagine that the DNA causes the production of RNA. In the event of a break, the RNA is then used to repair the DNA. But if the wrong strand of RNA is used, you get an error.

But those DNA breaks are healed with errors intact, sometimes.

Moreover, with bacteria at least, the healing actually causes the bacteria to reengineer its own DNA, so that poisons become food. How this happens, I cannot even imagine.

How might we do this with computers? By overly stuffing memory and HDD with redundant data records, and then in the event of a crash, trying to put the data pieces back together. Even better, it might be interesting to use P2P plus online storage to have computers back each other up. That is, suppose you use a RAID system. Then when your computer crashes, it guesses the gap length, then goes to other drives (and also gets on the internet and asks other computers) to look for identical strings. Then it stuffs the strings, and checks to see if the system works.

Re:I'm confused (2, Informative)

segfaultcoredump (226031) | more than 9 years ago | (#11155341)

Fault Tolerance implies the ability to not just detect the fault (i.e. a failed cpu), but to keep the processes running as if nothing happened. This is possible with Stratus and Tandem boxes. It is genrally not possible with common x86/Power/SPARC boxes (unless you put a lot of software on top of two boxes to make them look like one big virual system).

"Self Healing", in this context, is the systems ability to detect a fault (hardware or software), deal with it (restart a process, isolate hardware, etc) and then get on with life (in a possibly degraded mode). In a way, the venerable Veritas Cluster System is an example of a "self healing" system. (it detects a failure of a service group and restarts it, on another node if needed)

Note that with "self healing" systems, the process may die, and end users may notice a failure. But the system is 'back online' sooner than if it required manual intervention. Compare this to a Fault Tolerant systems that never went down in the first place.

UNIX is the problem. Tandem was the solution. (5, Interesting)

Animats (122034) | more than 9 years ago | (#11154308)

There are operating systems for which "self-healing" is quite feasible, but UNIX is all wrong for it.

The most successful example is Tandem. For decades, systems that have to keep running have run on Tandem's operating system. For an overview of how they did it, see the 1985 paper Why Computers Stop and What Can Be Done About It. [hp.com]

The basic concepts are:

  • All the permanent state is in a database with proper atomic restart and recovery mechanisms.
  • Flat "files" are implemented on top of the database, not the other way round.
  • When applications fail, they are usually restarted completely, with any in-process transactions being backed out.
  • Applications with long-running state are tracked by a watching program on another machine which periodically receives state updates from the first program. If the first program fails, the watching program restarts it from a previous good state.

Every time you use an ATM or trade a stock, somewhere a Tandem cluster was involved.

Tandem's problem was that they had rather expensive proprietary hardware. You also needed extra hardware to allow for fail-operational systems. But it all really does work. HP still sells Tandem, but since Carly, it's being neglected, like most other high technology at HP.

Re:UNIX is the problem. Tandem was the solution. (4, Interesting)

rlp (11898) | more than 9 years ago | (#11154438)

Tandem had a FT Unix division in Austin. One of the teams I managed that was responsible for an embedded expert system that monitored faults in the redundant components of the system. Every component was replicated. Each logical CPU actually consisted of four processors - two pairs running in lock-step. If one CPU in a pair disagreed with it's counter-part, the pair would be taken out of service. The expert system monitored transient faults and would "predict" that a component was going to fail, and could take it out of service. The system had a modem that would "phone home" in the event of a component failure, and a service tech would be dispatched with a part - often before the customer knew there was a problem.

The machines used MIPS processors (supporting SMP) and ran a Tandem variant of System V UNIX. Combine this with a decent transactional database, and application software capable of check-pointing itself, and you have a very robust system. Albeit a very expensive one.

Tandem was bought out by Compaq, and then by HP. When I left, Tandem had quite a few interesting ideas they were working on, but near as I can tell, they never saw the light of day.

Re:UNIX is the problem. Tandem was the solution. (1)

burns210 (572621) | more than 9 years ago | (#11154732)

When can such monitoring capabilities be added to things like Linux, Mac OS X (Server) or FreeBSD? Granted, some hardware is required for part of these redundant services, but not all.

Re:UNIX is the problem. Tandem was the solution. (2, Insightful)

upsidedown_duck (788782) | more than 9 years ago | (#11154800)


Knowing HP, your systems are probably being replaced by Tandem-branded PCs with ECC RAM and software RAID. A rescue DVD will provide instant system rebuilds so downtime is never more than two days.

So how's the health of the system that monitors (1)

melted (227442) | more than 9 years ago | (#11154334)

the health of the rest of the system is monitored, and what are you gonna do if it comes to wrong conclusions?

Everything I Know I Learned From TOS (0)

Anonymous Coward | more than 9 years ago | (#11154484)

So how's the health of the system that monitors the health of the rest of the system monitored, and what are you gonna do if it comes to wrong conclusions?
"Norman coordinate"

Ya, but... (0)

Anonymous Coward | more than 9 years ago | (#11154452)

In Soviet Russia, self-healing systems diagnose themselves.

Re:Ya, but... (0)

Anonymous Coward | more than 9 years ago | (#11154852)

No, In Soviet Russia Self-Healing Systems Heal YOU.

First step to self-awareness and AI? (1)

G4from128k (686170) | more than 9 years ago | (#11154564)

Self-healing would seem to be a critical step toward a self-aware artificial intelligence. Self-healing requires an ability for introspection that is sufficient to identify and correct corrupted internal states. Code that is able to introspect its own behavior and internal structure could lead it to interesting outcomes if tied to a learning algorithm (even a simple hill-climbing algorithm).

It is then a small step to go from simple feedback self-healing mechanisms to feed-forward control mechanisms. Feed forward system would learn (or be told) how certain precurse states and actions lead to fault or non-faulty operating states. A system that learns that particular states and invocations lead to crashes would "learn" to avoid those invocations or correct those states proactively.

Such a system might even begin to show emotional states in the sense that an emotional state is a summary of the condition of the system that guides future action. "Unhappy" systems would spend extra CPU cycles on introspection to try to understand and correct accumulating faulty conditions. Such systems might even get "angry" at other machines such as machines that send spam or worm packets and refuse to communicate with them.

Misread... (1)

brain007 (798589) | more than 9 years ago | (#11154565)

Did anyone else read this as 'A Diagnosis of Self-Heating Systems?'

I think my laptop could cook an egg...

full redundancy (almost) always works (1)

mo (2873) | more than 9 years ago | (#11154569)

From TFA: One approach is simply to make an individual system the unit of recovery; if anything fails, either restart the whole thing or fail-over to another system providing redundancy. Unfortunately, with the increasing physical resources available to each system, this approach is inherently wasteful: Why restart a whole system if you can disable a particular processor core, restart an individual application, or refrain from using a bit of your spacious memory or a particular I/O path until a repair is truly needed?

Because using stuff like stonith [linux-ha.org] or heartbeat works for many more types of failures. Bad network cable? Yup. Power supply? yup. Server Catch on Fire? Yup.

I'm not saying it wouldn't be nice to have the OS route around bad memory blocks or bad processor cores due to some fancy-pants algorithm (without having to rewrite my app). But you're still going to need a redundant server for when somebody trips over the power cord.

Re:full redundancy (almost) always works (0)

Anonymous Coward | more than 9 years ago | (#11154676)

> But you're still going to need a redundant server for when somebody trips over the power cord.

But you can have a robot that actually plugs it back on. Self-healing.

The redundant server is fault-tolerant. Once all spare servers go dark, you run out of medicine.

Re:full redundancy (almost) always works (0)

Anonymous Coward | more than 9 years ago | (#11154920)

I'm not saying it wouldn't be nice to have the OS route around bad memory blocks or bad processor cores due to some fancy-pants algorithm (without having to rewrite my app).

Solaris has done this (at least on SPARC) since at least release 8. You could offline and pull CPU/memory boards out of an E3500 (which are now quite dated) while the rest of the system runs. The OS scehduler will relocate processes on the fly. There does have to be some hardware support for this though.

Linux is only now beginning to do this.

People wonder why you would run Solaris and not Linux, or what makes Solaris "better" than Linux. Those that need this feature need it bad, and there's no easy way to do in Linux (if there's a way to do it at all).

One of my self-healing systems (4, Interesting)

skinfitz (564041) | more than 9 years ago | (#11154573)

I have it so that if one of our firewalls detects an attempt to access gator.com it enrols the machine into an active directory system group which the SMS server queries to automatically de-spyware it with SpyBot.

I'd call that a self healing system. I'm a network admin though so my perception of these things tends to be on a larger scale.

Re:One of my self-healing systems (-1)

Anonymous Coward | more than 9 years ago | (#11154781)

Pretty neat idea. More details, please?

The need for a "self" symbol (2, Interesting)

Etcetera (14711) | more than 9 years ago | (#11154597)


HAL: I've just picked up a fault in the AE35 unit. It's going to go 100% failure in 72 hours.

This is really something that, IMHO, calls for more interaction between the best of the futurists, science-fiction writers, and coders, and other complexity thinkers.

In order for any system to have an understanding of and proper diagnosis of its own operation, it needs to be able to conceptualize its relationship to other systems around it. Am I important? What functions do I provide? What level of error is proper to report to my administrator? Do I have a history of hardware problems? Has chip 2341 on motherboard 12 been acting up intermittently? If so, is it getting worse or better? How have I been doing over the last few days? Is there a new virus going around that is similar to something I've had before?

What good is a self-diagnosing system without a memory of its prior actions?

All of these questions imply some sort of context that will require the system to use symbols to represent "things" in the "world" around it. Clearly, the largest (though perhaps not qualitatively different) symbol will be a "self" symbol.

From there, all you have to do is follow Hofstadter [dmoz.org] 's path and you'll arrive at a system with emergent self-awareness or consciousness.

The end result of this will be something a) very complex and b) designed/grown by itself. You'll have either the computer from the U.S.S. Enterprise [epguides.info] or H.A.L. [underview.com]

Side question: What is CYC doing these days? [cyc.com]

Self-Healing Data Transfer (1)

Orasis (23315) | more than 9 years ago | (#11154624)

For swarmstreaming, we use the Tree Hash EXchange format (THEX) [open-content.net] to provide cryptographic integrity verification down to a single 1KB resolution so we can automatically repair the corruption.

Where does it hurt? (2, Insightful)

Doc Ruby (173196) | more than 9 years ago | (#11154737)

How about just systems that fail *verbosely*, so admins can quickly diagnose them? Once the patient can complain properly, we can get to work replacing the admin doctors with "self-healing" metasystems that use those diagnostics. It will be a lot easier just mimicking the best admins' best practices by automating them, than all this screwing around trying to compile marketsprach like "self-healing" without understanding how it even works in nature.

Re:Where does it hurt? (1)

ElvenSmith (555236) | more than 9 years ago | (#11155153)

I can second that. If just the error message is good/detailed enough, it cuts the time going thru obscure log files etc...
but i doubt many developers think from a user's point of view...

Re:Where does it hurt? (1)

TTK Ciar (698795) | more than 9 years ago | (#11155190)

That is more or less how things have evolved here at The Internet Archive [archive.org] .

Unixes and the services that run on them can be configured to be very verbose in their errors and warnings, and error messages can be used as triggers to check various logs and system states for additional information, but in a nontrivial cluster there are major problems with humans trying to digest this flow of information and make sense of it all.

Better tools help, but beyond a point it just makes sense to try and make the tools themselves react correctly to the symptoms and correct the problems.

This Sun guy's article overlooks the existing mechanisms and methods already available for implementing self-healing systems (though, he's spot-on regarding the need for better abstraction). Hmmm, I write these kinds of tools at The Archive, maybe I should write my own article?

Regarding the "how do you know the monitor is healthy?" issue: you never really know, but you can have the monitor monitor itself and fairly reliably diagnose many of its own problems, and/or have pools of monitoring peers (qv keepalived), and start with "gentle" solutions which won't hurt anything (much) in the case of a misdiagnosis, and ramp up from there.

-- TTK

Re:Where does it hurt? (1)

Doc Ruby (173196) | more than 9 years ago | (#11155265)

You have taken the diagnostics tools to the next level at TIA, keeping the human in the loop, but expanding both the complexity of the diagnostic data, and the tools to process that data. So both the senses and manipulators of the human are bionic :). Maybe that's why TIA is so cool, and reliable. Thanks a lot - with your content, you are keeping the computers as tools in the service of human intercommunication, instead of descending into communications primarily with the machine, as they appear to prefer at Sun. On both sides of the Archive, which winds up a medium for humans to communicate with each other, and ourselves, transcending time and space in every way (except perhaps actually sending a stream of me back to 1970-23-01::Honolulu-Civic-Center [archive.org] ;).

It's a long way (3, Interesting)

jd (1658) | more than 9 years ago | (#11154754)

...from what we have now to the Liberator (DSV-2) from Blake's 7, the Ultimate in self-repairing systems. At the moment, most "self-repair" is in the form of software error-correction and bypassing faulty hardware. (The "badmem" patches for Linux do this, for example.)


The former could be considered self-repair, but it is limited as you don't have to have much in the way of an error to totally swamp most error-correction codes.


The second form isn't really self-repair as much as it is damage control. This is just as important as self-repair, as you can't do much repair work if your software can't run.


On the whole, "normal" systems don't need any kind of self-repair, beyond the basic error-correction codes. Instead, you are likely better off to have a "hot fail-over" system - two systems running in parallel with the same data, only one of them is kept "silent". Both take input from the same source(s), and so should have identical states at all times, with no synchronization required.


If the "active" one fails, just "unsilence" the other one and restore the first one's state. If the "silent" one fails, all you do is copy the state over.


However, computers are deterministic. Two identical machines, performing identical operations, will always produce identical results. Therefore, in order to have a meaningful hot fail-over of the kind described, the two can't be identical. They have to be different enough to not fail under identical conditions, but be similar enough that you can trivially switch the output from one to the other without anybody noticing.


eg: The use of a Linux box on an AMD running Roxen, and an OpenBSD box on an Intel running Apache, would be pretty much guaranteed not to have common points of failure. If you used a keepalive daemon for each box to monitor the other's health, you could easily ensure that only one box was "talking" at a time, even if both were receiving.


The added complexity is minimal, which is always good for reliability, and the result is as good or better than any existing software self-repair method out there.


Now, you can't always use such solutions. Anything designed to work in space, these days, uses a combination of the above techniques to extend the lifetime of the computer. By dynamically monitoring the health of the components, re-routing data flow as needed, and repairing data/code stored in transistors that have become damaged, you ensure the system will keep functioning.


Transistors get destroyed by radiation quite easily. If you didn't have some kind of self-repair/damage-control, you'd either be using chips with transistors which may or may not work, or you'd have to scrub the entire chip after a single transistor went.

Re:It's a long way (1)

liangzai (837960) | more than 9 years ago | (#11155378)

However, computers are deterministic. Two identical machines, performing identical operations, will always produce identical results.

Fuck no! Computers have souls, man. That is because they are so complicated that the deterministic model no longer holds; there's a non-deterministic layer that gives the machines their personal features.

I'd rather have self-healing coffee. (1)

dpbsmith (263124) | more than 9 years ago | (#11154771)

I thought I saw an article about that earlier, but on second glance it turned out to about self-heating coffee. Yawn.

Now, as for self-heating systems...

They've copied Windows (-1)

Anonymous Coward | more than 9 years ago | (#11154807)

Basically, all they've done so far with Solaris is copied the ideas from the Service Control Manager and the Event Logs from Windows, and implemented them on top of the usual Unix shite.

Big deal. This isn't news, this is years old technology.

The article is a pretty good roadmap... (1)

CodeWanker (534624) | more than 9 years ago | (#11154831)

First of all, it has the phrase that pays: "graceful degradation "

Next, it talks about verbose and useful errors, so that a techy can make intelligent decisions about terminating a process, restarting it, altering a file, or some other fix. Presumably, once a tech marks a problem "successfully fixed" by a certain set of actions enough times, the system wiull try those series of actions before throwing an error message.

What will be nice is when the system recognizes what it is it's doing, so it'll have a "what" area and a "how" area in its makeup so it knows what's involved to accomplish the task. Then, if the "How" gets damaged, it can refer to the "what" to reconfigure available resources to meet the desired outcome. And, if the "What" gets damaged, it can be rebuilt by analyzing the "What" part. THAT will be realio coolio.

worst case? (1, Insightful)

bird603568 (808629) | more than 9 years ago | (#11155311)

this sounds good un till somebody meake a worn that uses an exploit that (for the sake of argument say there is one) was/(will be might be) found. The worm tricks the server in to thinking it is severly messed up so it orders a boat load of parts or shuts down or both. the tech shows up and its just a worm. now you have these parts and have to pay up. also the server shut down, now its lost time. did i mention its a worm so it spreads. thats just worst case, but i could be great unless you fix broken servers for a living.

Why the term "self-healing?" (1)

starseeker (141897) | more than 9 years ago | (#11155438)

That makes it sound like people want computers to be able to mechanically fix themselves when they break.

Wouldn't a "self-healing" system just be good at a) reporting what hardware is actually broken on the machine b) automating well defined responses to well defined programs and c) building parallel, fault tolerant hardware at all levels of the system?

As far as I know, even the best AI research hasn't come up with software that can diagnose and fix unknown, first time, bizarre problems. Ultimately, it all seems to be about providing better error reporting and automation of traditional technology, not some magical PC that can fix itself and reprogram itself with no human intervention. Which is great, don't get me wrong, but why the term "self-healing?" Does someone know the rational for that?

Oooh ooooh solaris (0)

Anonymous Coward | more than 9 years ago | (#11155447)

Solaris was mentioned!! Solaris was mentioned!!
Time to get all slashdotty and proclaim the
end of the company. Ooooh Solaris was mentioned!

We already have this... (2, Insightful)

JRHelgeson (576325) | more than 9 years ago | (#11155479)

The space shuttle, as old as it is, has an absolutely incredible computer system that is self healing.

The Shuttle has many thousands of sensors and backup sensors. Each sensor feeds into one of many computer systems. These computer systems talk to each other as more of a committee rather than just passing data amongst themselves. If a computer discovers a fault, another computer will see that fault as well, it will combine data gathered from other computer systems throughout the suttle and each computer system will literally cast a vote on what the best solution should be for the particular fault discovered.

If one computer system suffers a partial or complete failure, the remaining systems will work around the failed system.

This computer system has managed to keep our astronauts alive for every mission, except those two that suffered from a catastrophic mechanical failure. The second of which (Columbia) the computers kept the craft flying until it broke apart completely.

I say not bad for a system designed over 20 years ago!
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>