Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: Networked Back-Up/Wipe Process?

timothy posted about 2 years ago | from the can-you-put-it-in-worm-form dept.

Businesses 253

An anonymous reader writes "I am required to back up and wipe several hundred computers. Currently, this involves booting up each machine, running a backup script, turning the machine off, booting off a pendrive, and running some software that writes 0s to the drive several times. I was wondering if there was a faster solution. Like a server on an isolated network with a switch where I could just connect the computers up, turn them on and get the server to back up the data and wipe the drives." How would you go about automating this process?

cancel ×

253 comments

Homebrew (3, Informative)

Anrego (830717) | about 2 years ago | (#38203420)

Don’t know of any off the shelf software that does this, but should be easy to homebrew if you have the available skill set.

At the very simplest, you could probably build a custom livecd linux distro to automate the process after plugging in the machine and inserting the CD/pendrive. It’s not as complicated as it sounds if you base it off an existing livecd distro!

More complex, you could do PXE if the boxes are capable/configured for it (if not, probably more effort to change the bios settings than it would be to plug in the CD).

You’re probably content just with the backed up files, but I’ll also throw out there that I’ve found a very effective way to back up old machines/drives is to convert them into virtual disk files. Lets you boot up the old machine in a VM and poke around should the need arise. (disclaimer: I’m a dev not a sysadmin, so this is purely from “at home” experience).

Re:Homebrew (3, Insightful)

Anonymous Coward | about 2 years ago | (#38203488)

I kinda lean towards a linux PXE setup too.

Debian FAI (Fully Automated Install) with all the needed setup, can run tasks and such, in a way that would work for you. It takes some setup (PXE/bootp/dhcp + NFS etc), but it's very capable, and might be practical if you need to do "thousands" of machines.

Re:Homebrew (0)

pak9rabid (1011935) | about 2 years ago | (#38203496)

^ What he said

Acronis or Ghost Enterprise (4, Informative)

charnov (183495) | about 2 years ago | (#38203588)

Acronis or Ghost Enterprise can do this with every PC on a single network segment.

Re:Acronis or Ghost Enterprise (2)

ixidor (996844) | about 2 years ago | (#38204098)

alternativly, http://www.fogproject.org/ [fogproject.org] should also be able to handle most of what you asked or. may need to make a custom boot for the wipe process.

Re:Acronis or Ghost Enterprise (1)

Anonymous Coward | about 2 years ago | (#38204204)

Also this [github.com] may help you take system snapshots at a very granular level

Re:Homebrew (0)

Anonymous Coward | about 2 years ago | (#38203618)

I was thinking the same thing. Network boot would be a nice way to handle a lot of parallel operations without having to burn 100s of CDs. Though it will take some manual intervention to set BIOS settings if needed.

It could get tricky with the Live boot automation. Computer have a wide variety of drives, IDE, SCSI, SATA, Flash. You would have to write a very comprehensive script to cover all possibilities. If it gets stuck, it could be set up to log the problem to a central server. Sounds complicated but you need to do a rough estimate tradeoff between developing a solution like this may outweigh the time it would take to manually back up and wipe 100s of machines.

Assume 300 computers
If it takes 30 minutes of manual time per computer, That's 300 * 30 = 9000minutes / 60 = 150 hours.

If it takes you 60 hours to perfect the network boot solution, but reduces the manual time per computer to an average of 10 minutes, that's 300 * 10 = 3000minutes / 60 = 50 hours + 60 = 110 hours. Plus you now have a new automated tool that can be used in the future and potentially expanded on for other uses.

Re:Homebrew (1)

Anrego (830717) | about 2 years ago | (#38203768)

It could get tricky with the Live boot automation. Computer have a wide variety of drives, IDE, SCSI, SATA, Flash. You would have to write a very comprehensive script to cover all possibilities. If it gets stuck, it could be set up to log the problem to a central server. Sounds complicated but you need to do a rough estimate tradeoff between developing a solution like this may outweigh the time it would take to manually back up and wipe 100s of machines.

I guess it depends on how diverse their setup is. If it's a really diverse setup I can see it getting complicated. Would also depend on how they are doing the backup (just imaging the disks should be straight forward, especially with newer versions of udev that handle most of that variety for you). If you need to be mounting partitions and stuff.. then yeah, gets a little more complicated).

I'll do agree that the benifits of this need to be calculated vs just doing it manually. People always do underestimate how long it takes to develop a solution like this. It will take some research, some implementation, troubleshooting, and finally testing.. and it will still take time for the process to run on each box (although now you can do parallel). If we are only talking 10 or 20 boxes, probably not worth it.

Re:Homebrew (1)

Anonymous Coward | about 2 years ago | (#38203622)

This sounds like the best solution. Creating a script and the livecd is the only work. Have a livecd with clonezilla to backup a predefined server. Then when it's done, blow away the partitions and "cat /dev/zero > fill" to a new part until the drive is full. You will do not need to do anything else, Gutmann was paranoid about the theoretical. You can do as many machines as you have thumb drives.

Re:Homebrew (1)

Anrego (830717) | about 2 years ago | (#38203842)

I'll just add, verification of the backups is probably a good idea before you wipe.

For actually wiping the drive, would be just as easy to have something like DBAN on the livecd (and would alleviate any Gutmann inspired paranoia on the part of management/policy/requirements)

Re:Homebrew (1)

Keiichi25 (2520526) | about 2 years ago | (#38203718)

I'm sort of a low-level Sysadmin and I know of no simple or easy process that exists right now for what you want to do. Anrego's suggestion is as close as to something you wanted. I saw someone else post Ghost, which is only good for imaging a drives, however, it isn't ideal for a situation where if you want to 'bring it back' if you don't have the major portion of the 'same hardware' (Namely the Motherboard and its chipset), as imaging back a hard drive or 'transplanting' an OS from one system to another with dissimilar motherboards results in Windows going BSOD due to a harden HAL setup. Also, Ghost was never designed to do Gov/Mil disk wiping like what you are asking. I don't believe anyone has considered doing a software based mass disk wiping, as some resort to the more extreme method of large stack of drives and a large electromagnet to do that process or doing what you are doing right now.

Re:Homebrew (1)

GameboyRMH (1153867) | about 2 years ago | (#38203888)

Correct answer on first post, excellent work!

Turning Linux PCs into VMs is easy but for Windows computers it's a big PITA.

Re:Homebrew (1)

Anonymous Coward | about 2 years ago | (#38204140)

When was the last time you tried to virtualize Windows machines?
Linux is easy if you know how:
-rsync a live system
-stop daemons
-rsync again
-shutdown netwerk on old machine
-enable network on new VM
-send_arp to flush arp caches in network infra.
Resulting in downtime of a few minutes (depending on volume size and network thoughput).
But Windows is far from a PITA, http://www.vmware.com/products/converter/ [vmware.com] does it in the same way. It worked perfectly in a recent switch of virtualization provider for 4 Windows machines.

Re:Homebrew (1)

GameboyRMH (1153867) | about 2 years ago | (#38204188)

I've done XP machines manually, complete with manually pruning system files and hive-loading and modifying the registry to force hardware re-detection...

Re:Homebrew (0)

Anonymous Coward | about 2 years ago | (#38204144)

Correct answer on first post, excellent work!

That tells me something. It tells me the subject was a trivial matter that anyone employed in such a position should also be able to answer on his own.

It's not like it required experts in the field to enter deep discussion and reflect ideas off each other to finally come up with a working solution. No, the first post nailed it just like that. How about some non-trivial Ask Slashdot stories? The idea of Ask Slashdot has so much more potential than this.

FOG (PXE backup/clone) + DBAN (5, Informative)

Kamiza Ikioi (893310) | about 2 years ago | (#38204048)

FOG is a PXE cloning solution. http://www.fogproject.org/ [fogproject.org] Install FOG and storage where you want backups, setup PXE IP on network, and input all MAC addresses you want backed up. Through web interface to clone all. When done backing everything up, put a .img file of DBAN on the FOG server. http://www.dban.org/ [dban.org] Configure it in the FOG PXE boot menu, and make it an option but NOT default. Add appropriate start up flags for the level of wiping you want. Restart all computers you want to wipe, and select wipe option after PXE boot menu comes up.

I suggest you set that option with a password, since it will be available on all computers, not just the one's with the MAC address since only the FOG boot authenticates to MAC, not DBAN.

Are you an hourly employee? (5, Insightful)

Anonymous Coward | about 2 years ago | (#38203438)

Then don't automate it.

Re:Are you an hourly employee? (3, Insightful)

Mythran (2502540) | about 2 years ago | (#38203682)

That's just crap. "Lets be less efficient so we can get more money!" That's not the mindset devs or sysadmin should ever be in. I can't think of a career where less efficient just for greed is a good thing. Always strive to be better than what you are.

Re:Are you an hourly employee? (-1)

Anonymous Coward | about 2 years ago | (#38203762)

I can't think of a career where less efficient just for greed is a good thing.

Public sector.

Re:Are you an hourly employee? (1)

Anonymous Coward | about 2 years ago | (#38203798)

I can't think of a career where less efficient just for greed is a good thing.

You wouldn't make it ten minutes in the public sector then.

"Hey boss, we managed to meet or exceed all our goals AND we're significantly under budget!" "Dammit, we'll get less money next year if you don't find a way to spend the rest NOW!"

Sad but true.

Re:Are you an hourly employee? (1)

avgjoe62 (558860) | about 2 years ago | (#38203840)

I can't think of a career where less efficient just for greed is a good thing. Always strive to be better than what you are.

You, my son, will never have a career in politics...

Re:Are you an hourly employee? (0)

Anonymous Coward | about 2 years ago | (#38203690)

Unless you automate it and kick back. ;) /me lifelong slacker. HA!

Re:Are you an hourly employee? (1)

jellomizer (103300) | about 2 years ago | (#38203932)

You may get paid more for the job...
However you may loose the opportunity to get repeat business.

Try Norton Ghost (0)

Anonymous Coward | about 2 years ago | (#38203446)

You could set up ghost and re-image the machines, that would at least give you a head start... may be able to do a low level secure format as well.

Either LTSP or Lessdisks with scripts? (1)

mallyn (136041) | about 2 years ago | (#38203460)

Perhaps a diskless boot (either Linux terminal server project or Lessdisks or something like that) and then run a script automatically to do whatever you want to do. Establish a client name based on the client's mac address. Use that to store the disk's contents remotely. If done right, you can have a room full of these machines doing this automatically at once.

Re:Either LTSP or Lessdisks with scripts? (1)

Cylix (55374) | about 2 years ago | (#38203862)

I had an isolated network I was required to do this on once. I had no servers and several isolated independent networks. Unfortunately, the network isolation created several issues with our existing toolset so it wasn't a matter of spinning up a host and moving some software over.

I ended up using puppy linux on a usb stick which would spin up an instance that contained a pxe server containing another puppy linux ramdisk. It's sole function was to serve the ramdisk to other machines in the isolated network. The usb stick could be removed once the OS was booted. (Puppy boots and remounts it's ramdisk over /). Once all of the machines were showing tons of disk activity I simply hit the space bar and the puppy host would begin running the same utility set on the host machine.

It takes some of the work out of creating a ramdisk distro that supports dhcp/tftp/pxe. However, since it doesn't contain a great deal of the libraries that would normally be found in linux most applications need to be compiled statically. Despite setting CFLAGS/CXXFLAGS I remember having to manually edit the make files in a few instances to set the -static option.

Entirely doable and it should be a good project for the requester.

Clonezilla will back them up... (1)

djsmiley (752149) | about 2 years ago | (#38203484)

Well you can back them up using clonzilla however I've never used it that way before so I don't know exactly how you automate it....

but if you were to do that, you could then just write a bash script on the end which does the wipe with DD for you too. Job done :)

Re:Clonezilla will back them up... (0)

Anonymous Coward | about 2 years ago | (#38203504)

I use Clonezilla to push an image onto many machines at once. Haven't done the other way, but according to all documentation it supports it.

The setup of a Clonezilla server was easy and straightforward, and lots of customization options are available for working with devices and images.

Re:Clonezilla will back them up... (1)

SexyHamster (174881) | about 2 years ago | (#38203844)

The main complaint I've had with Clonezilla is what a pain it is to mount an image and restore individual files out of it. From what I recall you were stuck backing up partitions by themselves if you wanted to later pull individual files, but this was a pain if you ever wanted to redeploy the machine back to hardware.

The easiest time I've had deploying / backing up workstations was simply just a full-tower windows machine with multiple hot swap SATA bays using Ghost for images. Regardless of image machine OS and software I'll take backing up disks over SATA over network cable any day. If the computers don't have fast release hard drives I'd just buy a good power screwdriver.

It will be faster to only write 0s once (3, Informative)

Anonymous Coward | about 2 years ago | (#38203502)

Nobody has demonstrated the ability to recover data after that outside of a carefully controlled lab.

Re:It will be faster to only write 0s once (1)

egcagrac0 (1410377) | about 2 years ago | (#38203548)

Multi-pass overwrite may not be necessary to comply with your policies, but if the boss thinks he heard something once that it's better and insists it be done, we do it.

Re:It will be faster to only write 0s once (1)

Anrego (830717) | about 2 years ago | (#38203558)

If this is just for their own paranoia, then yeah, I agree.

However they are probably trying to be compliant with some standard/requirement (the backup makes me think that).. in which case it is probably mandated that they have to use a tool from some approved list with a minimum number of wipes.

Assuming it is windows (4, Informative)

BagOBones (574735) | about 2 years ago | (#38203510)

Microsoft User State Migration Tool + Microsoft Deployment ToolKit + Sdelete http://technet.microsoft.com/en-us/sysinternals/bb897443 [microsoft.com]

You should be able to backup the profile, load the OS and run a zeroing delete on all "empty space" on the drive.

Re:Assuming it is windows (0)

Anonymous Coward | about 2 years ago | (#38203550)

if you have an admin account on each box. hopefully, you're running a domain. you are running a domain right?

Re:Assuming it is windows (0)

Anonymous Coward | about 2 years ago | (#38203608)

Yep...thats what I was going to say.

Wipe Process... (3, Funny)

Oswald McWeany (2428506) | about 2 years ago | (#38203514)

There are two commonly used techniques to the wipe process.

In Europe the preferred method is to fold the paper in half before wiping. In the US the preferred method is to scrunch up the paper in a ball before wiping.

Check whether the PCs you are wiping did a number one or a number two. Male PCs do not need wiping for a number 1.

Re:Wipe Process... (0)

Anonymous Coward | about 2 years ago | (#38203598)

I'm a fold and tear kinda guy myself.

Re:Wipe Process... (1)

NatasRevol (731260) | about 2 years ago | (#38203934)

Tear what?

DBAN? (3, Insightful)

Anonymous Coward | about 2 years ago | (#38203518)

As for a whole problem solution, I think you will need to do a bit of DIY. But just a note on the wipe process. Just writing 0 to the drive repeatedly will not ensure all the possibly sensitive data is non-recoverable, you really need to write random 1's and 0's at least 3 times to each bit of the drive. For that there is no better program than Derek's Boot And Nuke (DBAN) that I think is available as a liveCD and is available to several distros, including The Ultimate Boot CD (UBcd) and that may be a good place to start for a single boot backup, wipe solution. if you can write a shell script that can run from a pen drive while UBcd is in the CDbay.

Re:DBAN? (0)

Anonymous Coward | about 2 years ago | (#38203874)

What you said is true.

However I doubt that the NSA or its adversaries will be trying to recover the data from these drives... which is about the level of expertise required to extract the data from a zero'ed drive.

Re:DBAN? (5, Informative)

EdZ (755139) | about 2 years ago | (#38203924)

Just writing 0 to the drive repeatedly will not ensure all the possibly sensitive data is non-recoverable, you really need to write random 1's and 0's at least 3 times to each bit of the drive.

This has not been true for a LONG time. Ever since the GMR head became widespread (first introduced in 1997), platter field densities became too high, and field strengths became to low, to be able to feasibly read any sort of residual field after a single pass. Never mind that even if you could read the residual domain, poring over a single 1tb drive with a MFM would take literally billions of man-hours (8796093022208 bits * 1 bit every 10 seconds = 24433591728 hours, or 2.789 million years) to recreate a even rough guess of the bit layout, and that you would then need to align the all guessed layouts for each platter perfectly (think a few million possible combinations at least) before you could even start trying to pull data from the drive.

Send the ATA SECURE ERASE command to the drive, then move on while the drive controller does it's thing. It'll even erase sectors in the G-list, which DBAN will not.

Use a screwdriver. (5, Insightful)

Scioccoballante (1417005) | about 2 years ago | (#38203520)

Take the hard drives out of them, label them, and stick them in a closet.

Re:Use a screwdriver. (0)

Anonymous Coward | about 2 years ago | (#38203804)

That is the fastest, cheapest and most secure method. Nobody will mod you up.

Re:Use a screwdriver. (1)

nine-times (778537) | about 2 years ago | (#38204128)

I think this is my favorite answer to this. It may not be *the* answer, but I applaud your approach of "rethinking the problem".

hmmm (2)

TheCarp (96830) | about 2 years ago | (#38203530)

I would look at FAI or kickstart. For FAI a pretty early hook to backup and wipe.... for kickstart a %pre script.

Of course, if you are working alone, and don't know how to configure DHCP/NFS etc.... it may take you a couple of days just to get the basic setup going, as they can be very finicky, but the quickstart guides out there should generally be able to get you going. If all goes well, you could be working on your scripting in a couple of hours, if not..... well....I hate troubleshooting NFS.. (and don't forget to check your IPTables setup if you are having trouble getting it working...amazing how much better NFS works when its packets are not being dropped.

Overall, I like FAI better than kickstart, but thats probably because I have used it less and those early stages (DHCP/NFS mount) are hard to troubleshoot with kickstart since stage2 (and thus a shell with which to troubleshoot) isn't available until that works.... though.... you probably don't have the same constraints I do and can just switch USB keys and boot off a fully functional system to test poke around.

Re:hmmm (0)

Anonymous Coward | about 2 years ago | (#38204034)

Of course, if you are working alone, and don't know how to configure DHCP/NFS etc...

... then how the FUCK did you get that job? Lie gruesomely on your resume? If not, who was dumb enough to trust you with their network?

automate with Linux of course (1)

dalesyk (302267) | about 2 years ago | (#38203544)

I would pxe boot each computer with some flavor of linux, mount drive, backup, unmount, and shred drive You should be able to create a script that runs at end of bootup to automate this.

Re:automate with Linux of course (2)

vlm (69642) | about 2 years ago | (#38203710)

Looks like you forgot the verify step. md5 the hardware drive, md5 the image, they better match (bet they occasionally don't!). Also if you're taking a bare image, you don't mount the drive, you just copy the raw partition. If you mount the partition, then you have "issues" if the filesystem is semi-corrupt, was powered down while active, etc.

Also you forgot your exception process/monitor/procedure/whatever. At least some of these "hundreds" are not gonna spin up, are gonna barf out read errors on obscure corners of the FS... How much is management willing to spend to "recover" the data? There are places out there willing to take 5 figures to recover bad drives, and their success rate is not 100% so you need a plan for that too.

Re:automate with Linux of course (0)

Anonymous Coward | about 2 years ago | (#38203912)

Yeah, and you forgot 296 other items too - like how to determine the machine's identity (mac address works if NICs don't get changed), the use of ddrescue (if errors occur depending on the group the machine belongs to it may either become a recovery case- no wiping, or a skip of imaging and straight to zeroing) and then creating a summary of machines that failed and the type of failures and sending that in email to him.

Seriously, there's plenty else that could be done. The OP is just giving some ideas to start from. He's not necessarily forgetting anything just because he didn't write a book or HOWTO in a Slashdot post.

Re:automate with Linux of course (1)

vlm (69642) | about 2 years ago | (#38203942)

I agree the OP had a workable solution under ideal conditions. Good engineering is figuring out the failure modes in non-ideal conditions. That's the big missing part in the OPs post. This is one of those situations much like safety engineering, where by far the easiest part is handling the "everything's working perfect" scenario and the hard part is figuring out the failure modes exist, and how to handle them.

Ghost (1)

RecycledElectrons (695206) | about 2 years ago | (#38203546)

I wonder if you could configure a boot disk to image the machine onto a NAS, and then wipe it? Do you care about verifying the image?

How configurable is a Symantec Ghost boot disk? Is there a FOSS alternative to Symantec Ghost?

How would you tell the images apart. It's inevitable...Mary will need a copy of her "My Documents" folder from 2 years ago.

small linux partition (1)

kdayn (874107) | about 2 years ago | (#38203552)

I once had to manage 30 PCs with Windows where users needed an easy way to quickly install one or another image on the machine. So I made that every computer had a small (few hundred Mb) partition at the beginning of disk with stripped down linux installation, instead of running init when linux started my script was executed where I had everything scripted (which image to transfer in which direction etc etc), and I just added necessary boot options to grub menu so that users can easly pick a task.

Backup from the pen drive (3, Informative)

billcopc (196330) | about 2 years ago | (#38203570)

There isn't a whole lot to optimize in your process. Backups and wipes take time. One thing that could save you a step is to run the backup from the pen drive. That would allow you to script the entire process, such that you only need to boot off the pen drive, preferably have it cache itself into a ramdisk and start the script automatically, then move on to the next box. That would bring the whole process down to maybe 2 minutes per box.

Having ghosted a bazillion machines this way, it's monotonous but if you create 4-5 of those pen drives, you can do a bunch in parallel.

Re:Backup from the pen drive (1)

TarpaKungs (466496) | about 2 years ago | (#38203692)

Debian's debirf tool allows fairly painless building of custom bootable ISOs which boot to ramdisk. The ISOs can usually be run through isohybrid for pendrive booting, depending on the hardware and how fussy it is.

Re:Backup from the pen drive (1)

hedwards (940851) | about 2 years ago | (#38203860)

Back ups should be completely automated and where the sysadmin only has to verify that they were completed and that they are viable for restore. Having to do anything more than that is an indication that things aren't being done properly and that you're going to lose data at some point.

Wipes OTOH can be automated, but it's going to depend how confident you are that you're wiping the correct machine and that the backups are completely current and haven't been corrupted.

network image boot (0)

Anonymous Coward | about 2 years ago | (#38203576)

I guess you can go away with the pendrive boot and use PXE [wikipedia.org] or any supported network boot so that you have one OS image for "backup boot" and another OS image for "wipe boot"

you can use iptables and other mac-based packet requests to serve the backup image on the first boot and the wipe image on second boot. I believe it's completely feasible with almost any dd-wrt router with an usb storage plugged in.

DBAN + PXE (1)

futuresheep (531366) | about 2 years ago | (#38203638)

DBAN + PXE could do this for you:

http://users.telenet.be/mydotcom/howto/linux/dbanbootserver.htm [telenet.be]

Re:DBAN + PXE + pxelinux + Clonezilla + ZFS (2)

bill_mcgonigle (4333) | about 2 years ago | (#38203756)

That's perfect for the wipe, but he also needs backup.

PXE boot to pxelinux for a nice menu, then choose the backup (e.g. Clonezilla), throw a sticky note on the computer to indicate state, then reboot after the backup and choose DBAN.

http://www.linuxjournal.com/magazine/pxe-magic-flexible-network-booting-menus [linuxjournal.com]

A clever setup would keep track of which machine is in which state and hand out DHCP options accordingly, making the menu unnecessary.

With that many computers, backing up the drives to a ZFS volume with deduplication enabled is probably worthwhile (but make sure you have at least a GB of RAM per TB of disk).

PXE is your friend (1)

Anonymous Coward | about 2 years ago | (#38203644)

First write a script that will push a Wake-on-LAN packet to each machine on your list--assuming you're doing this in situ as opposed to bringing it in for the wipe. Then author a PXE script that first:
1) Checks to see if it is backed up, probably using a file on the backup server that you're using to select a list of the machines to backup and wipe.
2a) If it is NOT backed up, back it up with the product of your choice, whether it's Microsoft's USMT (I'm assuming you're doing this on Windows machines, you should see what you can get away with using SCCM), Norton Ghost, Acronis, etc. Once the script signals completion, have it reboot.
2b) If it IS backed up, PXE boot to DBAN (a tiny linux image) with the security level of your choice. I can get away with DoD standard here. Shut down after completion.
3) PXE boot to the new image you're replacing it with.

Test this. Test this thoroughly. Automating a data wipe can lead to some spectacular scenes of failure a la Fantasia.

Storage (3, Informative)

vlm (69642) | about 2 years ago | (#38203654)

Everyone else (anyone else?) will answer the automation question, but if you're ever done a PXE based linux install, you're about 99% of the way there.

The mystery I have, is where are you going to store "several hundred" drives worth of backups? And who or what is going to back up and maintain and store and recover the backups?

I'm guessing the best answer is open all the boxes, remove the drives, install new blank drives, all done? Given the cost of storage and admin time, this might even be the cheapest solution.

If this is a forensics issue, its a heck of a lot simpler legally to stuff THE drive in a evidence bag and buy a new one, rather than try to explain how your image is a true image crypto signed so it wasn't altered after it was signed, except how do you prove it wasn't altered before it was signed, blah blah blah.

Are you talking about backups where you only store relevant user "my documents" type data which might be practically nothing, or merely all files on a stereotypically mostly empty drive which would be at most a couple gigs, or a full bit for bit forensics dump of hundreds of 1 TB drives?

There's a big difference between "it all fits on a single USB attached consumer grade 1 TB drive" and "We're gonna need multiple racks of multimillion dollar NAS to hold all the images".

How valuable is the data? If it leaked would you lose PCI / CC / HIPPA / SOX stuff and its the end of the world or at least your corporation and job, or is it just a university computer lab and the most valuable/sensitive thing is a couple rickroll videos and some lolcats?

What do you intend to do, if anything, with the backups? The simplest / cheapest / most efficient way to store backups might involve just throwing the machines in a rented storage room. Climate controlled if possible. You can rent a heck of a lot of storage space for a long time for the cost of a couple hundred hours of admin time.

Finally whats your liability? If for example, one doesn't boot due to hard drive failure or whatever, are you shipping it to one of those $10K data recovery places, in other words you actually care, or if you lose some, eh, whatever, it was just a "nice to have"? If you can lose one, can you lose all of them with the same "eh" attitude? If your liability is significantly lower than your costs, your best plan might be to skip the backup and destroy the drives.

In summary the problem isn't how to "transfer" a couple hundred terabytes, that is a long solved question, no big deal. The unsolved problem is how to store / collate / search / backup / distribute / secure a couple hundred terabytes.

Re:Storage (3, Informative)

vlm (69642) | about 2 years ago | (#38203816)

Whoops epic fail on my part, you have an endgame plan for the old machines, you are imaging their drives and wiping them, like today, or whenever you get off slashdot. That's just ducky.

Now, what's your endgame plan for the images. Keep them forever? Or just next financial quarter/year? Or whatever the IRS interval is (7 years, I think?) Does the NAS / RAID / external USB drive holding them need to get copied and wiped? If you're doing the geographic diversity thing, who's securely disposing of the offsite backups?

NFS/SMB Share + PXE Server + Boot script (0)

Anonymous Coward | about 2 years ago | (#38203658)

1) Setup a private network for all the PCs you need to backup/clone
2) Setup a NFS or SMB share, PXE boot server and DHCP service on a Linux box
3) Create and add your backup/wipe script to the PXE boot image, have it execute at boot

Setting up a PXE boot server is very easy, first hit on google has straightforward instructions: http://linux-sxs.org/internet_serving/pxeboot.html

Most desktop systems have supported PXE booting for some time, you should be able to select it from the boot selection menu at start-up.

Live with the tedium (1)

Bookwyrm (3535) | about 2 years ago | (#38203674)

Live with the tedium of doing in manually. It sucks, but unless you are going to have to do this exact operation again in the future, don't bother with automating it. Possibly the solution of taking out the hard drive, putting in a drive dock on another computer, and letting that computer back-up and wipe the drive might be slightly less tedious, depending on the situation.

Because, if you listen to what you are asking, you are trying to set up an automated back-up and erase system. Unless you have a Lot Of Time to Test this BEFORE HAND, you could easily end up with an automated screw-up-the-back-up and nuke-everything system. If you successfully manage to create a system that erases several hundred computers without making usable back-ups, that might be a career-limiting move.

You are asking for replacing a single-shot pistol with a high-powered Gatling gun -- this is not unreasonable. However, if you shoot yourself in the foot with such a thing because you are not careful, there will not be a lot of remains left over.

If all the computers are absolutely identical, you might be able to do an automated system, test it against a couple machines, and be able to get it to work. Otherwise, the amount of time you will spend making sure that the automated system does exactly what you need it to do, safely, without ever failing, may end up being as much time as it takes to do it manually.

Oh, you are verifying that your back-ups are usable before nuking the drives, right?

Re:Live with the tedium (2)

vlm (69642) | about 2 years ago | (#38203904)

Unless you have a Lot Of Time to Test this BEFORE HAND, you could easily end up with an automated screw-up-the-back-up and nuke-everything system

This might be the best (only?) justification for buying non-free beer non-free freedom software I've ever seen, because you can intentionally buy the cheapest cruddiest non-working commercial software out there, then when all the data is lost, you don't have to maintain, backup, search, restore and otherwise admin the images for eternity minus a day, and you can blame the commercial software provider instead of yourself... Everyone, especially in management, knows commercial software just doesn't work sometimes and its no ones fault. Heck you could just skip actually imaging and wiping the drives, blame the software, and spend the new spare time playing skyrim.... Hmmm.... I think we're on to something here.

Re:Live with the tedium (0)

Anonymous Coward | about 2 years ago | (#38203926)

A better approach for this kind of thing is to do an automated backup and then replace the drive and hold the backed up drives for at least a week, its still faster and less likely to have a massive failure.

Think about this a little different (1)

Dynedain (141758) | about 2 years ago | (#38203700)

If you're doing this for secure disposal, there's a much easier solution:

Pop the drives out and do your work via external slot-loading drive caddies. You can get rid of the big machines as usual and work your way through the drives as time permits between other tasks. If your software has command-line APIs, it should be pretty easy to setup scripts to do this.

- or -

Do the backup as a separate task. Deploy a dedicated backup tool (for de-duplication and compression) or use rsync. Then setup DHCP with NetBoot to run a distro configured to auto-format and install from an image hosted on a server. Even Windows supports this.

PXE (1)

silas_moeckel (234313) | about 2 years ago | (#38203704)

Single server pxe boot into a live linux distro with clonezilla and your drive wiper of choice. Some simple scripting to get clonezilla to backup all drives to the server under the name gotten from a prompt and wipe when it's done. Throw the same bits on a USB drive if you want.

PXE w/ Clonezilla and DBAN (1)

frooddude (148993) | about 2 years ago | (#38203714)

PXE booting is not difficult to set up and Clonezilla is dead simple to automate after that. DBAN also has instructions to PXE boot, but I've never used it that way. Extra points for setting it up to do both in 1 pass. Clonezilla also has the nice feature of verifying that you have a good backup.

Mass HDD Duplicators? (1)

Mister Transistor (259842) | about 2 years ago | (#38203760)

For software backup, Norton Ghost enterprise is the way to go unless you have some solution you already are using / have to use...

For wiping the hard disks, they used to make bench-top hardware boxes you could hook up 4 drives to directly and mass-copy them all 4 at a crack. You can use a clean formatted drive as a source and "duplicate" that to wipe the drives clean, 4 at a time simultaneously.

I'm not sure if there are similar devices that do Ultra-ATA or SATA, but it might be worth looking into getting one if you have that many devices and time is critical.

Acronis (0)

Anonymous Coward | about 2 years ago | (#38203782)

Acronis will do this off the shelf.

If you are working with Windows Server 2008 R2 there is a way to automate all this as well as automate the djoin process using sysprep.xml. Depends on your level of knowledge I guess.

Why leave the drives in the original boxen? (0)

Anonymous Coward | about 2 years ago | (#38203788)

The easiest way, if you have a server with hot-plug drives, is to remove the drives from all the workstations/servers that need to be wiped and plug them into the hot-plug server (if you have enough carriers). Then you can run enough copies of your wipe utility to wipe them in parallel. We recently had to wipe about 400 drives, and we had two servers with 8 hot-plug SATA slots that we used to wipe them all over a couple of days. We were required to keep an audit log for the federal government of each drive wiped, so the software we used was modified to fetch the drive serial number and record that along with the time/date wiped. A modern chipset will support hot-plug SATA, so you don't even need to reboot the server. Just swap drives.

Re:Why leave the drives in the original boxen? (0)

Anonymous Coward | about 2 years ago | (#38204116)

Use of the term "boxen" is a flag for a complete and utter tool.

Why would you write 0 to the drive more than once? (0)

Anonymous Coward | about 2 years ago | (#38203790)

Fags will be fags, I guess. Do you even know why you want to write "zeros" to the drive more than once?

Simple PXE setup .... (0)

Anonymous Coward | about 2 years ago | (#38203796)

Set up a PXE server, where every device default boots the backup script.
Once that script is completed, that PC reboots, and the mac address for that PC gets shifted in the PXE config to a static the wipe boot script.

So have a watch process modify/update the PXE config for each machine as it transitions from backup script reboot (drops ping response?) to next PXE attempt.

How About (2)

sexconker (1179573) | about 2 years ago | (#38203806)

How about:

1: Pop open case.
2: Remove Drive.
3: ???
4: Profit (through continued employment).

?

You only want 1 copy of the data, so the original is the backup.
Just convince the nearest PHB that it's a waste of your time and their money to wipe drives. If the machines are going to be repurposed or sold, it's cheaper and easier to buy new drives, or sell them without the drive. As for the backup, same deal - cheaper and easier to store a bunch of HDDs in anti static bags in a box somewhere. Safer than storing their data on a live backup device anyway. And if you need immediate, constant access to that data - why are you killing the machines in the first place?

But if you have to do it their way, get Acronis. It lets you backup an entire drive to a network location from a bootable cd / usb thingamajig.
It works with like, every fucking SATA / RAID controller there is, and if it doesn't, you get official support for building a BartPE disc that includes those drivers so you can get your shit on. You can even do a full drive backup from within windows. It's pretty fast, too. You can encrypt backups with AES-256, and you can browse through the backups with any machine that has Acronis installed. You could also of course dump the full contents of the backup into a 7zip archive or some other format if you wanted to not depend on Acronis's file format and software in the future.

Just make as many copies of the disc as you have network ports in whatever dark corner you'll be working in, line em all up, tell them to go, then play Minecraft until it's time to switch out machines. I think the bootable cd also has basic 0 wiping tools, as well, but I don't know for sure.

on another note... (0)

Anonymous Coward | about 2 years ago | (#38203810)

... where is /dev/one. we have /dev/zero but no /dev/one.
could be useful when erasing a drive with nothing but ones. dd if=/dev/one of=/dev/sdx

Ball Peen Hammer (0)

Anonymous Coward | about 2 years ago | (#38203818)

Three words...Ball Peen Hammer

You're backing them up before wiping them? (0)

Anonymous Coward | about 2 years ago | (#38203832)

Why not:

1) Remove disk
2) Put disk in safe as backup
3) Put new blank disk in machine
4) ?????
5) Profit!

0's? Seriously? (1)

Annirak (181684) | about 2 years ago | (#38203836)

running some software that writes 0s to the drive

That seems unwise. You're not really wiping the drive, just making it harder to read. Most modern wipe software overlays the drive 7 times with random data.

Re:0's? Seriously? (0)

Anonymous Coward | about 2 years ago | (#38203960)

Writing zero's IS wiping the drive. There is nothing else to read. "Modern" wipe software can overwrite 23929348 times, that doesn't mean it's actually doing anything neccessary.

yes (0)

Anonymous Coward | about 2 years ago | (#38203850)

2 networks, one for clonezilla and the other for dban.
  both booting via pxe with defaults set. /end

software that writes 0s to the drive several times (0)

Anonymous Coward | about 2 years ago | (#38203852)

I'm pretty sure the prize money for recovering any data from a drive that has had only one parse of 0's is still up for grabs.

Hasn't anyone thought to... (1)

Mike Savior (802573) | about 2 years ago | (#38203854)

Why not roll a small Linux distro (something really titiny, like DSL, or picolinux, write a script to back up to thumb drive, then reboot the machine and wipe it? Push that image to every machine via network.

Do the wipe first (5, Funny)

md65536 (670240) | about 2 years ago | (#38203858)

That will make the backup a lot easier.

Question: how secure (1)

nine-times (778537) | about 2 years ago | (#38203898)

The first question that pops into my mind is, what is determining this secure wipe procedure, and how secure does it really need to be? If you're looking to speed things up, you could wipe everything with zeros once instead of "several times". The difference in security is minimal.

Aside from that, there are open source solutions that will image a drive and others that will wipe the disk. It shouldn't be too hard to chain them together, though I don't know of any pre-built solution. I'm stating the obvious here, but if you do try to build your own, definitely be careful of 2 main things: make sure users can't boot into this procedure accidentally, and make sure that the procedure absolutely will not wipe the drive unless the backup was successful.

Re:Question: how secure (1)

vlm (69642) | about 2 years ago | (#38204042)

You say chain them together, like 1 ms after backing up, you start wiping. I say, how long can you wait with the images and hardware in storage before wiping?

At least back them all up, then wipe them all in two separate processes? Whatever you do, don't manually start one process after the other because at least 1% of the time (several machines, in your case) you'll accidentally start wipe before backup. At least that'll compress pretty well if you're wiping with zeros.

Wiping is faster and "what if" the images are somehow bad or corrupt or otherwise need access to the hardware (like, whoops, we tossed out the software license hardware dongle, best find it).

Computrace (0)

Anonymous Coward | about 2 years ago | (#38203908)

Computrace by Absolute Software has a cloud-based end-of-life wipe (the product is called Computrace Data Protection). All the device needs is an Internet connection.

Consider External Drives (1)

Anonymous Crobar (1143477) | about 2 years ago | (#38203930)

I'll go on record with a preemptory "what he/she said" to include all of the previous comments. This is one of those things that nearly every slash dotter has to do at one point or another and in my experience, you'll have to deal with at least one of three bottlenecks: time, money, or bandwidth.

If you are doing this to several hundred machines, sneaker net is likely a faster solution than your network. Take advantage of the higher bandwidth and save the backup images to a portable USB drive. With a large enough drive you can also keep the new image local as well. Using this method you can boot into Your Favorite Backup Solution, take the backup, securely erase the disk, and write the new image in one fell swoop. After the imaging is complete collect your various USB drives and march them back to the server.

With large enough USB disks, you could repeat this process several times before off-loading the images to your server.

ssh & rsync & "backdoors" (1)

cowtamer (311087) | about 2 years ago | (#38203972)

If the machines are Linux (or booted temporarily into Linux), use ssh (or rsh) to script most of what you're doing. Be sure to configure them to not require passwords for ssh. Then use rsync to back up, and remote ssh scripting to do the wipe on all machines. You can get smart with transferring scripts to the machine & running them with ssh scripting without doing anything manual.

If the machines are Windows boxes, you might want to look at some remote access/backdoor solutions (of the "gray" hat variety, perhaps -- since you presumably do not want to go to each machine and log in manually to do ANYTHING). Do what the biologists do and turn attack vectors into something useful!

You call yourself a sysadmin? (0)

Anonymous Coward | about 2 years ago | (#38203988)

Boot off something, pxe or usb will do, a recordable cd or dvd if you must. Either have it give you a boot menu (grub works) with the options to run the various steps, or run one script fully automatically: Backup, verify backup, if it verifies log it and run the auto-wiper. If not, log it and call for attention.

Really, why do you have to ask? With a hundred boxes to go, yes it does pay to spend a day tooling and testing, then do the rest automatically and in parallel. Don't even have to wait for all the smart-ass commentary on slashdot. Get to work.

Simple. (0)

Anonymous Coward | about 2 years ago | (#38204006)

Ma Deuce, Bobcat.

Maybe a large bonfire in between.

homebrew (1)

Murdoch5 (1563847) | about 2 years ago | (#38204014)

What about writing a simple script that would do the same job.

1) Just have the script goto a computer
2) upload the data you need to a SCM hub of some sort or any kind of backup source.
3) Then have the script start a DD on the computer to wipe the drive, or a comparable windows program
4) Have the computer reboot and connect to a PXE server which has an auto seeded OS install on it.
5) Have your script push the information back to the computer

I wrote something remotely like this last year, it didn't do all these steps but it did handle remote access, backup and restoring of certain files. It was C based.

Use a management tool (1)

onyxruby (118189) | about 2 years ago | (#38204020)

Most modern infrastructure management tools like Altiris can easily perform a pxe boot function. Set up a wipe job, link it to the MAC address and wipe it. Bonus points for having an auditable trail if that's required by your flavor of regulation.

Is there a single documented instance (1)

jefe7777 (411081) | about 2 years ago | (#38204028)

of recovering any data after a successful single pass with

dd if=/dev/zero of=/dev/sda bs=4k

I'm just curious. I've read all the theoretical stuff, but wouldn't the drives have to be disassembled in a clean room and the platters installed on some machine that can read the faint magnetic residuals...

Who has these facilities and machines, if anyone, beyond the alphabetsoup gangs?

Re:Is there a single documented instance (1)

jefe7777 (411081) | about 2 years ago | (#38204076)

I guess large corporations have the resources as well. IBM, Apple, Google....

quick and easy disposal = woodchipper (0)

Anonymous Coward | about 2 years ago | (#38204032)

Doing things right isn't allows quick and easy.

Hundreds of machines once? (1)

Culture20 (968837) | about 2 years ago | (#38204050)

If this is the only time you'll ever do it, a pen drive sounds good enough, although a CD image might be better since you could make a ton of them quickly.
Otherwise, piecing together a PXE solution would be a waste of time since you still have to plug the machines in, configure the BIOS for PXE, unlock the BIOS if you're planning on donating the machines (the bios steps can be done with automated utils if you're using HP or Dell machines).
If you can leave the machines where they are, and they're already unlocked and set to PXE boot, then making the PXE server only let their MACs attach would be the only big step. Then backup each machine to a folder based on its MAC.
Another option is to just take all the HDDs out, dump the machines, and backup/wipe at your leisure using a few computers with 6 drive cables each.

Use Encryption (0)

Anonymous Coward | about 2 years ago | (#38204074)

When I got data that I know is going to be over-written, I just assign a strong encryption-key to it. So, when I got a hard-drive with 2TB of data I flush the encryption-key and file-table to reset it. The encrypted data without the key is just noise, anyway. So, no hard deletes are necessary. This is a very secure way to work, and I am surprised not more people use this method. There is some software that can help you, but there are also scripts that can do the same job.

PXE boot??? (1)

CliffH (64518) | about 2 years ago | (#38204154)

Why not just PXE boot a small OS image which does the backup and wipes the drives?? Linux or DOS will do the trick just fine. If you're trying to backup anything NT based (NT-Win7) get a read-only NTFS driver on the PXE image too. It's pretty simple really and doesn't take any real time IF all of your systems are semi-modern.

ultimate boot cd (0)

Anonymous Coward | about 2 years ago | (#38204202)

ubcd has several utilities disk wiping and other disk management tools.

http://www.ultimatebootcd.com/

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...