×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Time Bomb May Have Destroyed 800 Norfolk City PCs' Data

timothy posted more than 3 years ago | from the philadelphia-experiment dept.

Security 256

krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

256 comments

Just so you get the pronunciation right... (5, Funny)

Overzeetop (214511) | more than 3 years ago | (#31174150)

It's Naw-Fuck.

And it's nowhere near as embarrassing as how we pronounce Buena Vista.

Re:Just so you get the pronunciation right... (1)

Anonymous Coward | more than 3 years ago | (#31174310)

How do you pronounce Buena Vista?

Re:Just so you get the pronunciation right... (4, Informative)

Overzeetop (214511) | more than 3 years ago | (#31174458)

Byoo'-nah Vis'-tah

The locals have taken the whole diphthong pronunciation (when two vowels go walking...) to an extreme.

We also have Staunton, which is pronounced Stan-tun (short a sound).

Re:Just so you get the pronunciation right... (2)

TXFRATBoy (954419) | more than 3 years ago | (#31174606)

How about Buchanan - pronounced Buck-nun Or Botetourt - pronounced BOT--tot

Re:Just so you get the pronunciation right... (2, Insightful)

Overzeetop (214511) | more than 3 years ago | (#31174816)

One of my first interactions in the state after being in California for a couple of years was at a Wendy's drive-though. The attendant was kind enough to tell me "I put you some salt and ketchup in the bag." Is there such a thing as hillbillionics?

Someday I'm going to run for public office, and this thread is going to come back and bit me in the ass. I just know it.

Re:Just so you get the pronunciation right... (1, Funny)

AndersOSU (873247) | more than 3 years ago | (#31175170)

Nah, it'll be your effete voice, meticulous faggy pronunciation, and vocabulary that contains words like effete.

Re:Just so you get the pronunciation right... (0)

Anonymous Coward | more than 3 years ago | (#31174774)

I fail to see the funny part of Byoo'-nah Vis'-tah.

But, maybe that's just me.

Re:Just so you get the pronunciation right... (1)

xaxa (988988) | more than 3 years ago | (#31175130)

We also have Staunton, which is pronounced Stan-tun (short a sound).

With pronunciations like that, I think you're well on the way to pronouncing English place names [wikipedia.org] :-)

Southwark: Su-thuk
Marylebone: Marl-i-bun
Norwich, Norfolk: No-rij, Nor-fuk (short o for both).

Re:Just so you get the pronunciation right... (1)

hardie (716254) | more than 3 years ago | (#31175258)

There is a town in northern Maine, Calais, on the Canadian border. Mainers pronounce it exactly like the word 'callous'.

Re:Just so you get the pronunciation right... (1)

pspahn (1175617) | more than 3 years ago | (#31175266)

When I first heard Coloradoans pronounce Buena Vista, I immediately wondered why Pueblo wasn't pronounced Pube-low.

Re:Just so you get the pronunciation right... (3, Funny)

wintercolby (1117427) | more than 3 years ago | (#31174404)

Yes, and their Highschool cheer is:
We don't drink! We don't smoke! Norfolk! Norfolk!

Pronounced as specified above.

Re:Just so you get the pronunciation right... (0)

Anonymous Coward | more than 3 years ago | (#31174748)

it's nowhere near as embarrassing as how we pronounce Buena Vista.

Or even Botetourt County, Virginia [Baht-uh-tot County]

Re:Just so you get the pronunciation right... (1)

Hatta (162192) | more than 3 years ago | (#31174760)

There's a Norfolk here in Nebraska. It's called "nor-fork". And there's a Buena Vista University just across the river in IA. I cringe every time I hear a radio ad for them. Bew-nah Vista. Just awful.

Still if you're going to complain about odd spellings and pronunciations, I'd say the British still take the cake with "Worcestershire".

Re:Just so you get the pronunciation right... (1)

JustOK (667959) | more than 3 years ago | (#31175264)

Not that it's pronounced funny (like Nauwigewauk ), but Saint-Louis-du-Ha!Ha! is kinda fun to say

Re:Just so you get the pronunciation right... (1)

xaxa (988988) | more than 3 years ago | (#31175292)

Still if you're going to complain about odd spellings and pronunciations, I'd say the British still take the cake with "Worcestershire".

It's reasonably consistent with the other -cester places (which were all Roman towns):
Leicester (Les-ter), Gloucester (Glos-ter), Alcester (Ol-ster), Bicester (Bi-ster), Towcester (Tow-ster). And "Wus-ter-shire", for anyone that's still wondering about Worcestershire (Worcester is the city, Worcestershire the county).

Unfortunately, Cirencester isn't Si-ren-ster, but Si-ren-ses-ter.

Re:Just so you get the pronunciation right... (1)

xaxa (988988) | more than 3 years ago | (#31175364)

Ooops, I forgot to point out that "shire" in a county name is "shur". "The Shire", as in LotR, is pronounced like shy-er.

Re:Just so you get the pronunciation right... (4, Funny)

xaxa (988988) | more than 3 years ago | (#31174858)

It's Naw-Fuck.

In proper Norfolk... well, I'll let Wikipedia [wikipedia.org] explain: More cutting, perhaps, was the pejorative medical slang term "Normal for Norfolk", referencing the county's supposedly high rate of incest. In truth, Norfolk's incest rate is no higher than the rest of England. The term is now discredited, and its use is discouraged by the profession.

(Sorry, did you want an on-topic comment?)

Re:Just so you get the pronunciation right... (1)

Coren22 (1625475) | more than 3 years ago | (#31175072)

Maybe true for the English Norfolk, still up in the air for the Virginia Norfolk...

Not only that, someone's going to jail (0)

Anonymous Coward | more than 3 years ago | (#31174158)

for making bomb threats.

Essentially destroyed? (0)

Anonymous Coward | more than 3 years ago | (#31174214)

Whatever it was, it essentially destroyed these machines.

Unless this time bomb was something from the 90's like Win32.CIH and nuked the bios, I doubt that the computers are "essentially destroyed"

You guys have backups, right?

Re:Essentially destroyed? (3, Informative)

CorporateSuit (1319461) | more than 3 years ago | (#31174298)

Hardly. It's just something that messed with the Win32 folder. This could be fixed by a few temps over the weekend if the city government was half-competent.

Re:Essentially destroyed? (4, Insightful)

v1 (525388) | more than 3 years ago | (#31174346)

if they were running backups, they wouldn't be scratching their heads and behaving completely ignorant of what exactly it was or when it was put in. They obviously lost everything, which I'm sorry but I find some darwinism/justice in that. If you don't even have a backup to look at to see what it was sitting on the hard drive waiting to blow up, you're just beyond help. Maybe better luck next time.

But too many out there simply must learn their lessons the hard way. That will never change.

Re:Essentially destroyed? (4, Insightful)

MightyMartian (840721) | more than 3 years ago | (#31174416)

We've instituted offsite backups, both over the tubes and physically taking images of our servers (all virtualized of course) offsite to a bank safety deposit box. If, for whatever reason, the whole damned building explodes tomorrow, we've got the data sitting on servers in two other geographically distant locations. But if we can't get to those, we have the VM images, so as long as we can get our hands on a server capable of running Linux KVM, we could be up and running in short order (I estimate 3-4 hours, including host OS installation).

The days when a physical or digital attack can fuck the whole organization are gone. There are enough traditional and newer backup schemes out there that even long downtimes aren't necessary.

Re:Essentially destroyed? (1)

HockeyPuck (141947) | more than 3 years ago | (#31174532)

You must have a pretty small site if all of your data is contained within the .vmdk files and you can restore an entire datacenter (from bare metal) in 3-4 hours (including OS install time).

Re:Essentially destroyed? (1)

MightyMartian (840721) | more than 3 years ago | (#31174600)

It's relatively small, but we're actually backing images up to hard drives, not to tape or over the wire. The files themselves are both backed up to tape, and use DFS and some other mechanisms (like robocopy replication) to our remote servers. In a worst case scenario, I could pretty much drive the 100 miles, grab the remote domain controller and file servers from one of our satellite sites and drop them in the main office. The guys out there might not be happy that they were accessing everything through terminal services, but oh well.

Re:Essentially destroyed? (1)

Em Ellel (523581) | more than 3 years ago | (#31174780)

(I estimate 3-4 hours, including host OS installation).

I've done this in some small VMWare setups: using snapshot feature on FS (LVM works) plus a few very large external drives (those USB to SATA cradles work great), automate a backup of the snapshots of the OS and VM partitions once every X days take the drive offsite and use another one. With 3 drives, you can rotate them and always keep one offsite. What you now have is essentially a fully working drive you can insert into another server and just turn on, no OS install, no fiddling with VMWare install and versions, recovery time is down to essentially the time it takes to get the drive(if you have to use offsite drive) and get new hardware. Best thing is that the costs are that of a few USB drives and a bit of scripting...

-Em

Re:Essentially destroyed? (1)

MightyMartian (840721) | more than 3 years ago | (#31175088)

The basic idea behind storing snapshots is simply to allow faster recovery of operations even in the case of absolute disaster. We still have nightly differential backups, a weekly full backup, plus Server 2003 DFS and some scripted replication (via robocopy) of file servers. Nothing replaces a good backup scheme, a major pain in the ass to develop, and sometimes a pain to maintain. When we formulated the project, the basic notion was "If a fire/meteor/other disaster takes out one of our offices, how can we reduce downtime and data loss to the barest minimum". Rather than relying on a single backup strategy (ie. tape or distributed FS), we adopted a scheme of using multiple strategies. Daily and weekly backups are still important for accidental deletions and corruptions. Quarterly and annual backups are still important for archival purposes, and this is still the area where tape is king. But trying to restore something like a Server 2003 domain controller or Exchange server purely from backup has always been for me a nightmarish prospect, consuming considerable amounts of time. The idea behind virtual guest snapshots on a weekly basis is that I can get these servers up and running ASAP and use weekly and daily backups to refresh everything to get data up to date.

If the tape fails, well the worst is that I lose at most four business days of data, but hopefully not even that with DFS and other replication strategies. But let's take a worst case scenario, that somehow someone breaks into the network, destroys all the data on all domain controllers, the Exchange server and the file servers at all sites (something I don't find terribly likely). I still have the full backups of all files plus the Exchange and AD domain controller images sitting offsite on an external hard drive in a bank vault. I might lose about five days worth the work at the outside, which would be bad, no doubt about it, but certainly not the catastrophe of losing all my data, but that's only in a worst-case scenario.

Re:Essentially destroyed? (4, Insightful)

Lumpy (12016) | more than 3 years ago | (#31174690)

You got it. it's also a great example of how incompetent most City's IT staff are, Hey municipalities... you get what you pay for. How's those $25,000 a year IT staff working out for ya?

Largely From The Prevalance Of Machines (-1, Flamebait)

Anonymous Coward | more than 3 years ago | (#31174216)

running Winblows [microsoft.com].

Yours In Astrakhan,
Kilgore Trout

P.S.: without even reading the article !!! Take that Windoze fans.

Windblows 98 and other parodies (0)

Anonymous Coward | more than 3 years ago | (#31175280)

Search the Intertubes for "Windblows" and "Windoze" and enjoy the laughs.

Parroty Interactive had Microshaft Windblows 98.

Wait a minute.. (1, Funny)

VMaN (164134) | more than 3 years ago | (#31174220)

... this is the internet... Isn't the apostrophe in the title supposed to be further to the left? :|

I had to read it twice to confirm it was used correctly.

Re:Wait a minute.. (-1, Flamebait)

Anonymous Coward | more than 3 years ago | (#31174476)

Plural possessive. You should've been taught this in 5th or 6th grade.

Re:Wait a minute.. (0)

Anonymous Coward | more than 3 years ago | (#31175084)

Reading comprehension. You should have been taught this in the 3rd or 4th grade.

Re:Wait a minute.. (1)

travdaddy (527149) | more than 3 years ago | (#31174530)

... this is the internet... Isn't the apostrophe in the title supposed to be further to the left? :|

So, you're complaining that correct grammar was used?

You're like the opposite of a Grammar Nazi, or an incompetent one!

I live in VA Beach (1)

bsDaemon (87307) | more than 3 years ago | (#31174242)

I live in VA Beach, which is the next city down the road (I live a few blocks from exist 20 264, and down-town Norfolk is exit 13ish), and I work in a security-related position, so we tend to keep up on news like this, but this is the first I'm hearing of it, though it looks to have gone down last week (apparently the boot.ini files were modified between 16:30 and 17:30 on 9 February, and only the computers which rebooted during that time period were affected).

It doesn't sound like the attack was particularly complex or anything, so maybe that's why it isn't exactly "newsworthy" (I also don't watch local TV news, so I don't know if they mentioned it), but still, sucks for them. I hope they have good backup policies.

Re:I live in VA Beach (1)

sakdoctor (1087155) | more than 3 years ago | (#31174456)

Bomb! Destroyed! Meltdown!

Judging by the hyperbole, the reason you haven't heard about it, is because the destruction was so great, there were no survivors left to report it.
The blast radius of 800 computers, all exploding at once, would have caused devastation and little radioactivity symbols, the likes of which you've never seen before.

Re:I live in VA Beach (1)

bsDaemon (87307) | more than 3 years ago | (#31174644)

I don't know, I used to play with all disasters turned on in SimCity 2000, and then try and cause them.... shooting the nuclear plant with the microwave beam from the power satellite and stuff. Plenty of radiation symbols when that got done.

Re:I live in VA Beach (1)

YrWrstNtmr (564987) | more than 3 years ago | (#31174842)

(I also don't watch local TV news, so I don't know if they mentioned it)

It was mentioned on the Tuesday (I believe) news.

Re:I live in VA Beach (1)

idiotnot (302133) | more than 3 years ago | (#31175404)

WTKR had it last night at 11, but were kinda sketchy on details. Big emphasis on NO CITIZEN OR EMPLOYEE DATA WAS AFFECTED.

I live in Norfolk; let's just say that the best and brightest aren't working in IT for local governments. Defense companies pay a lot better.

When I worked for another local city, they were still running an ancient 16-bit version of Netware (would have been like 2002).

It happened on Patch Tuesday. (4, Interesting)

gimmebeer (1648629) | more than 3 years ago | (#31174246)

I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches...

Re:It happened on Patch Tuesday. (2, Insightful)

Chrutil (732561) | more than 3 years ago | (#31174572)

I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches

Sounds like it happened on reboot of these machines, which could imply that patch installation is responsible for the timing (if it mandated a reboot), but not necessarily for the cause.

Re:It happened on Patch Tuesday. (1)

operagost (62405) | more than 3 years ago | (#31174590)

Duh. It would figure that their entire IT department didn't read the news about the Microsoft update causing PCs to BSOD on bootup if they had been compromised by a specific malware.

Re:It happened on Patch Tuesday. (1)

gimmebeer (1648629) | more than 3 years ago | (#31174798)

This is what I was getting at, I read about this problem but I'm far too lazy to find a link at the moment. It was a known issue with a recent MS patch, but I don't recall whether it would trash the System32 directory or not. I'd venture to say not since it had to do with a specific .dll or something similar being updated.

Sounds like... (0)

Anonymous Coward | more than 3 years ago | (#31174260)

... someone has a case of the 2 A.M. Worm [theonion.com].

"How the hell did it get in here?"

Norfolk VA. (-1, Offtopic)

Anonymous Coward | more than 3 years ago | (#31174268)

Only stinky cunts live there

It took them a week to notice? (1)

flaming error (1041742) | more than 3 years ago | (#31174316)

> We don't know how it got into our system... We speculate...

As long as we're speculating, may I nominate last week's "Operation Cyber Storm" (http://www.dhs.gov/xnews/releases/press_release_0853.shtm).

A healthy System32 dir is 1.5 GB (2, Informative)

caseih (160668) | more than 3 years ago | (#31174348)

At first glance that blows my mind. That's absolutely huge. Then I check my linux box and /usr/lib64 is 1.7 GB.

That may be the size of the debugging symbols (0)

Anonymous Coward | more than 3 years ago | (#31175268)

If you're not a developer, and you don't install the "-dev" or "-devel" versions of the library packages, you won't have the dwarf2 source code debugging symbols, which can be quite verbose.

No explaination (4, Insightful)

HotNeedleOfInquiry (598897) | more than 3 years ago | (#31174350)

As to why they couldn't just boot to linux or a recovery CD and salvage the data....

Re:No explaination (1)

gurutc (613652) | more than 3 years ago | (#31174488)

You're right. This is big enough to spend some money on data recovery. Especially the mechanism so they can identify the perp.

Re:No explaination (0)

NevarMore (248971) | more than 3 years ago | (#31174500)

Because their computers used a CDROM based on a Flahbersen 3401-B chipset and Linux only has drivers for the 3401-A chipset if you set the ARS_BOGGLE compile flag.

In all seriousness, its because there are 800 computers, unspecified if they're servers or desktops, and it takes a LOT of time to recover that many machines. Varying speed depending on if you just need to recover My Documents (which should have been on a network share anyway) or the whole disk or just files requested by users.

Re:No explaination (1)

jedidiah (1196) | more than 3 years ago | (#31174592)

In other words: Those machines really had nothing worth saving anyways.

They could all have been a bunch of VT-220's for all anyone cares.

No, it's not really like that. The same entity that allowed this to happen
can't be bothered to make sure that there wasn't any data lost during the
whole shenanigan. It's corner cutting from top to bottom.

Re:No explaination (1)

berwiki (989827) | more than 3 years ago | (#31175214)

...because that sounds ridiculously slow/annoying/complex for 800 PCs.

gotta love the 'token Linux' reply without thinking about their response first.

Norfolk's IT is fail. (5, Insightful)

castironpigeon (1056188) | more than 3 years ago | (#31174366)

So the data is wiped because the System32 folder is fucked up? Uh-huh... guess they have to throw out all those computers and order new ones. Looks like the data's gone forever.

Look, if you're the IT guy and this happens (1)

wiredog (43288) | more than 3 years ago | (#31174682)

You just restore the image from a ghost backup without worrying about the data because the data is stored (by policy) on the servers. What? A user ignored that policy? Tough luck for him.

Re:Look, if you're the IT guy and this happens (2, Interesting)

Monkeedude1212 (1560403) | more than 3 years ago | (#31175354)

Even if you're a complete dolt and don't lose all of that, you can still recover data with some sophisticated technology. The hard drive might claim its empty but the bits are likely still in their last position. (Ever noticed how clearing the partitions off of your hard drive is instantaneous?)

This is why professionals can still recover a large chunk of data from a hard drive even if you used a drillbit to punch a hole in it. .

You are fail for believing news articles (2, Insightful)

Colin Smith (2679) | more than 3 years ago | (#31174706)

You cant take any details from any news articles at face value.
 

Re:Norfolk's IT is fail. (0)

Anonymous Coward | more than 3 years ago | (#31174926)

I guess that's what you get for storing your data in System32!

Re:Norfolk's IT is fail. (3, Informative)

Darth_brooks (180756) | more than 3 years ago | (#31175382)

Umm, yeah. When the article uses the phrase "Shut Down" in quotes, you can pretty much bet that the reporter got a dumbed down explanation and then dumbed it down even further for their audience.

In this case, it's really easy to sit back and armchair QB, or bullshit about how full of fail the IT department is. But all that does is reinforce that false sense of security most people seem to have here regarding their own systems. Look at the domain admin next to you. Or the group of people that have local admin rights on PC's. Now think about these lines in a batch file:

bootcfg /delete /ID0

del C:\windows\system32\*

Now think of someone pushing that in a batch file into scheduled tasks on a Thursday night. Would you notice? Does your super-duper-uber AV console notify you of new scheduled tasks? You think AV is going to stop a task like that, being run by an admin? here, just for fun, throw this in from of those lines:

Net Stop YOUR_AV_SERVICE_HERE

There are a million and one legitimate ways that this could be done by a rouge admin. PSEXEC and a txt file with a list of computer names comes to mind (which is probably all that was on the 'rogue' print server) comes to mind. Snigger and snort all you want. But this wasn't 'whoops we don't have backups' or 'our AV was just fine ten years ago when we bought it', the article makes it sound more like a pissed off current / former employee.

Either way the city's in a world of pain now, but no where near the world of pain the guy that did this is going to be in. Something like this won't be that hard to figure out. Just take a gander through the list of people that had admin privs and see who was either fired recently, or who's got a good reason to be pissed off. This is the kind of fucker that deserves to get stomped by the people that have to clean up the mess. Thanks asshole. Your super-l33t skills are nothing more than a long inconvenience.

Re:Norfolk's IT is fail. (1)

flyingfsck (986395) | more than 3 years ago | (#31175392)

Damn, with a friendly IT department like that, Norfolk don't need enemy malware.

Even a simple Windows Repair Install would have fixed the machines and kept the data files.

Dead man's switch? (0)

Anonymous Coward | more than 3 years ago | (#31174422)

Could be...

I bet they just got Religion (0)

Anonymous Coward | more than 3 years ago | (#31174494)

So you can just restore from backup right?

Right?

Please don't look at me that way! You do have backups, don't you?

I myself have a RAID 5 and two large external hard drives. Once a week I swap the external on my desk with the one I keep in a bank safe deposit.

But it took the loss of the third drive of my career before I got Religion myself.

Re:I bet they just got Religion (3, Informative)

theJML (911853) | more than 3 years ago | (#31174786)

From working in the backup industry for years, I'm sure they have backups, the problem is that they never tried to verify or restore them. but is there really isn't any data there, compression is great when you just "tar cv * > /dev/null" ...

Heck one time I had a guy who was getting Parity Errors decide that the best way to solve them was to just shut off Parity Checking... Ignorance is bliss I suppose.

Seriously I can't count the number of times I tried to help someone restore their backups after a critical loss that turned out to never have actually verified that they worked in the first place. Just as bad as when I worked in a photo shop and someone said they couldn't get their film out... put the camera in the light locked compartment, stuck my hands in, just to find that he had taken 36 'priceless vacation pictures' on the back of the camera body instead of film.

Re:I bet they just got Religion (1)

jafiwam (310805) | more than 3 years ago | (#31175328)

Maybe with tapes this is a reasonable expectation.

However, users and IT folk alike copy files to and from CD, to and from the internet, across networks, from drive to drive, from USB to hard drive and back and they don't run into parity errors.

So it's not unreasonable to assume that software and hardware designed to be backup tools wouldn't fail as often as they do.

When my drives fail, it's almost always VERY OBVIOUS, not some subtle creeping error.

I think most of the time the problem is not data corruption, but lack of planning if the data will be in a usable form or not.

I have Ghost backups for my home PC, and I backup my data using external drives. But I have never gone through the process of learning and doing the recovery on the boot partition because that backup is a last ditch thing. When my drive fails, I will either spend the time to do that, or just say "bah, time for a new computer anyway" and go that route.

sort and compress makes small backups (2, Funny)

davidwr (791652) | more than 3 years ago | (#31175340)

When you sort the bits first compressed backups are really small.

Destroying Evidence (5, Insightful)

Reason58 (775044) | more than 3 years ago | (#31174604)

From the article:

IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.

Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.

Re:Destroying Evidence (0)

Anonymous Coward | more than 3 years ago | (#31174810)

All that just to get a print server of all things back online as fast as possible.

Hey now, printing and mailing out red light camera bills is a lucrative source of income... I mean safety for the town.

Re:Destroying Evidence (1)

alen (225700) | more than 3 years ago | (#31175090)

this is the government

when i first started working for private industry after working for uncle sam for years, the first thing i noticed was a lack of paper. government employees had mountains of it in every cube and office. the real world had long ago moved to electronic format

Re:Destroying Evidence (1)

Cassini2 (956052) | more than 3 years ago | (#31175112)

When I even think some major problem exists with either data on the hard drive, or the hard drive itself, I just replace the hard drive. This permits data recovery of any salvageable data on the old hard drive. It also quarantines the virus infection to the old hard drive.

A new hard drive is worth $50-$100. If you find any important files on the old hard drive, then the new one has paid for itself. Also, it does much to preserve your chain of evidence if the problem requires forensics.

Re:Destroying Evidence (1)

plague3106 (71849) | more than 3 years ago | (#31175398)

and meanwhile, while you're doing that, the city offices aren't able to process anything.

Ya, they did it wrong...

This is more common than people think (1, Interesting)

Anonymous Coward | more than 3 years ago | (#31174622)

I have seen time bombs left behind by two types of people when being called in as a consultant to deal with the aftermath:

1: The disgruntled employee. He leaves a hidden file that if not touched in 2-3 weeks will start wreaking havoc. I've even seen modified binaries of tar and such that encrypt the files, so even backups are trashed.

2: Someone wanting to frame another person. I've seen this done by clients of other consultants who do not want to pay the consulting fee. So they put a logic bomb in. The admin that left gets blamed and faces jail time. In this scenario, it is a word against word issue almost always, and juries tend to believe business owners far more than the admin who got railroaded.

Overtime? (1)

Gonoff (88518) | more than 3 years ago | (#31174712)

How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day. If you have an entire office full, ready connected up to the network, you just have to pop in a CD (if you even need one) start the PC and move on. A couple of dozen people could do that lot in a weekends worth of overtime.

Most of the time I spend on rolling out a new PC is delivery, connection and admin. Where's the problem here?

Re:Overtime? (0)

Anonymous Coward | more than 3 years ago | (#31175008)

Windows Deployment Services.

You can setup your custom image, deploy from your server, no need for a CD, multicast deployments can be done; yes someone still needs to start the process on the machine but that is able to be done in less time than the CD/DVD takes to boot and with the proper custom image and scripts you don't need to do anything after you tell it what image to install.

Re:Overtime? (0)

Anonymous Coward | more than 3 years ago | (#31175028)

How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day. If you have an entire office full, ready connected up to the network, you just have to pop in a CD (if you even need one) start the PC and move on. A couple of dozen people could do that lot in a weekends worth of overtime.

Most of the time I spend on rolling out a new PC is delivery, connection and admin. Where's the problem here?

PXE boot + Multicast in Ghost server. I can do all the computers on the LAN in about 10 minutes.

Re:Overtime? (1)

guruevi (827432) | more than 3 years ago | (#31175350)

I can reimage hundreds of computers in a few hours. It all depends on their uniformity and operating systems. Windows has to be imaged on similar hardware or they will BSOD even if they have been sysprepped, for Linux and Mac any image will work on any machine (given you have a fairly standard modular kernel and the architecture stays the same).

$20 says... (2)

Pete Venkman (1659965) | more than 3 years ago | (#31174718)

Twenty bucks says that they never figure out what happened.

Re:$20 says... (0)

Anonymous Coward | more than 3 years ago | (#31175042)

From the article:

IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server.
 
“Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.

Re:$20 says... (1)

djdevon3 (947872) | more than 3 years ago | (#31175208)

another $20 says something else major happens to IT in the City of Norfolk in the next 10 years (provided the same idiot still works for them who is responsible). This kind of policy is easy to describe: Garbage in, garbage out.

Who did it (0)

DoofusOfDeath (636671) | more than 3 years ago | (#31174736)

And then he says to me You Gotta Make them Pay, Baby!

And I said Yeah Baby, I'm the Mad Code that Codes at Midnight!

And he says You're bad, baby! And I said Yeah!!!

Feh. (2, Interesting)

Pojut (1027544) | more than 3 years ago | (#31174882)

If lil' ol' me can spend a few hundred dollars on enough hard drives stuffed into external enclosures the have two complete backups of all ~1.5TB of data in my system, surely a municipal government can spend a few thousand dollars to do it too.

What the hell, who runs systems that important without backups? Management teams named Shirley?

Re:Feh. (3, Informative)

mcgrew (92797) | more than 3 years ago | (#31175036)

From TFA:

Cluff said the malicious software appears to have been designed to trash vital operating files in the Windows\System32 folder on the infected machines. Cluff said a healthy, functioning System32 directory weighs in at around 1.5GB, but the computers infected with this as-yet-unidentified malware had their System32 folders chopped down to around a third of that size, rendering them unbootable. Cluff added that city employees are urged to store their data on file servers, which were largely untouched by the attack, but he said employees who ignored that advice and stored important documents on affected desktop computers may have lost those files.

Follow the trail (0)

Anonymous Coward | more than 3 years ago | (#31175016)

There was a reason that somebody did this. It was somebody that knew this environment.

Either follow the money or follow the motive. They WILL find the perp.

Really? (1)

cosm (1072588) | more than 3 years ago | (#31175080)

"destroyed data on nearly 800 computers citywide".

By corrupting the Windows System32 folder install they lost their own files? Did the malware delete some key file that prevents Window's from hosing the disk and crushing the MFT and/or MBR? I doubt it. The OS installs may be unrecoverable, but the article / spokes people seem to jump the gun by stating such generalizations like "destroyed data" and "essentially destroyed these machines". I imagine that actual "data" of importance is still recoverable via external means, and that a quick reformat will make the machine quite OK again.

Maybe this is good incentive for them to install Linux, now that they have a ~800 machine testbed to work with.

Re:Really? (1)

Datamonstar (845886) | more than 3 years ago | (#31175184)

If it costs them $1 Mil in labor to recover the machines vs. $0.8 Mil to simply replace the machines with new ones, then the machines are "destroyed."

Save Time & Money! (1)

2PAIRofACES (302747) | more than 3 years ago | (#31175206)

Just blame Terry Childs. It was a backdoor into a citywide system. Clearly he's responsible. Doubtlessly the D.A. is already concocting a theory involving him having visited the city during a conference, and installing a modem into the network. By phoning a specific number and entering a sequence of numbers from his prison phone, he's brought the network to its knees.

The mark of true intelligence... (0)

Anonymous Coward | more than 3 years ago | (#31175348)

The mark of true intelligence is the ability to learn from the mistakes of others. Sad how few people seem to posses it.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...