Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

We've Been Hacked... or Have We?

Cliff posted more than 10 years ago | from the better-security-fu-requested dept.

Security 65

hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"

Sorry! There are no comments related to the filter you selected.

One of the first rules... (4, Insightful)

AKnightCowboy (608632) | more than 10 years ago | (#9533959)

If you don't know what was changed then you need to rebuild any machines suspected of being compromised from scratch and restore the data from a clean backup. Unless you're very sure your file checksum database is accurate (you run tripwire or aide hopefully) you will need to rebuild. Don't screw around and contribute to the global insecurity of the Internet.. especially with people's credit card data at stake!!! WTF is your company thinking keeping that on a public network?

firost psot (-1, Offtopic)

TheMysteriousFuture (707972) | more than 10 years ago | (#9533963)

firost psot

Re:firost psot (-1, Troll)

Anonymous Coward | more than 10 years ago | (#9534329)

fucking n00b

Sounds like (2, Insightful)

hammock (247755) | more than 10 years ago | (#9533987)

Clifford Stoll, Cuckoos Egg.

Lesson to learn is secure that stuff, what they don't know will hurt them.

http://www.programming-reviews.com/Cuckoos_Egg_Tra cking_a_Spy_Through_the_Maze_of_Computer_Espionage _0743411463.html [programming-reviews.com]

Re:Sounds like (5, Funny)

PD (9577) | more than 10 years ago | (#9534061)

Those were the days. Adminstrator discovers *one* hacker, catches him, then has enough time to write a book about it before worrying about the next one.

Re:Sounds like (2, Funny)

MarkGriz (520778) | more than 10 years ago | (#9534116)

No kidding. Try catching 'em with a teletype on a 9600 baud connection now.

Re:Sounds like (0)

Frizzle Fry (149026) | more than 10 years ago | (#9534350)

Or discovers one hacker, catch him, and then doesn't catch the next one because he becomes a total luddite, as the case may be.

Some tips for seeing what's going on... (4, Informative)

PhaseBurn (44685) | more than 10 years ago | (#9534001)

Windows: use Moosoft's The Cleaner (http://www.moosoft.com/products/cleaner/download/ ) - it can detect trojans and things that virus scanners don't pick up...

Linux: nmap the box from a trusted PC on the same network, and then build a copy of netstat on the trusted PC for the server in question... copy the binary over, and run "netstat -pultw" as root... it'll list all ports that are listening for connections, and, the processes that are opening them (by PID, and usually by name). The reason for the clean copy is that a lot of root kits replace netstat on infection...

Hope this helps...

Re:Some tips for seeing what's going on... (3, Informative)

benjamindees (441808) | more than 10 years ago | (#9534045)

Ugh. Running *any* binary on a possibly compromised machine is questionable. That's why Windows virus scanners suck: once you bypass them *once* they become irrelevant.

Under Linux, boot from a read-only media (rescue cd) and verify the md5sums of the files installed against the installation cds. Any binaries that have changed will stick out like a sore thumb. Debian (and other .deb based distros) includes a program to do this, debsums, as well as rpm-based distros (Redhat/Fedora), rpm -V.

I don't know if there's anything similar for Windows, but if there is, it probably isn't free ;)

Re:Some tips for seeing what's going on... (3, Informative)

flonker (526111) | more than 10 years ago | (#9535277)

On Windows,...

"netstat -n -a | more" will tell you what ports are in use. This is a simple preliminary check. It will give you an idea of what to look for.

Fport [slashdot.org] is a great little tool that will tell you what processes are listening on what ports. It's many times better than netstat. This will likely give most trojans. Look for ports you don't recognize, and programs you don't recognize. Keep in mind that fport lists outgoing connections as well as incoming ones, and doesn't differentiate. Any ports or programs you don't recognize, google it and figure out what it is, how it got there, etc.

Next, check the standard startup locations. HKLM/Software/Microsoft/Windows/CurrentVersion/Run *
and the same under HCU/Software/Microsoft/Windows/CurrentVersion/Run* are the most popular places trojans are placed.

Also, look for hidden directories, and large files. You may be hosting an FTP Warez dump. Look for *.nfo;*.rar;*.ace;*.0*;*gamez*;*appz* and anything else you can think of.

Unfortunately, I don't know of any Md5 sum tools or anything for windows.

Finally, rebuild, if you can. Rebuild from current data, and known good code. Don't trust code on the compromised machine. Best practice for recovering from a compromise type stuff. That really should be your first, last, and only step, but I doubt you'll be allowed, considering that your boss isn't taking security seriously.

Re:Some tips for seeing what's going on... (2, Informative)

zoloto (586738) | more than 10 years ago | (#9535593)

Here is a WinMD5 for Windows:
http://www.blisstonia.com/shareware/WinM D5/
WinMD5 [blisstonia.com]

This works very well and it's simple.

Re:Some tips for seeing what's going on... (1)

k12linux (627320) | more than 10 years ago | (#9559929)

Of course, don't trust the results of ANYTHING you run on a compromised system including what you see in regedit.

Re:Some tips for seeing what's going on... (3, Informative)

CyberVenom (697959) | more than 10 years ago | (#9535508)

Even the clean netstat binary will not work on a Linux machine that has been compromised with adore. Adore is a kernel module that blocks netstat (and others) at the kernel level from seeing the compromised ports, as well as making certain parts of the filesystem (such as those holding the backdoor software) invisible to userspace. Really an amazing little hack.

The only real way to bypass it is from outside the compromised kernel, which means either a clean boot off of a boot CD or such (which would be pointless for netstat monitoring because the system should be clean at that point anyway) or a view of the net traffic from an external clean machine.

I would recommend the external sniffer method. I would use ethereal and/or tcpdump on a clean machine to monitor traffic to the suspected compromised machine, and filter out "normal" traffic. (of course this will not detect a backdoor that is piggybacked ontop of another valid protocol such as HTTP if you are filtering out "normal" traffic, but you gotta start somewhere...)

Of course, if the hacker were really slick, he might even flash some malicious code into the BIOS so it could survive reboots and even reinstalls, but in reality, how many really good hackers are out there? (Just kidding, but not about the BIOS idea!) Of the rootkits I've seen so far, adore is one of the best.

How to spot what is happening (2, Interesting)

Karora (214807) | more than 10 years ago | (#9534004)

Here's a couple of things you could do:

Download and build chkrootkit [chkrootkit.org] . This will detect a lot (most?) stealthed kits on Linux systems, and it is always my first port of call when I'm invited in to clean up after a breakin.

Plug in a hub (so all traffic can be seen by multiple machines - a switch ain't as good, unless it has a monitoring port) in front of the machine(s) and run tcpdump or ethereal on another system to watch traffic from the machine. This will let you watch exactly what traffic is happening on those weird ports, or watch outbound SMTP traffic for spammer activity.

We don't put Windows-based systems on the internet, partly for security reasons, and partly because we don't have any Windows specialists, so I can't help for on-the-box detection there, although I would expect a commercial virus scanner should find everything.

Re:How to spot what is happening (3, Informative)

ManxStef (469602) | more than 10 years ago | (#9536926)

Another good rootkit checker, which seems to have a more active development cycle, is Rootkit Hunter [rootkit.nl] . Here's a Newsforge article on it [newsforge.com] , with a few more details.

A few other comments:

Virus scanners won't help on jot against a custom hack (as Valve found out, for instance). They can be helpful, but don't put full reliance on them.

Running an Intrustion Detection/Prevention System such as Snort, Samhain, Prelude, etc. will help you manage the monitoring side of things; more than a few machines becomes a pain without additional help. Also take a look centralising all your logs on a syslogng server or something similar, if you don't already (note that there are various solutions out there to get Windows boxes to log to a syslog server).

A honeypot may distract the hacker from your production servers for long enough for you to identify that there's a problem.

Also take a look at "HoneyTokens": specifically created database records that trigger alarms if they're accessed - usually high profile fictious targets that would make excellent trophy hacks - there's more info on this over at SecurityFocus [securityfocus.com] .

If you suspect that a machine has been compromised, as other have said, the ONLY WAY TO BE SURE is to rebuild the box from scratch. While this may be a real pain, hopefully it'll help you get the procedures in place to make this as painless as possible, so it's not all bad.

Perform security audits/pentests every now and again. Tools like Nessus [nessus.org] help: here's a good series on using Nessus [securityfocus.com] (part 2 [securityfocus.com] , part 3 [securityfocus.com] ).

Get familiar with security tools such as the top 75 recommendations at Insecure.org [insecure.org] (home of Nmap).

Remember that security is a PROCESS, so be thorough; get an entire plan together and cover all the bases that you can, taking special care to identify and cover the weak points. Your company's security is only as good as its weakest link; for instance, priviledge escalation of weak user account passwords is a good one.

Read SecurityFocus [securityfocus.com] , PacketStorm [packetstormsecurity.org] , CERT [cert.org] and the like, and try to get involved in their communities; they can be invaluable! They're also got a lot of good tutorials, such as how to lock down Apache, IIS; securing PHP, ASP; etc.

Would Firewall Analytics Help? (1)

philntc (735836) | more than 10 years ago | (#9534009)

I was looking into ways of approaching this type of problem, and it seems that a good firewall monitoring solution could help. One I came across is made by OnGarde [ongarde.com] . They run a service that allows deep inspection of traffic, and they support alot of devices.

tcpdump/snort/ethereal/any sniffer you can use (3, Informative)

Korgan (101803) | more than 10 years ago | (#9534039)

First thing you are going to want before you can go to the boss and shove it up his nose is proof. Put a machine on a monitor port of the switch right beside the egress point of the firewall and sniff away. You know what sort of traffic should be going out so you can filter a lot of that out. You also know that given you're hosting sites, most of your connections are going to be originated from the outside, coming in. So filter those out and look for connections from the inside going out.

Ultimately you're going to need to build up some pretty decent filters or you'll just overwhelm the machine doing the sniffing, but if you know what traffic should look like going across your firewall, you can look for the anomalies quite easily.

Probably the easiest way for you to do this would be to use something like Snort with some pretty decent rules. The downside to IDS is that it takes a lot of effort to get the rules setup properly for your network, but that could easily be done in your spare time, off the clock as it were.

Virus scanners aren't always the best solution to finding back doors and such. On Windows there sure to be malware out there that just hasn't been noticed yet because it doesn't do anything overtly "virus-like". Trojans and malware like NetBus or BackOrifice (to use two very well known examples) are easily altered to hide from many AV apps (just that no one bothers anymore).

Because of this, any machine you think is compromised should be rebuilt and patched up from scratch. Once you've got it rebuilt, and before you put it back on the network, use Tripwire (or even a shell script) to take a baseline MD5 hash of every single windows and application file on your harddrive. This has two advantages. 1) You build up a list of known good files on your machine. 2) It becomes easy to spot new/added files and investigate them. You can do this on any platform, *nix or Windows.

To get the evidence you want though, you're going to need to use tcpdump/snort/ethereal (or any other sniffer you are comfortable with) to prove there is traffic going across your [firewall|egress routers] that shouldn't be.

OT: Your Sig (1)

Nasarius (593729) | more than 10 years ago | (#9534077)

In case you didn't know, that's not a Voltaire quote. It's a description of Voltaire's beliefs, as written by Evelyn Beatrice Hall in "Friends of Voltaire".

Re:OT: Your Sig (0)

Anonymous Coward | more than 10 years ago | (#9534419)

Yes, we may not agree with him saying that it is a Voltaire quote when it isn't, but by God I'm going to defend his right to say it!

Hmm (Score: -1, Too Obvious).

Re:OT: Your Sig (1)

Korgan (101803) | more than 10 years ago | (#9534678)

A little research on Hall has turned up...

----8----
Hall herself claimed later that she had been paraphrasing Voltaire's words in his Essay on Tolerance:

"Think for yourselves and let others enjoy the privilege to do so too."

Hall died in 1919.

In his A Book of French Quotations (1963), Norbert Guterman suggested that the probable source for the quotation was a line in a 6 February 1770 letter to M. le Riche: "Monsieur l'abbé, I detest what you write, but I would give my life to make it possible for you to continue to write."
----8----
http://www.classroomtools.com/ voltaire.htm

Thanks for pointing this out. I had never even heard of Hall before this. However, in my defense, from that paper:

"In any case, the phrase was too eloquent, so it became quoted, and famous names attach themselves to quotes, to the detriment of the less well-known originators."

To be honest, I had completely forgotten I even had a sig setup as I've configured / to not show them.

Re:OT: Your Sig (1)

NighthawkFoo (16928) | more than 10 years ago | (#9559976)

I had the same sig with the same error in it. Someone here pointed out my mistake and it's been correct ever since.

Er, what do you mean "strange"? (1)

dasunt (249686) | more than 10 years ago | (#9534058)

but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, ... strange ports being open...

Perhaps I'm being naive, but what's preventing you from using 'netstat', 'nmap -sV', and plain old 'netcat' to figure out what the strange ports are doing?

Re:Er, what do you mean "strange"? (1)

Hubert_Shrump (256081) | more than 10 years ago | (#9534446)

just for one - any binary on your victim machine has a possibility of being compromised... so netstat data would be questionable.

but nmap and netcat would be fine. from elsewhere. maybe nessus too, while you're at the old security console.

Re:Er, what do you mean "strange"? (1)

Jerf (17166) | more than 10 years ago | (#9534500)

In a word, rootkits [tech-faq.com] .

In theory, once a root-level compromise has occurred, you can trust nothing a computer says anymore. Crackers have, through steady effort, made that almost 100% true, and if they aren't there already, they don't have far to go.

No binary can be trusted on a hacked system.

For the curious, I recomend Googling "rootkit"; there are a lot of helpful resources out there.

mystery worm out there (5, Informative)

imsmith (239784) | more than 10 years ago | (#9534059)

There are stories just now breaking into the general consciousness of IIS servers that have been compromised for months and feeding executable code tacked on the end of http calls to IE and spreading malicious code and feeding something (reports and opinions vary) to servers in Russia (also with varying opinions as to the owners - organized crime seems to be the evil de jour).

These compromised IIS servers often have the server attached to the explorer.exe process and are therefore not detectable by virus scans. Using netstat or filemon you can find the open ports. The only solution is a bare-metal rebuild.

Have fun if that is what you are dealing with.

As an aside, if this company is unconcerned about the compromise of credit card information you might want to find a new place to hang your hat. The civil and criminal liabilities are pretty steep for the compromise of financial transaction information (if you are in the U.S.) and they extend to individuals inside the company, not just the board and officers.

Re:mystery worm out there (1)

Neop2Lemus (683727) | more than 10 years ago | (#9534231)

What you say is excellent, I wouldn't have thought of that.

But by posting this question to /., and by being able to show a record of his attempts at fixing the problem, he should be off the hook. He can point to this story as a time-stamped description of his stopped efforts. The the blame would shift to his boss.

But I could be wrong.

Re:mystery worm out there (1)

GreyyGuy (91753) | more than 10 years ago | (#9534264)

Heh. You don't work with any lawyers do you? Posting a question to Slashdot is not even remotely close to due dilligence or evidence of a conversation between him and a third party.

Re:mystery worm out there (1)

jazman_777 (44742) | more than 10 years ago | (#9534265)

But by posting this question to /., and by being able to show a record of his attempts at fixing the problem, he should be off the hook. He can point to this story as a time-stamped description of his stopped efforts. The the blame would shift to his boss.

I am not a lawyer, but I play one on Slashdot!

But I could be wrong.

Yup.

Re:mystery worm out there (1)

Beryllium Sphere(tm) (193358) | more than 10 years ago | (#9535328)

>varying opinions as to the owners - organized crime seems to be the evil de jour

If they're in Russia, and making money illegally, and not owned by organized crime -- they will be soon.

Tell ya what... (5, Funny)

FFFish (7567) | more than 10 years ago | (#9534066)

...post the IP address here, and I'm quite certain your worst fears will be so perfectly confirmed that your boss will have no choice but to admit you were right!

Re:Tell ya what... (4, Funny)

Anonymous Coward | more than 10 years ago | (#9534317)

66.35.250.150

Re:Tell ya what... (1)

Lehk228 (705449) | more than 10 years ago | (#9535356)

that looks like slashdot's IP to me....

Hire an expert. (4, Insightful)

ajayrockrock (110281) | more than 10 years ago | (#9534068)

Listen to your boss and do what you do best, web programming. But convince him to get an outside security consultant to scan your network for problems and then listen to their advice.

My company has an outside security company run quarterly checks against our network, and they sometimes catch stuff that I miss. Just don't let them talk you into buying a over-priced checkpoint firewall when all you need is a Linux box and Iptables.

--Ajay

Re:Hire an expert. (1)

k12linux (627320) | more than 10 years ago | (#9560216)

Don't just recommend this to your boss in a conversation. Write him a memo (yeah.. the paper kind.. remember those?) In it explain why you think systems are still compromised and that all of your customers' data (possibly including credit card and more) may be at risk.

Just make sure that if he does nothing it is clear that he was warned and that you tried to do the right thing. Then make a copy of the memo for yourself before giving it to him.

Finally, follow your boss's orders and get back to web programming. Until the security sweep, assume every box IS compromised and don't do foolish things like VPN into it or use the same password you use on your own systems/accounts.

If the boss does nothing and things get really bad, I suppose an anonymous tip to the FBI is in order. (After all, if MY credit card info were on one of those servers, I'd want something done.) When customers start sueing, you have your memo to use as a shield for yourself and to stick it to your boss in court.

Do your job? (4, Insightful)

legLess (127550) | more than 10 years ago | (#9534081)

Quoth the poster:
I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?
Not to be a dick, but did you miss the part where you got clear directions from your boss not to try to "detect these guys?" It seems like you identified a problem, brought it to management, and they told you to ignore it. Sounds like you need to decide if you want to be an employed web programmer or an unemployed security consultant.

Re:Do your job? (4, Insightful)

ehetzner (94010) | more than 10 years ago | (#9534202)

Not to be a dick, but did you miss that part in life that went something like this: you were right, boss was wrong, shit hit the fan, you got blamed? That is standard operating procedure, and if you don't look out for it, you're going to be the one in trouble.

Re:Do your job? (2, Insightful)

legLess (127550) | more than 10 years ago | (#9535024)

That's certainly true, but he's obviously made his boss aware of the problem (in writing, if he has any brains), so he seems fairly well covered. If the organization's so broken that it would nail him in the face of evidence that he tried to fix the problem and his boss told him to get stuffed, well, he's fucked no matter what.

Welcome to the wonderful world of corporate webdev (5, Insightful)

ezraekman (650090) | more than 10 years ago | (#9534145)

I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline.

You're already doing your job, and not being listened to. Since I'm not a sysadmin, I've got no direct advice for you regarding the tracking of such activity. However, it seems to me that this is the smaller of two problems. The first is being able to do your job from a technical perspective. The second (and it seems, more immediate) problem is being able to do your job from a political perspective.

Your boss has already watched his public facing site(s) and servers go down due to his failure to listen to you. Now it sounds like he's about to make the same fatal mistake. This, of course, places you in the lovely position of having to remind him that he's about to make another major tactical error... but you also have to do so in as subtle manner as possible, so it doesn't sound like a recrimination. All I can suggest is to try to make it blatantly obvious to him without coming out and saying it, thus giving him the opportunity to "discover" his error and correct it on his own.

One other suggestion: document, document, DOCUMENT! Make sure that you can prove later (should it be necessary) that you did everything you could. This is another area of vital importance for your job security that also must be done very carefully. Simply CCing the higher-ups will likely piss off your boss (and possibly the folks you're CCing as well), and may look like unnecessary whistle-blowing or complaining. Do it as unobtrusively as possible, but make sure that you're covered, in case there's any question later.

One possible solution to both problems is to communicate all of your concerns in an e-mail. Write it during your lunch hour, so he can't get upset at you for "wasting" more company time on it. Make it clear exactly why you think there is an issue, and mention (if you can do so gently and without provoking him) the past incident. Remember that if you want someone to do something, you shouldn't tell them why *you* want them to do it. They don't care about what you want. Tell them why *they* want to do it. Best-case scenario, he listens. Worst-case scenario, he gets a little more annoyed at you, but you've got your documentation.

I don't envy your position, and wish you luck.

Run, don't walk. (5, Insightful)

gaudior (113467) | more than 10 years ago | (#9534203)

Get your resume up to date, and get moving. This outfit clearly has no real concern for security, but since you've started poking around, asking questions, and raising hell, you are the messenger that gets shot when the company really does start to take security seriously, if they ever do.

On the other hand, if they continue as they are, they may not survive, and you are screwed again.

Re:Run, don't walk. (1)

innosent (618233) | more than 10 years ago | (#9534740)

Have you worked in the US? They are already outsourcing jobs, do you seriously think he should quit to find the magical place you call a "company" that takes security seriously? Where is this place? Do they pay well? Company executives have a responsibility to focus on their core business, making a profit, etc., most started working before the letter "e" was prepended to every word in the language, and don't even know what the internet is, much less how important security is. They know buzzwords, sometimes, but there aren't many buzzwords to describe why you need to double your department's budget to focus on security.

Watch out (3, Interesting)

schmaltz (70977) | more than 10 years ago | (#9534281)

Since you're asking "how do I disinfect these boxes" (essentially), you make it pretty clear you don't already know how. You could download Nessus and chkrootkit, and maybe get lucky - e.g. the cracker who's built a nest inside your server doesn't detect your attempts to detect him.

The worst case scenario here is that you detect a problem, attempt to fix it yourself, and trigger Something Bad[tm] in the process: the cracker retaliates, or you break a working app because you upgraded something out-of-sync with glibc (or whatever), or you otherwise become the catalyst for noticeable downtime that will piss off your boss and get you fired, or worse - they turn you into a scapegoat (see the Intel case against that security chap.)

Just make sure you cover your ass. You've notified your boss, copy those emails to a nice safe place (headers and all), and don't do anything stupid.

Best scenario is to build a fresh box, backup the old box's data, restore it to the new box (clean! no code! only data!) Don't bother trying to salvage a compromised O/S installation. Too many things to miss. And, when you're building fresh, don't ssh via one of the infected boxes! Don't inadvertently give *any* info to the crackers that you're setting up a new machine. Better yet, build it with the ethernet cable unplugged, if possible. Do it from CDs.

Sounds like a breakin... (2, Insightful)

Mercury2k (133466) | more than 10 years ago | (#9534299)

Anyways, the subject says it all. Its very possible that you have been comprimised if your kernel is panicing. I think its time to boot from a safe copy of your distro (or some other custom distro...etc) and confirm the checksums of everything and do a good scan of the filesystems.

Also, one good thing to do is place a clean box in between the comprimised server and its internet connect and run tcpdump/tethereal on the brided connection. The first thing you need to do is be able to identify _all_ traffic going in and out of the box. If you dont know what something is, research it. You never know what you might stumble upon.

Good luck.

shut up, or quit (0)

Anonymous Coward | more than 10 years ago | (#9534397)

Imagine you are a chef. Maybe not the world's greatest but you can sear a mean steak. You've been out of work for a while and you find a nice-looking restaurant, do the interview, and you're hired.

Once you step into the kitchen, you want to throw up. Cockaroaches everywhere. Nobody washes anything. They scrape off the filth before using the pots and then toss them in a pile. When the wind is right, you smell rotting food.

You point this out and the boss says, sorry you're a chef, not the cleaning crew, mind your business.

You have two choices:

#1: do what the boss says (and DOCUMENT all conversations because at this kind of outfit, you will be the first to get blamed when the restaurant gets sued by a sick and angry customer).

#2: quit, find another job, even if you have to take a pay cut to find one. Find a place that harmonizes with YOUR idea of a good place to work.

I would take #2 every time. Your example is not the kind of place I'd like to work, either as a programmer or a sysadmin. EVERYBODY should be acutely aware of security issues, and willing to hear anything that improves security. It's like not having insurance, or bathing in sewage with open wounds. You'll eventually regret it.

If youre boss is not a techie (3, Insightful)

mnmn (145599) | more than 10 years ago | (#9534418)

All you have to do is stand straight and announce that we've been 'hacked'. If they ask to what extent, how bad etc, just say we've completely been hacked. Its all gone.

You'll be given all the time and budget to fix it. FUD doesnt always require proof, unless someone calls in some consultant.

Resume time (1)

kalidasa (577403) | more than 10 years ago | (#9534648)

think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security.

You've got a money hemorrage on your servers. Don't bother trying to talk sense to your boss, just strengthen up your arms and be read to jump ship and swim when the next boat comes along. Start shopping your resume now, and remember: when everyone is out to get you, paranoia is just good sense.

Undetectable spammers (-1, Offtopic)

DaoudaW (533025) | more than 10 years ago | (#9534993)

Meanwhile, the average Internet surfer is left with few options. Besides choosing the highest security settings for Internet Explorer, Windows users could download an alternate browser, such as Mozilla or Opera. Mac users are not in danger. Are you married to IE or what???

Sue them. (1)

J2000_ca (677619) | more than 10 years ago | (#9535047)

Honestly letting a credit card server be comprimised and not doing anything about it until it becomes urgent. Then ignoring more possible exploits? If I was a customer I would be might pissed. Buy something from your company and then go get you a lawyer.

Re:Sue them. (1)

Spudley (171066) | more than 10 years ago | (#9536129)

There are a *lot* of companies out there with that kind of attitude. A hacked server is not seen as a problem unless it's immediately noticable - that is, the page has been defaced or the machine has gone down completely. If you can't see the problem, people will tend to ignore it.

And therein lies the problem. As a consumer, there are thousands of companies I could buy from online, but as long as security is given such low priority, how can I trust any of them?

Let the boss know... (1)

ewhenn (647989) | more than 10 years ago | (#9535179)

...but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised.....and says that I need to be working on paying work, not security.....

Tell him that keeping your servers secure IS paying work. As a customer, if I know that a company is not keeping my credit cards, etc. secure, I will not do business with them. Ask him what would happen if there was a big 'scandal' (for lack of better terms), and thousands of customers credit was exploited. How much of an opportunity would you have to sell your so called 'paying' work then?

Sounds Kinda like my job.....LOL (2, Interesting)

Anonymous Coward | more than 10 years ago | (#9535310)

While my situation isn't *exactly* similar, I am interested in people's thoughts...Here goes:

At my job, I am one of two web developers. Besides us, there are the two owners and our systems admin. The owners want to become a viable commercial hosting service with secure storefronts, etc. Fine says me.

The problem lies in that one of the two owners (The husband) is a pig-headed idiot. Recently he asked us to implement a RAID solution for the webserver (notice the lack of an 's' at the end of webserver). Not a problem says sysadmin and myself, we come up with the plan, and present it to him, it involves RAID-5, blah, blah, blah, all the standard normal stuff that people do...He quickly scoffs at the idea, hands us a OLD P.O.S. [ebay.com] with a couple 10'ish gig drives and says make it out of this and use RAID-1, and promises to order some large drives for the machine, so we can implement his RAID-1 solution but insists that nobody in their right mind would ever use RAID-5...

We of course are like what the hell? You want something that is enterprise level, and expect us to make it out of this P.O.S.???

We resign ourselves to doing the best we can with it and get a crummy webserver up and running with Slackware.. It is not the fastest machine, but it works for now. We currently have no RAID, becuase the large drives he said he would order have not been ordered for 2 months.

A couple weeks ago, we are talking about a file server for our internal software, etc. and he loves the idea..GREAT!!! So we spec out a modest system that will fulfill our needs and he says, oh I have a perfectly good fileserver at home that you guys can use to make it, and the COOL thing about it is that it runs on something similar to (but not) this [rebyte.com] . We research his little linux memory card thing and yes, it is cool, however it is not capable of doing what we need to have it do, and from what we can find out about it online it is not capable of performing one of the tasks without substantial work being done, the least of which is compiling and installing netatalk (which is no big deal) for some machines that cannot connect via Samba (MacOS 9 that would require DAVE [thursby.com] that he is not willing to purchase).

So we add in the 2 120-Gig Drives to the PII/166 with 64M RAM that he gave us to make a file server from and find out that

  • Floppy drive is bad
  • 1 on-board IDE controller is shot
  • CD-ROM Drive is broken
So we switch out to some other P.O.S. motherboard he has lying around and find that it has some popped caps, so it won't work either.

We eventually come up with a working P.O.S. motherboard, put some WRONG entries in the BIOS to make it recognize the 120Gig Drives and install a 4.xG drive to use as the system drive running samba and netatalk. All is looking well...

So we get FreeBSD installed today and are in the process of setting up the Xserver so he can have his GUI, since he doesn't know jack about the command line and then we are going to implement his RAID-1 that he loves so well on this machine also. We are stoked to say the least that after all that hassle we have a working system and FreeBSD sees the entirity of the 120G drives...

So he comes into work today and sees XWindows (twm) up and running and asks what we are doing, and we tell him, making the file server like we had talked about. He asks what it is running and we tell him FreeBSD 5.2.1 and we are finishing the config, then implementing the RAID and it will be ready to go. This is where it gets good.

He flies off the handle and says, did I tell you to use FreeBSD?? We are like...ummmmmmm...we talked about that memory card thing and explained to you why it wouldn't work and therefore have implemented a solution that fits our needs like we talked about. His response was "I told you to use the memory card thing and just make it work!! You don't listen, you want to do things your way and don't care what I want and I don't have to listen to you because this is my company"

Our response is "We explained the technical details as to why the solution you *want* to use is not a viable solution and the reasons that it will not work [name], we can change it out in a few seconds, but then we will have a system that doesn't do what we need it to do, and we will just end up having to rebuild it anyway"

He just gets more pissed off talking about how we don't listen to his expert advice, because he knows what he is doing, blah blah blah...The other web developer, myself and the systems admin all know that he is a *user* at best.

I couldn't take his incompetence any longer today so I left work early and told him I'd be back monday...

I am at a complete loss as to what to do other than bail on the job...He would rather implement something that everyone knows will not work because he thinks it's cool than to do something right[1]. He refuses to try to understand our reasons for doing things a way that he doesn't think is cool and the worst part (for me) is that after a customer brought their machine in and myself, the sysadmin and the other web developer determined that it had a bad hard drive, he had to check for himself, by taking the machine completely apart changing cables everywhere, and spending like 5 hours of time (that he will charge the customer for) to finally determine that which 3 of us had already concurred on.

Between the 3 of us that are not owner/owners spouse we have ~40+ years of experience in various aspects of I.T. all of us having a healthy bit of experience in hardware repair and upgrade as well.

I am at my wits end with this idiot..advice?

[1] By right, I mean with the available P.O.S. hardware that we are allowed to use for projects...

Re:Sounds Kinda like my job.....LOL (1)

Jinsaku (729938) | more than 10 years ago | (#9535460)

I worked for a guy like that once. I was job-hopping and this guy had big ideas, and the price was right. However, I was the *sole* IT/Programmer/Webmaster/Network Analyst/etc guy.. after about 2 months of him telling me that because he made his fortune in Metalworking, he knew more about computers than I did, I hopped away.

What I'm actually surprised at is that three established guys (for which you say have 40+ years of experience between you.. hope you're not counting high school computer classes :P) are working at a show like that.

Find another job, it's not going to work out.

Re:Sounds Kinda like my job.....LOL (2, Insightful)

dtfinch (661405) | more than 10 years ago | (#9535520)

He's paying you. So you're sort of obligated to tolerate him. Occasionally my boss buys unusual things but never like that. The best you can do is to educate him and do a good job despite his mistakes.

Maybe ask for an IT budget whereby you and the others get a fixed amount of cash to spend on hardware, since he's obviously paranoid about spending (forgetting that he pays you tons more to deal with the crap he buys) and wants to avoid getting something more expensive than it needs to be, and you want to avoid spending on things that don't meet your needs. Even a measly $500 can go a long way when you, the expert, has to do the shopping.

Re:data managers (2, Insightful)

bfg9000 (726447) | more than 10 years ago | (#9537006)

I've seen situations like this in the past many times, especially in smaller companies where the boss started it and built it up by himself and feels the need to micro-manage.

What I said to one guy like that is "Sir, we respect you and will do whatever you want us to do here, because you're the boss and it's your company, but you hired us to take the load off you so you'd be able to do less work and make more money. Trust us to do a good job and we will. We might not do everything exactly the same as you would, but we know the end result will still make you happy."

Of course, it takes more than that to change a personality flaw that massive, but it's a good start.

Re:Sounds Kinda like my job.....LOL (1)

kronosaurus (621035) | more than 10 years ago | (#9557754)

You can't help everyone. If your boss doesn't respect what you have to offer then sooner or later you are going to get fired.

The longer you stay under these conditions the worse your reputation will be. The money you are making had better be enough to cover personal attacks on you by your boss that may haunt you in the future.

Nuke/Pave (1)

Lehk228 (705449) | more than 10 years ago | (#9535371)

you need to nuke and Pave the OS's sumultainiously if possible, and restore data from a trusted backup if you have it, otherwise notify customers that THEY need to restore from THEIR backup (they do backup their data right?).

simple, ntop (1)

Kevin Burtch (13372) | more than 10 years ago | (#9535499)


Forget running ethereal or other packet sniffers, they're far too fine-grained for what you're trying to do: prove they're being abused.

Connect a small box running your favorite *nix running ntop [ntop.org] to the service port (or whatever they call it, I'm half asleep) of the switch/router to which the box(es) in question are connected.

That's it.

Ntop will give you very nice graphs to print out for the guy who needs a clue, showing not only the IPs involved, but the ports, percentages of traffic per protocal/port/whatever.

If they're being used for SPAM, for example, you'll see tons of outgoing port-25 traffic.
Just make sure that's not what the box is supposed to be doing!

If this isn't enough for him, and you're sure this is something against the company's policy, bring it to his boss' attention immediately (calmly) with a full explanation of why you are doing so.
If you delay this action, you risk having your boss "poison" his perception of the situation, and end up thinking that you're a troublemaker (that is, if the politics are as bad as I'm guessing they are there).

So either fight it all the way, or drop it... or be prepared to find a new job (these situations always suck).

Security is Paying Work (1)

basking2 (233941) | more than 10 years ago | (#9536751)

Security is paying work, unless your company has a reputation without worth. You can get that sentiment out of any of the good security books out there, but saddly management can only see the spending cycle, not the averted crash-burn-patch-fix cycle. Remind him that if we were all "paranoid" to begin with, we wouldn't have had a hacker tell us to patch our stuff. What if the hacker didn't like you so much?

Security really isn't a game or a buzz word. Folks have to own that concept, not just parrot it.

You are a web programmer. (1)

MatrixBandit (709610) | more than 10 years ago | (#9536799)

I was in a very similar situation awhile back when the call center I worked for had some major internal and external network security issues. When I initially stumbled apon this I emailed the IT dept explaining the issue and I was quickily brushed off and told it wasn't an issue. Not being satisfied with that I dug a little deeper and was fired for it. They ended up making wild accusations regarding my intentions etc etc.

So my best advise to you is that if it's not in your job description, leave it the fuck alone. You don't want to end up being the scape goat should something happen or should you step on someones toes in the process of honestly trying to help. Besides; if they want network security, they need to open a position and pay for it.

Re:You are a web programmer. (1)

DavidH_Mphs (657081) | more than 10 years ago | (#9537047)

I couldn't agree more.

If you find a problem, whether it's security or something else, notify the appropriate people (CC your boss). If they chose not to fix it, that's their problem. At least your a** will be clean if/when there's legal action resulting from the issue.

explain it to your boss (0)

Anonymous Coward | more than 10 years ago | (#9536830)

tell him about rbl's and how his company would loose the ability to reliably send email if your netblock ended up on some of them. also, install and run ethereal and/or ntop and see what is really going on on your network

This sounds like a job for ... (1)

deek (22697) | more than 10 years ago | (#9536841)


... Snort! [snort.org] Faster than a speeding packet, more powerful than a trojan, able to detect small intrusions with a single cycle.

But seriously, get a Snort box installed and be more active(*) in your intrusion detection. Surely your boss can't object if you slap down a printed snort long on his desk, and show him proof of intrusion.

dave

(*) I absolutely refuse to use the word 'proactive'. I'm not playing buzzword bingo here.

Nuke the site from orbit. (1)

anonymous cowherd (m (783253) | more than 10 years ago | (#9536894)

It's the only way to be sure.

Also have a look at EtherApe (1)

cheros (223479) | more than 10 years ago | (#9537004)

I've seen all the usual goodies mentioned (Ethereal, tcpdump, nmap, nessus) but I found when it comes to detecting a virus infection I find Etherape [sourceforge.net] impossible to beat.

The main reason for that is that etherape actually maps out live traffic patterns (and volume) on a network. I personally am quite surprised it's not a default part of more security bootdisks. You'll spot an infection straightaway as they tend to 'broadcast' on the network.

As for protection, host checksumming has one disadvantage: it's AFTER the fact...

Good luck - you'll need it. You may want to remind your boss that being negligent with CC data might lose you your merchant status, and losing customer date could result in lawsuits from clients or, in EU countries, a harsh fine from data protection regulators. IANAL but frequently involved in cleanign up situation you just described...

The Bottom Line (0)

Anonymous Coward | more than 10 years ago | (#9538101)

Your boss seems to be interested in the bottom line $$$. So thats the answer to the problem. It seems (not being a lawyer) that your company would be responsible for securing the credit card data. A large financial company recently had to disclose to its customers that it (or rather the offsite storage company) had lost one of its backup containing customer information.

Even if they don't take the CC data, its costing your company bandwidth (you can't serve up your data if your pipe is full serving the spammers and viruses), disk space (with viruses), CPU time from running those "extra" programs, and downtime either from being hacked by someone who is tired of your contributions to the network, your provider taking you offline because your causing them problems from the GIGO, or the worm itself killing the system.

It will also cost your company lost revenue due to the loss of business from your customers moving else where due to the services issues resulting from the above.

Even if he does want you to work on something that will "profit" the company, how much will it cost vrs save the company from bringing in a consultant to evaluate the systems, practics and policies? I bet its cheaper than going out of business.

warezzzzzzzzz (1)

COBRAws (740616) | more than 10 years ago | (#9604648)

lol, heyyy share your server with us! i want some 0days hahahaha
Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?