Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Bug

Vulnerability Assessment Scanners Comparison 36

Posted by Hemos from the interesting-comparisons dept.
Roberto writes "Network Computing is running a comparison between various commercial and vulnerability assessment scanners - and open-source wins, thanks to Nessus, even though none of the tools could do spot all the vulnerabilities that were present in the test lab."
This discussion has been archived. No new comments can be posted.

Vulnerability Assessment Scanners Comparison More

Vulnerability Assessment Scanners Comparison

Comments Filter:
  • >>It would take a well trained, intelligent human being to discover security flaws.

    Ph34r M3 4nd 411 0f my 31337 fr13nd5! W3 kn0w 4b0u7 y0ur ftpd fl4w!

  • Where are the version numbers, particularly of Nessus. How are we to know that they tested the current version, or a version bundled with a 6-month old Linux distribution?

    One other valid point missed in the review is the frequency of product updates. What mechanisms exist to check for newly discovered vunerabilities? Nessus can be made to automatically install updated scripts, but I have absolutely no idea of the other products reviewed. An out of date security tool can be worse than no security tool at all, as it installs a false sense of confidence. Would you put all your trust in a 1-year-old security scanner - I wouldn't.

  • We use ISS Internet Sec Scanner and it is pretty cool when it comes to updates and reporting. The update tool is called XpressUpdates. Run that program and it's a wizard that steps you through updating all of your signatures. The XpressUpdates also works with thier IDS network sensors.
  • I've worked with closed source scanners that have done some very strange things, such as scanning unauthorized IP address ranges. This can be a very dangerous thing to do in a customer environment. For the product in question, it took several days of vendor help desk contacts to determine that yes, this was a known bug with no fix available for the immediate future.

    My company now uses open source scanners exclusively. We do, however have our own very competent programming staff capable of reviewing and possibly modifying the code. This gives us some assurance as to what the tool is doing, and the capability of fixing problems quickly if required.

    Having a security scanner malfunction due to software error is much more serious than having your word processor freeze. Malfunctioning scanners can crash servers and in general wreck havoc in networks.

  • I'm not really sure who you are or why you are posting anonymously, but I've got sitting in front of me here a signed copy of the letter written by Jennifer Carroll, Marketing and Business Development Manager of eEye. It's dated Aug 25th. If you do indeed work for eEye, I'd encourage you to formally address us. Not only was eEye fully aware of our review, eEye gave us a full license key. And yes, we did update the product. I'm continually amazed by the amount of BS posted on these threads. I'll post the serial # if there are any further doubts.
  • The comparison is quite detailed, considering the fact that it appears in a magazine that can be bought on the newsstand.

    The most interesting part that I find about this entire article is the fact that this magazine (which I subcribe to) is a free subscription. The magazine doesn't make any money off of subscriptions. The magazine effectively makes all of its money from advertisements. The fact that they would review a opensource competitor is surprising in itself. The fact that they gave it the nod, is going to do nothing but hurt their advertising deals with the commercial products that they reviewed.

    Of course, that's only one way to look at it. The other way to look at is that they just effectively said that if you want to get all your vulnerabilities detected, you need to buy at least one thing. Combine that one thing with the open source product, and you've got a complete solution.

    Is the glass half empty, or half full? Hmmm...

  • Almost impossible to do it right (Score:5)

    by scott1853 ( 194884 ) on Friday January 05, 2001 @06:13AM (#528550)
    It would take a well trained, intelligent human being to discover security flaws. If your need for security is more than the average home-based internet surfer running ZoneAlarm, then you should hire a 3rd-party company specializing in security to evaluate your system.

    I would use scanners only to perform automated checks to make sure that known holes have not been opened after the initial check. Periodically, the 3rd-party company should be hired to come back and recheck the system for old holes as well as new ones that have been discovered since the previous system test.
  • shouldn't that be "between various commercial and open source vulnerability assessment scanners"?
    `ø,,ø!
  • The Network Computing Vulerability Assessment Scanner Article is very well written and is particularly helpful to server administrators who have not focused on security issues. I think the Slashdot article could be improved by citing the following passage from the review:
    We set up 17 of the most common and critical vulnerabilities out there, and not one product detected them all.... The closest was the Nessus Security Scanner, which nailed 15 of the 17. But even one hole is too many. Because all the products failed to identify key vulnerabilities, none of them received our Editor's Choice award.
    The comparison is quite detailed, considering the fact that it appears in a magazine that can be bought on the newsstand.

    It may be a bit unfair to take the paragraph I cited out of context because the article goes on to do a good job of weighing the individual pros and cons of the highly rated scanners. Nevertheless, I think the article's key finding is that even the best of the tools they evaluated failed to catch all of the vulnerabilities that they had intentionally installed. Every opportunity should be taken to emphasize this point to the readers.
    --

    Dave Aiello

  • nessus-update-plugins would've helped. My guess is that these people used the stock nessus installation without retrieving the latest scans.

    The two vulnerability missed by nessus are at

    http://cgi.nessus.org/plugins/dump.php3?id=10318 [nessus.org] and

    http://cgi.nessus.org/plugins/dump.php3?id=10260 [nessus.org]

    Again, I'm no security expert, but these people should've at least updated the list.

  • Get a sense of perspective. (Score:2)

    by Anonymous Coward
    The whole point of the test was to see which scanners could be used for proper security tests.

    Closed source software showed itself to be just as good, or in this case poor, as open source.

    The "only open source can be trusted" argument only holds up if everyone looks at all the source, which rarely happens except in the luckiest projects.

    I would trust a competently written and tested closed source product more than a crappy open source one any day. It is a matter of quality. If an open source product is better, or as good as a commercial one then I would use that instead.

    Ideology should never be allowed to get in the way of practical concerns, doing so is hjust another way to shoot yourself in the foot.
  • We set up 17 of the most common and critical vulnerabilities out there, and not one product detected them all.... The closest was the Nessus Security Scanner, which nailed 15 of the 17. But even one hole is too many. Because all the products failed to identify key vulnerabilities, none of them received our Editor's Choice award.

    Seeing as many of the scanners are free and open source.. I would hope that admins would maybe use a combination of 2 or 3 different ones.. which would hopefully capture everything?

    Or are these scanners the lazy way out anyway (compared to servere audits..) and running Nessus and Sara would be just too much work?

  • _Network Computing_ is one of the good guys out there. Since Novell lost its dominance in corporate networking, _NC_ hasn't had a majority of its ad pages from any one vendor (that may or may not correlate to ad revenue). They seem to be pretty vendor neutral and willing to call shots as they see 'em. They have certainly p**sed off Cisco several times in the last year, which takes some courage. This is one of the few trade rags I trust.

    sPh
  • Commercial scanners are not produced by "a person off the street". They're produced by professionals that work for companies that have a significant motive for ensuring the accuracy of their products: money

    You poor, misguided, niave thing. Have you ever worked for a "company" that has "professional programmers"? Let me give you a clue. Programmers are programmers. Regardless of whether they work for a "company" or are donating time to an Open Source project, they are just as prone to stupidity as the next guy.

    Commercial products, contrary to your utopian notion, are produced by professional marketing departments that have a significant motive for ensuring that the average consumer THINKS their product is worth its cost. Accuracy, efficiency, function, and stability have nothing to do with it - the Consumer, more often than not, is never in a position to objectively judge the quality of a product.

  • I thought that was just win.com... :)
  • Did the dickhead that posted originally, supposedly from eEye, get back to you or was it a 15 y.o with more time than brains?

  • The "only open source can be trusted" argument only holds up if everyone looks at all the source, which rarely happens except in the luckiest projects.


    Absolutely false.

    If even ONE person who is not directly involved in the project looks at the source, we ALL benefit.
    Case in point: proftpd. Look at their mailing list sometime. People are constantly checking the diffs from new versions for new bugs that may be introduced. Rouge code would be found and squashed in a heartbeat. I may not look at the code, you may not look at the code, but there are plenty of people who do. This may not be true for all those "version 0.01a" freshmeat announcements, but any mature, sizable project (like, say, vulnerability scanners) it tends to hold very true.

  • If this is the case then why did none of the software detect all 17 of the issues?

  • I'm one of the authors of the article in question, and would like to address a few points being made: 1. Thanks for the feedback - it will be well received. 2. NWC has a lead time of 2-3 months. There is no way around this - the mag goes to film almost a month before the world sees it. So those tests were done back in October - it's not that we are THAT behind, the content is just a little older. 3. Please don't assume that we are idiots. Yes, we applied all of the Nessus plugins. No, we did not install it from some old Linux distro. Ask Renaud who I am, he'll tell you. Give us some credit here guys.... 4. I've been writing for NWC for 4 years, and I have not ONCE been asked to modify my writings due to advertising. NWC takes editorial integrity very seriously. Take that for whatever it's worth... 5. The point on the update times/procedures is a valid one, and unfortunately not one we had room to address properly (we're limited in word count). Next time we do this we'll attempt to address this. Again, thanks for the feedback. -Greg
  • You have a very unrealistic, unhelpful and unnecessarily cynical attitude brought on by reading too much Dilbert.

    Don't bother waking me when you've grown up.
    ~Tim
    --
    .|` Clouds cross the black moonlight,
  • Cisco was invited, but declined to participate.
  • I have reason to believe that this person was, indeed, from eEye, but who knows. No - he/she/it never got back to me/us. -G

  • They missed a very important point - (Score:4)

    by djrogers ( 153854 ) on Friday January 05, 2001 @05:32AM (#528566)
    Using a proprietary (closed source) vulnerability scanner is sort of equivalent to asking a person off the street to give your home a security check. Do you know what internal code audits are done on the software? What sort of 'reporting' it may do during 'updates'? I don't mean to sound too paranoid, but all it takes is one programmer...

    Another, more down to earth point is the ability to write your own checks for the scanner - are you stuck with paying maintenance fees to a company for updates of dubios quality, or can you go out and write them yourself?
  • what about Nmap?

    You can only portscan with Nmap, altough it is very sophisticated. Even Nessus uses Nmap to do part of its scans.

    Nmap is defnitely *not* a vulnerability scanner.
  • Renaud Deraison and Fyodor will be present, and this could be a subject of discussion. http://www.osdem.org
  • Great to see Nessus and the SATAN off-springs included, but it seems they forgot Cisco's Netsonar [cisco.com].

    Jacco
    ---
    # cd /var/log

  • If... (Score:3)

    by PigleT ( 28894 ) on Friday January 05, 2001 @06:35AM (#528570) Homepage
    "Worse, if you are a consulting firm basing your assessment services on these products, you better have some system in place to cover for their shortcomings, as these products don't cut it."

    Er, yeah? A security consulting firm that uses only a few of these as anything more than a starting-point for further hole-research and criticism is doing nothing that I couldn't do myself, and will not seen on my Pigsty. When I consult, I expect to give proper service, and if I get consultants in, I expect perfection.

    "Because all the products failed to identify key vulnerabilities, none of them received our Editor's Choice award."

    If a company relies on an Editor's Choice Award to distinguish good from mediocre from bad, it has altogether too many other problems...!
    ~Tim
    --
    .|` Clouds cross the black moonlight,
  • PLEASE! Automatic scanning *and* hand auditing *every* service is the only way to do vuln. assement or a pen test!

  • Re:One Word (Score:3)

    by tooth ( 111958 ) on Friday January 05, 2001 @05:22AM (#528572)
    I know it was a review of system level security/scanners, but here's my one word (for websites) :)

    whisker [wiretrip.net]

  • nmap is not a vulnerability scanner, it is a portscanner. Does nmap tell you that a vulnerable ftpd is running on host X? No. All it does is tell you that the port is open, and what other ports are open, while providing you with a ballpark guess at the OS (which is better then nothing, but far from accurte).

    -r

  • Great article. I enjoyed very much putting this on the desk of our "security admin" who keeps insisting that we buy a product because "big corporations make better products"

    It should mention how Nmap will help cover the tracks that Nessus misses.

  • Commercial scanners are not produced by "a person off the street". They're produced by professionals that work for companies that have a significant motive for ensuring the accuracy of their products: money. If Joe Blow Security company said your home was secure but it wasn't, they're not going to get a very good reputation around the industry, will lose customers, and eventually get such a bad name they'll be run out of business. Definitely not something the investors like to see. Capitalism works for good and evil fortunately. Open source scanners are the ones that are more like asking a guy off the street. 95% of the time people are not going to be writing your own exploit modules for them so you have to take the word of that guy off the street you got the scanner and modules from to say that your network is secure. If you ARE skilled in writing modules and evaluating your network, then the best thing would be to use a combination of all of these methods. Get a couple of commercial scanners, get some open source stuff, and write some of your own. With the security of your network at stake there is no such thing as overkill.
  • While true that automated checks don't constitute a complete pen, complete pen tests are expensive, while automated checks are quite cost effective. I'd much rather see someone run at least a good automated audit of their site than no audit at all.

    What's sad: Every day we (www.securityspace.com [securityspace.com]) have examples of customers that KNOW they have high risk security vulnerabilities (holes that would get their box rooted according to Nessus), and don't even bother to pay $50 for an automated audit. It's this type of "the net is so big, and I really won't be hit by a break-in" mentality that will

    • move the major banks/credit card companies to introduce security requirements of their on-line merchants (the way I believe Visa will be forcing firewalls as a requirement)
    • force government legislation on security policies and practices (I believe Spain is already moving there on this)

    I'd almost say site operators are getting what they deserve when they are broken into, except for the fact that it is the visitor of the site these days that ends up paying for it...

  • *sigh* Apparently no one reads comments without instantly assuming they're flamebait anymore. If you'd read, what I said was to use a variety of tools. NOTHING will detect all vulnerabilities. You shouldn't expect it to, but by using multiple tools you'll increase your chances of detecting them, especially obscure stuff that maybe only some OSS hacker might have thought of including in his tool. Consider it a second opinion on the health of your network.

  • I was replying to the section of your message that stated:

    ==
    Commercial scanners are not produced by "a person off the street". They're produced by professionals that work for companies that have a significant motive for ensuring the accuracy of their products: money.
    ==

    I was implying that apparently this motivation was insufficient.

    As for the moderation of your post as "flamebait", I certainly did not do that nor do I feel it is flamebait. A typical example of mis-moderation.

  • List of tools (Score:3)

    by raffe ( 28595 ) on Friday January 05, 2001 @07:07AM (#528579) Journal
    Here [insecure.org] is a list of good tools

Slashdot Top Deals

And it should be the law: If you use the word `paradigm' without knowing what the dictionary says it means, you go to jail. No exceptions. -- David Jones

Close