Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Applied Security Visualization

samzenpus posted more than 5 years ago | from the protect-ya-network dept.

Security 45

rsiles writes "When security professionals are dealing with huge amounts of information (and who isn't nowadays?), correlation and filtering is not the easiest path (and sometimes enough) to discern what is going on. The in-depth analysis of security data and logs is a time-consuming exercise, and security visualization (SecViz) extensively helps to focus on the relevant data and reduces the amount of work required to reach to the same conclusions. It is mandatory to add the tools and techniques associated to SecViz to your arsenal, as they are basically taking advantage of the capabilities we have as humans to visualize (and at the same time analyze) data. A clear example is the insider threat and related incidents, where tons of data sources are available. The best sentence (unfortunately it is not an image ;) that describes SecViz comes from the author: 'A picture is worth a thousand log entries.'" Read on for the rest of rsiles's review.This is a great book that joins two separate worlds, visualization and information security (infosec). The first chapter is an excellent introduction to the human perception system, its basic principles, and how we analyze, discern, and assimilate information. It is an eye opener for those new to the field. Chapter two is similar from an infosec perspective, and summarizes the main challenges and data sources, such as packet captures, traffic flows, and firewall, IDS/IPS, system, and application logs. The third chapter details different graph properties and chart types, including some open-source and online tools for chart and color selection. Although we (infosec pros) are familiarized with link graphs to represent relationships between botnet members or hosts, the book provides a whole set of charts for different purposes; one of the most useful types, and we are not very used too it in the security field, is treemaps. The chapter includes a really useful table to select the right graph based on the purpose of the analysis and the data available.

Then, the previous chapters are smoothly mixed together through a reference methodology that defines what is the problem to solve, and the process to manipulate the available data and generate a (or set of) graph(s) that allow gathering relevant conclusions and answers. The methodology is complemented with an introduction to the standard Unix-based text processing tools (grep, awk, Perl, etc). This methodology is later on applied, with a strong hands-on and how-to spirit, to an extensive set of common security use-cases, such as the perimeter threat, compliance, and the insider threat.

The perimeter chapter offers a deep insight into common attack scenarios, such as worms, DoS or anomaly detection, and operational tasks, like firewall log and ruleset analysis, IDS tuning, or vulnerability assessments. I could never forget how useful were SecViz techniques for anomaly detection on a huge DNS-related incident I was involved about 5 years ago. Thanks to the performance and statistical graphs we had available at that time, we were able to easily identify and solve a very complex and critical security incident.

When I saw this chapter included a wireless section I got really excited due to personal interest. However, I was disappointed as it was just a couple of pages. I think it could be extended to gather a whole set of useful information about complex wireless attacks and client and access points relationships, just by inspecting the different 802.11 management, control, and data frames, and even radio-frequency signals (from a spectrum analyzer). SecViz opens the door to a whole new wireless research area!

The compliance chapter offers a whole methodology to check and manage regulations, control frameworks, auditing, and risk monitoring and management from a visual perspective.

The same applies to the insider threat chapter, as it provides an impressive framework, not only visualization-based, to deal with malicious insiders. It is based on setting up scores for certain behaviors and activities (precursors), generating lists of suspicious candidates, and apply thresholds to accommodate exceptions. It also contains an extensive and directly applicable precursor list at the end to detect suspicious insider activities.

Finally, the book contains a whole chapter, full of references and comparison tables, of open-source and commercial visualization tools and libraries that allow the reader to select the appropriate tool for specific tasks and scenarios.

Although the book hands-on component is very significant, with lots of detailed examples of commands, scripts, and tool options to generate the different graphs, I would have liked to see a thorough usage of the how-to portions, as for some sections there are no specific details about how the graphs have been generated. The book layout makes it the perfect candidate to become a fully interactive technical book. I would suggest to add (for a 2nd edition ;)) practical sections to each chapter where the reader could reproduce all the steps discussed. The book CD is the perfect tool to provide the reader with all the (sanitized) data sets and logs used to generate the graphs, and even allow to include some challenges where the reader needs to analyze the data and answer some questions after generating the appropriate graphs.

To sum up, this book is a mandatory reference for anyone involved in the operational side of infosec, doing intrusion detection, incident handling, forensic analysis, etc, and it can be applied to both, historical analysis and real-time monitoring. Additionally, I found it useful too for auditing and pen-testing professionals, as it provides great tips to generate relevant and efficient graphs for the associated reports.

The accompanying DAVIX Live CD is an excellent resource to start applying the techniques covered throughout the book through open-source tools, SecViz is the Web portal to expand your knowledge on this topic, and AfterGlow is (one of) the most relevant SecViz open-source tools.

You can purchase Applied Security Visualization from Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

cancel ×


Sorry! There are no comments related to the filter you selected.

Sad news on talk radio today (-1, Offtopic)

citylivin (1250770) | more than 5 years ago | (#25646667)

I just heard some sad news on talk radio - author Michael Crichton was found dead in his Claremont home this morning. There weren't any more details. I'm sure everyone in the American community will miss him - even if you didn't enjoy his work, there's no denying his contributions to Society and true American patriotism. Truly an American icon.

Re:Sad news on talk radio today (-1, Offtopic)

Anonymous Coward | more than 5 years ago | (#25646831)

Huh. It's actually true [] . Don't you know that you're supposed to make these deaths up?

wow (0)

Anonymous Coward | more than 5 years ago | (#25646675)

a book that tells you how to make a flowchart.

the tools to populate the flowchart are kind of cool I suppose, but nothing i haven't figured out on my own anyways. oh well, if others find this book useful who am i to judge. but if you're in this industry i would hope you already know most if not all of what looks to be in this book.

frist psot (0)

Anonymous Coward | more than 5 years ago | (#25646677)

viewing first post?

What a worthless sales pitch... (-1, Flamebait)

tjstork (137384) | more than 5 years ago | (#25646751)

It is mandatory to add the tools and techniques associated to SecViz to your arsenal, a

"it is mandatory." who the hell are they? We should mandatory these people up against a wall just for making the claim. Hey, visualize this security... 300 rounds per minute turning them into hamburger meat.


Re:What a worthless sales pitch... (1)

djarum72 (122163) | more than 5 years ago | (#25646895)

I apologize for the shortness of digital temper, I just quit smoking.

May I suggest restarting? I did a double take on "mandatory" but if you flash into dreams of turning things into hamburger after that something is wrong.

Re:What a worthless sales pitch... (1)

tjstork (137384) | more than 5 years ago | (#25647299)

I did a double take on "mandatory" but if you flash into dreams of turning things into hamburger after that something is wrong.

At what point does the continual erosion of our rights, the continual battery which smashes the individual into a robot of compliance, how low must we sink, before we can revolt?

Re:What a worthless sales pitch... (0)

Anonymous Coward | more than 5 years ago | (#25649071)

No need to sink lower, I already find you revolting.

Re:What a worthless sales pitch... (1)

tjstork (137384) | more than 5 years ago | (#25652909)

No need to sink lower, I already find you revolting

Yes we can!

A picture is worth a thousand log entries? (3, Insightful)

Anonymous Crowhead (577505) | more than 5 years ago | (#25646769)

I think not. Unless it is a picture of those log entries.

Missing the point (4, Interesting)

Jay L (74152) | more than 5 years ago | (#25649161)

I haven't read this book yet, but visualization tools ARE a significant part of pattern detection that we've mostly overlooked.

Much as we try to create smarter algorithms that can do feature extraction, clustering, etc., the best pattern-detection engine we have is still the human brain. There are very few systems that can detect patterns when we have NO idea what we're looking for; the brain comes pre-installed. Have you ever tried to do logfile analysis on a few thousand machines? Playing "management by exception" doesn't work at scale; even the rare errors show up a few times a second.

I saw a presentation a few weeks ago by Deb Roy, who's heading the Speechome [] project at MIT. He's set up a bunch of cameras recording continuous audio and video in his house, in an attempt to map the language development of his son. That's a LOT of data to sift through - some 90,000 hours. Way too much for standard audio scrubbing/speedup, which would be the equivalent of our grep-a-log-file.

So they've had to develop some incredible visualization techniques that let you view higher-level patterns across multiple "rich data" streams - things like frequent patterns of motion (there's baby playing with his toy car with Daddy), eye-gaze focal points (there's baby looking at the car before saying "KA"), etc. that just pop out at you as you view the full data stream. It's truly jaw-dropping stuff, and it's applicable to far more than speech.

Anyone here ever defragged a hard drive (yeah, I know, ext3/HFS/etc.)? Would you get a better feel for the operation if you saw a list of sector numbers that were being relocated, or the usual 2D colored-block graph?

Anyone ever seen TreeMaps for finding large files on your drive?

Anyone ever known when a process is about to crash because the patterns of UI hesitation and hard-drive head-movement sounds change as the core files get written out?

That's all that info-vis is. It's presenting data in a way that lets you use intuition and subconscious cues to find what you're looking for - even if you don't KNOW what you're looking for.

Here's Deb Roy showing how you turn motion patterns from multiple video cameras into a two-dimensional, printable chart:

Visualization generation []

Re:Missing the point (2, Funny)

cffrost (885375) | more than 5 years ago | (#25657007)

[...] (there's baby looking at the car before saying "KA") [...]

Give the kid some credit; that's not "KA," it's "Car," with a perfectly enunciated Boston accent.

Re:Missing the point (1)

OolimPhon (1120895) | more than 5 years ago | (#25659359)

Perhaps not...maybe Daddy works for Ford...

Re:Missing the point (1)

mikael (484) | more than 5 years ago | (#25661947)

I've always wanted to visualize the connections between lists of open TCP/IP ports, processes, local files, and external file requests running on my Linux system. Maybe as a 3D bubble graph with each process being a sphere, and UNIX pipes being visualized as 'pipes'. The connection to the outside world would be one giant sphere containing everything, with all the TCP/IP ports being on the surface of this sphere. External file requests would appear as text labels on the outside of the corresponding TCP/IP port.

Re:A picture is worth a thousand log entries? (1)

Kugrian (886993) | more than 5 years ago | (#25649223)

How about a video [] ?

Jeff Goldblum says no (4, Funny)

MosesJones (55544) | more than 5 years ago | (#25646847)

Come on we've all seen independence day and Jeff looking at the 1s and 0s and then just spotting the pattern and the problem.

Jumped up security people with there fancy visualisation tools. 1s and 0s is where it is at, all you really need is a very very large green screen monitor and the force.

Personally I don't even use the monitor but instead lay hands on the ethernet cables and just squeeze out the bad packets.

Obama picks Rahm Emanuel as Chief of Staff (-1, Troll)

Anonymous Coward | more than 5 years ago | (#25647159)

Change we can believe in. I guess those Chicago thugs stick together...

Re:Jeff Goldblum says no (1, Funny)

Anonymous Coward | more than 5 years ago | (#25647979)

Interestingly, I detected a worm once based on one LED blinking differently from what I expected.

Re:Jeff Goldblum says no (0)

Anonymous Coward | more than 5 years ago | (#25673725)

If you're that good then you should enter the World Worm Charming Championships []

Re:Jeff Goldblum says no (1)

RockWolf (806901) | more than 5 years ago | (#25653011)

Personally I don't even use the monitor but instead lay hands on the ethernet cables and just squeeze out the bad packets.

Is that what you call it?

who's this written for... (1)

ruin20 (1242396) | more than 5 years ago | (#25646973)

really, I mean do you really want a person who's that ignorant of these things to be running your IT security? maybe this is for a college class or something?

he methodology is complemented with an introduction to the standard Unix-based text processing tools (grep, awk, Perl, etc). This methodology is later on applied, with a strong hands-on and how-to spirit, to an extensive set of common security use-cases, such as the perimeter threat, compliance, and the insider threat.

Do you really want someoen who doesn't know grep to be security admin? And is Perl correctly included in that list?

Re:who's this written for... (1)

ducomputergeek (595742) | more than 5 years ago | (#25647317)

PERL? Absolutely, now go away before I replace you with a small PERL script you insensitive clout!

Re:who's this written for... (1)

Tekfactory (937086) | more than 5 years ago | (#25648197)

Not everyone knows UNIX, or Linux.

The live CD it comes with is based off SLAX, but should with a tiny bit of handholding be useful to Windows admins that have never used UNIX/Linux before.

I work in IT Security, I meet folks all the time, successful people, that do not know UNIX/Linux. The only UNIX/Linux in my environment are security machines that I have built for IDS and running other tools.

I have a Masters graduate, and Master candidate both from good schools that work for me, and neither one of them knew any UNIX/Linux when they came on the job. One bought an EEEPC and built an Ubuntu box at home because he recognized the blind spot in his training. The other one really wants to learn and sat with me last week while I put a DISA STIG on a CentOS Snort box.

We hired them because they were smart, hard working, and eager to learn things, not because they are uber UNIX admins. We can teach them the tools, and the tools we use with few exceptions, happen to all be *NIX based.

If your shop is primarily UNIX/Linux based then I understand your criticism, but there are plenty of people in non-UNIX/Linux environments that can benefit from better security. Isn't that the whole point. A Grep/Vi tutorial couldn't have been too much a strain on the page count.

I actually started putting up some tutorials on my website when I decided give them people I met a knoppix CD, was closer to giving them fish, than teaching them to fish.

Now I give them VMWare Server and teach them to build images. RAM and Hard Drives are so cheap now that there really is no excuse for someone not to learn to use a UNIX/Linux.

modd parent up (1)

ruin20 (1242396) | more than 5 years ago | (#25655619)

Thanks for the concise and to the point post!

I don't work in IT, but I do work with very sensitive data that is high risk, so IT security is an important topic and I try to understand it as much as possible.

the reason I'm shocked is that I'd expect people to at least recognize that as a blindspot in their training before they ever graduate school. Our system is exclusively proprietary so I understand not everyone would need it, but it seems like it would make sense to know it since a good seventy percent of papers I see involve unix solutions. Granted, I might just be operating in a biased environment but I figured everyone would know at least the basics about linux.

and to ducomputergeek, I was infering that I don't see perl as being a unix based language... as far as I know it's platform independent and isn't strictly Unix based.

Re:modd parent up (1)

mvdwege (243851) | more than 5 years ago | (#25664359)

No, despite it being cross-platform, Perl has very visible roots in Unix. The fact that it picked its variable notation ($variable) from Bourne Shell, or that until 5.6 the only decent way to do multi-processing was to fork(), or the fact that one of the most common idioms is the while (<>) loop that iterates over STDIN, those are all Unixisms. And those are just the ones I can name off the top of my head. If you read the Camel Book, you'll see even more Unixisms, and the History of Perl section clearly mentions its Unix origins.


Re:who's this written for... (0)

Anonymous Coward | more than 5 years ago | (#25655359)

You would be surprised just how many security 'professionals' out there don't know much about computers at all. I've found them most often in larger corporate environments where people are more specialised and given much more opportunity to fail to grasp the importance of what they are dealing with.

These people are highly skilled in the art of delivering the appearance of security and being an extension of the HR department. However computers are a mystery to them beyond reading peoples' personal email and updating the virus scanner. I think many of them would be amazed if they could comprehend what grep, awk and perl could do for them.

Hire Neo (1)

PHAEDRU5 (213667) | more than 5 years ago | (#25647003)

What you do is tail all your logs to a console, sit Neo in front of said console, and have him detect changes in the patterns.

I once worked for a network security startup that had almost exactly this strategy.

They didn't do very well, I'm afraid.

Intrusion visualization (2, Funny)

chord.wav (599850) | more than 5 years ago | (#25647087)

I prefer the Gibson [] aproach. It shows cool little green/blue translucent cubes that turn red when something odd happens to them. You then have a whack-a-mole kind of interface to kill those bad processes.

Correlation (2, Funny)

athdemo (1153305) | more than 5 years ago | (#25647109)

Did I see the word correlation in a summary and then not the "correlationisnotcausation" tag? I'm proud of you /.! So proud!

Re:Correlation (1)

Spazztastic (814296) | more than 5 years ago | (#25647741)

You spoke too soon :(

Re:Correlation (1)

athdemo (1153305) | more than 5 years ago | (#25648329)

I ruined everything. Maybe without me this article stood a chance.

I'm sorry, everyone. :'(

Re:Correlation (1)

Kijori (897770) | more than 5 years ago | (#25650129)

It may not have been your fault - correlation is not causation.

The Spinning Cube of Potential Doom (1)

HoneyBeeSpace (724189) | more than 5 years ago | (#25647811)

A nice port-scanning secviz realtime animation was mentioned (I think here?) back in 2003. See this paper [] (images and animations are at the bottom) from DOE/LBNL/NERSC.

Looks good (1)

gweihir (88907) | more than 5 years ago | (#25647827)

I have ordered one. I have doen quite a bit of SecViz myself, but a survey and reference would be most welcome.

Q1 Labs (1, Interesting)

Anonymous Coward | more than 5 years ago | (#25647903)

I used to work at a firm called Q1 Labs [] . Their founders quite brilliantly mapped aggregate network statistical information into visual presentations. In effect, the computer did what it does best: aggregate information. However their product offloaded the detection of anomalies in that aggregate information onto the human, and in particular it presented data in a way that the human brain's visual centre could readily observe patterns and deviations from those patterns (but which patterns and deviations are quite difficult for computers to detect naturally/automatically).

It's kind of offtopic, but I thought that was cool enough to share. :)

Netcosm (0)

Anonymous Coward | more than 5 years ago | (#25648439)

I thought netcosm from NetQOS was pretty sweet... if only they'd push it out for end use.

See video here

Use the cycles for automated analysis (0)

Anonymous Coward | more than 5 years ago | (#25648687)

I have found graphs of log data to be somewhat useful, but my (admittedly limited) experience with visualization tools in security is that they give such a high level view, they are only applicable to certain kinds of activity--DoS is a good example. But at some point, the actual traffic must be analyzed to determine what the activity is. I was once peripherally involved in a "DDoS" that turned out to be Monday morning logins after weekend maintenance cleared DNS caches!

The more serious exploits are typically trying to be stealthy and usually don't show up in the data visualizations. The best analysis tool is access to log data in a format that allows it to be arranged in many ways, such as by timestamp, source or destination address, and also by sorting on multiple fields.

I prefer to spend the CPU cycles on the initial reporting code (IDS sensor, firewall, Email filters, etc.). And as a blatant plug, I have been working on an IDS that does this. It captures both halves of TCP sessions and correlates multiple detects from one or both halves before cutting a report. This allows for fine tuning of rules. The reports are stored in a DB which allows the UI to display many views of the data.

Later . . . Jim

The Realeyes IDS, check it out at:

Chernoff Faces (1)

Zerth (26112) | more than 5 years ago | (#25648825)

Did it earlier and better. See the article "The Use of Faces to Represent Points in K-Dimensional Space Graphically" or just skim the Wikipedia article []

NetGrok (1)

hide_drive (988787) | more than 5 years ago | (#25649051)

If you liked the Spinning Cube of Potential Doom and AfterGlow, you should check out NetGrok at [] . It allows real-time analysis of a live packet capture and is released under the BSD license on Google Code. It was presented at the VizSec 2008 conference on security visualization. Disclaimer: I'm one of the project developers.

Re:NetGrok (1)

zrlram (1400917) | more than 5 years ago | (#25649687)

Or have a look at [] to see many more examples of how to visualize security data.

Mandatory. (0)

Anonymous Coward | more than 5 years ago | (#25651201)

mandatory? I don't think that that word means what you think that it means.

Sensitive metrics first, visualization second (1)

starfishsystems (834319) | more than 5 years ago | (#25651417)

Visualization is a natural companion to security metrics. But I'd stress that unless you have sensitive metrics in the first place, visualization is not going to help.

For an excellent, intellectually rigorous treatment see Andrew Jaquith, "Security Metrics", ISBN 0321349989

baobab? (1)

supernova_hq (1014429) | more than 5 years ago | (#25657571)

Ever seen the program called baobab?

It makes a really nice multi-dimensional pie chart representing disk usage.

I'm sure something similar would be very useful for locating security problems such as addresses and subnets with unusual activity levels, etc.

you gonna be da wormface (0)

Anonymous Coward | more than 5 years ago | (#25664485)

"...extensively helps to focus on the relevant data and reduces the amount of work required to reach to the same conclusions."

Unfortunately, most of my bruthuhs in the security biz DO 'reach to conclusions'. And the companies who hire them pay dearly.

A company I work with is on their second outsourced computer and network contract with a very incompetent company, which seems to be par for the course nowadays. They know just enough buzzwords to sucker the technologically illiterate masses and reap the rewards.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>