Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Raw Data Collection Software?

Anonymous Coward writes | 1 year,26 days


An anonymous reader writes "I have been looking for a tool (hopefully open source) that would allow some more intelligent management of data collection. Even better would include some data processing. Specifically I am looking to manage and chronicle raw data collected on hardware (in my case engines) and create a timeline of events (engine shut down, engine repair, parts swap, etc) allow for commenting and searching, etc Is there a good tool for this? I haven't been able to find one. Thanks for the help!

Currently, we just take each file and dump it into a share drive. It's not easily searchable and there is no good means for batch processing it. Data gets lost all the time and multiple groups collect data from the same engine. If we have a question about a particular behavior, we can't search these, we rely on someone remembering about a test and where they put it. PLEASE HELP!"

cancel ×

1 comment

Sorry! There are no comments related to the filter you selected.

Logstash (1)

rwa2 (4391) | 1 year,26 days | (#44689549)

Well, this is kinda abusing a logfile collection utility, but give Logstash [] + Kibana a try. It's very fast, and sounds perfect for the kinds of "events" you're looking at:

howto []

You could also try paying for something like Splunk, which is a bit slower, but gives you a much richer analysis & graphing toolset (though it's frustratingly difficult to do some things that you'd think should be simple, like plotting multiple searches on a timeline so you can do correlations.)

Now if you want to graph lots of numbers, you might want to look at something like munin [] , which is a dirt-simple way to write a small script that will pull a value and dump it into a lean RRD (round robin "database") that it can plot. I find this useful for tracking various things like temperatures, etc. , so you can see what happened over the past hour / day / month / year.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>