VMware's Serengeti Brings Hadoop To Virtual, Cloud Environments 28
Nerval's Lobster writes "VMware's Serengeti is a new open-source project for deploying Apache Hadoop in virtual and cloud environments. Serengeti 0.5 is available as a free download under the Apache 2.0 license. It has been designed as distro-neutral, with support for Apache 1.0, CDH3, Hortonworks 1.0 and Greenplum HD 1.0. Of course, VMware isn't the only company seeking to leverage the increased interest in Hadoop. In June alone, midsize IT vendors such as Datameer, Karmasphere, and Hortonworks have all announced platforms that utilize the framework in some way. Research firm IDC recently predicted that worldwide revenues from Hadoop and MapReduce will hit $812.8 million in 2016, up from $77 million in 2011."
I have to admit... (Score:3, Funny)
But now, with only a little overhead, I can pretend to be running the same application in a distributed manner on a cluster, even though it's actually still running on the single server.
I have to admit this is pretty awesome.
Re: (Score:3)
Or you have a smaller cluster of big boxes running a big cluster of smaller boxes(vms)
Re: (Score:1)
Just imagine, you can build a beowulf cluster with just one computer, instead of dozens!
Modup. (Score:2)
Sadly, no mod points today.
Re: (Score:2)
So there are a lot of new names and jargon in the summary that I am not yet familiar with, but could you not just do this before using virtual machines?
I am sure I am missing the bigger picture here somehow...
Re:I have to admit... (Score:4, Interesting)
Yes, of course you can manually set up Hadoop in whatever environment, but it's a pain and generally speaking management is annoying. This new project appears to alleviate at least some of that, making it easy to remotely deploy and manage a Hadoop cluster. At least, that's what I got from the demo video - there's probably more to it.
Regarding Hadoop, I'm always surprised by its popularity given the relative fragility of HDFS (the NameNode is a single point of failure; other distributed filesystems have beaten this problem) and the dubious, beta-like quality of the tools built on top of it (Pig, etc.)
Re: (Score:2)
Regarding Hadoop, I'm always surprised by its popularity given the relative fragility of HDFS (the NameNode is a single point of failure; other distributed filesystems have beaten this problem) and the dubious, beta-like quality of the tools built on top of it (Pig, etc.)
So what's the alternative you recommend?
Re: (Score:2)
Re: (Score:2)
So you don't actually know of any that are competitive. Got it, thanks for playing.
Re: (Score:2)
Be careful what you ask, next they will be telling you to just run it on mysql since it works great for their wordpress site.
Shiny - High Revenue (Score:3)
From TFS:
Notice that the revenue is directed toward the few companies supporting and extending Hadoop. If you're working for one of those companies, congratulations. If you're working for one of the companies that is spending its money on this new shiny thing, you're probably in for a ride (one way or another). The technology is definitely good, I'll grant you that. But it is not the solution (or, not a very good solution) for many of the problems IT/data shops have. It really seems that a lot of people are jumping on the Hadoop bandwagon because "everyone else is getting it" and not because it will solve particular, concrete, existing problems. Or, it will solve exactly one relatively small, concrete, existing problem while erecting a complex infrastructure that must be supported for several years, making it more of a PITA than a solution.
Anyway, back to my original point: I think this revenue citation is more of an indication of a technology bubble and successful marketing than anything else. The price IT will pay for that bubble will probably far exceed the original cost.
Re:Shiny - High Revenue (Score:4, Interesting)
As someone who actually uses Hadoop, you're so far off the mark you've hit a bystander in the head. Dealing with large amount of data is a major PITA. If you don't understand that then you must never have worked with anything but trivial data sets. Hadoop fixes much of it, period. Without having to spend insane amount of money on databases, DBAs and still not being able to scale properly. It's not optimal but it works, it scales and it's flexible.
That's why companies are moving to it.
Re: (Score:1)
+1
Once you have collections of hundreds of millions of objects and need to work with a trillion properties,
everything that you know about working with data stops working.
Hadoop is not being adopted because it is fun and trendy, but because it addresses real needs now with resources that are attainable.
Re:Shiny - High Revenue (Score:4, Insightful)
Really, so what would you use to deal with 10Petabytes of data while going through 1Petabyte of it per day? I'm sure Google, Ebay, LinkedIn, Twitter, Yahoo and Facebook have no idea what they're doing. Hahaha. Get back to me when you're running a multi-billion dollar company.
That just indicates you lack the skills and knowledge to deal with big data sets.
Keep thinking that, I'm going to be over here enjoying my guaranteed job security with occasional breaks to beat hordes of recruiters away. No, seriously, keep thinking it, more idiots like you out there the better my job security will be.
See, I've actually got experience dealing with big data which I'm guessing is a lot more than you can say. I've talked to companies that think like you, it's downright hilarious to watch their jaws drop when you casually mention how trivial going through data is for us on Hadoop. A month long data project for them is a ten minute query for me that I let run for a few hours. I've personally played in both worlds, you can keep the alternative while I get actual work done.
Re: (Score:2)
As someone who also works with large amounts of data every day, I know exactly what I'm talking about. You may want to reread what I actually wrote.
Hadoop is a decent technology and is one approach to dealing with "Big Data" problems. There are other products out there, and for the most part they have all been around a lot longer than Hadoop. The problems all these products address have been around for quite some time, as most people know.
So what is the difference at this point in time? Did everyon
Re: (Score:2)
Hadoop is a decent technology and is one approach to dealing with "Big Data" problems. There are other products out there, and for the most part they have all been around a lot longer than Hadoop. The problems all these products address have been around for quite some time, as most people know.
So what are these alternatives? I like how people keep mentioning "alternatives" but never state them by name. Afraid of their actual flaws being ripped apart I guess. Always a vague "other options" statement.
Hadoop is inexpensive, flexible and well supported. It's cheaper overall than paying for some silly clustered RDBM licence which is optimized to solve a problem you don't actually care about. If you don't realize the specific set of problems Hadoop excels at solving then, frankly, you really don't unde
Re: (Score:2)
Also, a million rows of data? Most any decent web startup that does data is probably running at a million rows of data per day. Minimum. Maybe closer to a 100 million once they get around to collecting everything and got a few companies or users on board. Especially once you remove silly idiotically low restrictions on scaling and storage (unless you spend $$$$$). Got more data? Add more nodes, problem solved, get on with running the company. And they want to run complex analysis over the last year of data
Real I/O on Virtual machines (Score:2)
I've not tried the tech I will mention yet, but if your motherboard includes an IOMMU you can use physical networking and storage controllers on a virtual machine. it works as long the "passed-through" device is PCIe or PCI, be it onboard or on a card. so you can have racks of physical servers, with for instance one VM on each used as a node for your distributed file system. Virtualization still is useful for using the remainder of your physical server's capacity for other purposes. Or so I imagine it to be