Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Is It Time For NoSQL 2.0?

Unknown Lamer posted more than 2 years ago | from the hyperhack-the-hypergibsoncube dept.

Cloud 164

New submitter rescrv writes "Key-value stores (like Cassandra, Redis and DynamoDB) have been replacing traditional databases in many demanding web applications (e.g. Twitter, Google, Facebook, LinkedIn, and others). But for the most part, the differences between existing NoSQL systems come down to the choice of well-studied implementation techniques; in particular, they all provide a similar API that achieves high performance and scalability by limiting applications to simple operations like GET and PUT. HyperDex, a new key-value store developed at Cornell, stands out in the NoSQL spectrum with its unique design. HyperDex employs a unique multi-dimensional hash function to enable efficient search operations — that is, objects may be retrieved without using the key (PDF) under which they are stored. Other systems employ indexing techniques to enable search, or enumerate all objects in the system. In contrast, HyperDex's design enables applications to retrieve search results directly from servers in the system. The results are impressive. Preliminary benchmark results on the project website show that HyperDex provides significant performance improvements over Cassandra and MongoDB. With its unique design, and impressive performance, it seems fittng to ask: Is HyperDex the start of NoSQL 2.0?"

cancel ×

164 comments

Sorry! There are no comments related to the filter you selected.

No SQL, Know SQL (1, Funny)

PortHaven (242123) | more than 2 years ago | (#39127717)

Er...

Um...

Yeah...

Why not both? (3, Interesting)

Sarten-X (1102295) | more than 2 years ago | (#39127777)

Er...

Um...

Why not learn both, and use whichever's strengths suit the application the best?

Re:Why not both? (0)

Anonymous Coward | more than 2 years ago | (#39127861)

And don't forget that there are multi-dimentional array database engine (so called "NoSQL") out there that also support SQL, so you don't even have to choose between the two if you need or want both.

Re:Why not both? (2, Insightful)

LostCluster (625375) | more than 2 years ago | (#39127975)

Can somebody explain how this NoSQL stuff works? It's a database without SQL, so what replaces it? Is this just the difference between BASIC and C being expanded.

Re:Why not both? (5, Informative)

Sarten-X (1102295) | more than 2 years ago | (#39128313)

NoSQL is a terrible misnomer, in that the difference is far more than just "doesn't use SQL", and there are NoSQL systems that do actually support SQL. It's really just referring to data storage systems that aren't based on relations. That change in paradigm has its advantages (speed (in some cases), scalability, and flexibility) and disadvantages (speed (in some cases), lack of consistency, less restriction on bad programming). Of course, each NoSQL system tries to mitigate the disadvantages, and each RDBMS tries to prove itself better than all of NoSQL's advantages. It's a big fun party involving lots of mud-slinging.

Most NoSQL systems I've worked with are distributed hash tables, in a basic sense. Each value has a key, and that key determines where it's stored on a cluster. Values are not tied to any other values, so things like "foreign-key relations" are silly in a discussion of NoSQL. Rather, the algorithm to retrieve the data does all of the processing to connect data, using massive parallelization across a cluster to handle huge amounts of data at once.

With a traditional RDBMS, the application must fit its data to the schema completely before any data can be stored. This, of course, means that all data in the database can be assumed to be complete. You won't find references that don't exist, which makes queries straightforward.

With NoSQL, the database is treated as a more flexible bucket. Data is dumped in with a key, with little concern for fitting the design of the application's model. This, of course, means a bit more planning at design time, but the data can be arranged to better fit whatever it actually represents. Some details are present, and some aren't, but that's okay. The retrieval algorithm (typically a MapReduce program) should check for the existence of whatever data it needs, and handle errors accordingly. Those MapReduce programs are far more complicated than a simple SQL query, but the database's backend is conceptually simpler as an abstract key/value store. Key/value stores have been around for decades, and studied extensively. They can be made more fault-tolerant and scalable than RDBMS shards, but lack the support for large set-based comparisons.

The comparison to the BASIC-vs-C battles is appropriate. Both BASIC and C serve their purposes well (education and system programming, respectively), but neither should be used where the other is better suited. NoSQL and RDBMSs also both have their places.

Re:Why not both? (5, Funny)

w_dragon (1802458) | more than 2 years ago | (#39128483)

Thank you for a concise summary of the difference between the 2 systems. I must say though that such informative and level-headed comparisons have no place in a slashdot discussion ;)

Re:Why not both? (5, Funny)

lysdexia (897) | more than 2 years ago | (#39130165)

I concur! Shocking really. Gentlemen! Seize that miscreant's pants!

Re:Why not both? (2, Insightful)

Anonymous Coward | more than 2 years ago | (#39128687)

Perhaps I'm just in an old state-of-mind, but what good is data without relations? I don't mean that as a gripe about the system, just, how would one ever pull members of a group, or messages belonging to a user, etc? I guess I don't understand how that's more efficient.

Re:Why not both? (2)

Massacrifice (249974) | more than 2 years ago | (#39128917)

I guess he's saying that with NoSQL the relations are done at application level rather than database level. You still have the equivalent of schema and queries, but they are managed by the code, not the DB engine.

Re:Why not both? (2)

Grishnakh (216268) | more than 2 years ago | (#39129413)

Not all data needs to be related. Look at the history of databases: relational databases are fairly new actually, as they only came around in the 1970s. Before that were hierarchical databases, such as IBM's IMS, which was used on the Apollo program to track all the millions of parts used in that project. Wikipedia has a nice article about it here:
http://en.wikipedia.org/wiki/IBM_Information_Management_System [wikipedia.org]
Even though it was first used in the 60s, it's still in widespread use now, and every time you use an ATM machine, you're probably triggering a transaction on an IMS database.

Re:Why not both? (1)

Anonymous Coward | more than 2 years ago | (#39129569)

Not all data needs to be related.

Relation as in the mathematical term, not as in relationships. Relations exists at the conceptual level in the model, most RDBMSs use table-like structures but they are not required to. It is actually one of the strength of the relational model to strictly decouple conceptual, logical and physical levels. On paper....

Re:Why not both? (4, Informative)

Frnknstn (663642) | more than 2 years ago | (#39129647)

A hierarchy IS a relationship. In a hierarchical databases, child segments and parent segments were the main kind of relationship used.

All relational databases did was allow the relationships to be more freely defined.

Further to that, a key / value pair is also a relationship, in that the key symbolically represents the data. That's why it is correct to call them NoSQL databases: They forgo the complexity of a general query language. In doing so, they also lose the ability to inherently store anything except the most basic relationship: the key / value lookup.

Re:Why not both? (5, Funny)

justforgetme (1814588) | more than 2 years ago | (#39128701)

(speed (in some cases), scalability, and flexibility) and disadvantages (speed (in some cases), lack of consistency, less restriction on bad programming).

You have a background in Lisp right?

Re:Why not both? (4, Funny)

Sarten-X (1102295) | more than 2 years ago | (#39128931)

(not (know (I) (meaning (you))))

Re:Why not both? (2)

gman003 (1693318) | more than 2 years ago | (#39128937)

Seriously? No Lisp (or lisp-like) hacker (in the classical, "neat hack" sense (as opposed to the Faux-News "hackers stealing all your identities!" (or even the related but distinct "haxxor"))) worth his salt would be caught dead using only two nested parens. Real LISP Hackers (see previous nested comments) use at least ((2 * n)^(log n)) nested parentheses.

Re:Why not both? (4, Funny)

shentino (1139071) | more than 2 years ago | (#39129981)

Yeth

Re:Why not both? (1)

lysdexia (897) | more than 2 years ago | (#39130187)

Sthenthino winth the thread. Congratulathonths.

Reminds me of ISAM/VSAM based DB's in a way (0)

Anonymous Coward | more than 2 years ago | (#39129067)

Which, iirc, is SORT of how modern filesystems work (barring IBM's DB/2 driven one), but that's GREAT FOR READS (not so great for writes iirc). ISAM/VSAM also use hashes.

* Lastly/iirc: Some filesystems are even based on the ISAM/VSAM design too, but ones like NTFS use binary search pattern methods.

APK

P.S.=> The use of Hashes also makes it different than most RDBMS (those based on SQL usage), because they're based on binary trees iirc as well...

... apk

Re:Why not both? (0)

Anonymous Coward | more than 2 years ago | (#39129913)

So it's like a RDF triple store, but with pairs instead of triplets?

Re:Why not both? (2)

larry bagina (561269) | more than 2 years ago | (#39128527)

SQL = Structured Query Language. NoSQL = key/value stores. With SQL, you have a query, the database parses, plans, and executes it. With NoSQL, you have a key (string, number, etc). The database hashes it and finds the previously stored value.

It's dbm or perl tied hashes, updated for the cloud.

Re:Why not both? (2)

LostCluster (625375) | more than 2 years ago | (#39128713)

So that's good at finding the record if you already know the key, but there's no help in finding a record if you don't know the key, or getting a count of records with the same attribute attached... SQL for the win.

Re:Why not both? (4, Informative)

Anonymous Coward | more than 2 years ago | (#39128993)

So that's good at finding the record if you already know the key, but there's no help in finding a record if you don't know the key

Most NoSQL databases have indexes, and the indexes can be searched to find the key(s) you need. As an example, straight from the MongoDB examples:

db.things.find({colors : {$ne : "red"}})
{"_id": ObjectId("4dc9acea045bbf04348f9691"), "colors": ["blue","black"]}

In other words "find all the objects which do not contain 'red' in the field 'colors'". The ObjectId that is returned happens to be the key.

Re:Why not both? (2)

Wee (17189) | more than 2 years ago | (#39129125)

So that's good at finding the record if you already know the key, but there's no help in finding a record if you don't know the key, or getting a count of records with the same attribute attached... SQL for the win.

This isn't totally true. In MongoDB, for example, you don't even really have to think about the "primary key" for every document. Many times I don't know it or even care to. If you wants to look up customers in by name, you'd index the last_name and first_name fields and then do your query like so:

db.users.find({last_name : 'Cluster', first_name : 'Lost'})

Since there's a compound index on those two keys, the key/values being looked up are those. That will return everything in that document which matches that name. A count is done by replacing the find() method call with the count() method call.

You get a lot of flexibility. Let's say that for the above some users had an avatar. Then for those who have one, I just save it with their stuff. If not, no big deal. But I never have to go back and add a "column", I just save a new document that happens to have an 'avatar' key and there it is right alongside records, and it doesn't matter if some documents have an avatar or not. In fact, I could store binary data, then a shopping list and then a bunch of key/value pairs in the same collection (a collection is analogous to a table).

It does take a mental shift, but definitely has its uses. And like everything, right tool for the job. SQL for the win in some case, not in others.

-B

Re:Why not both? (0)

Anonymous Coward | more than 2 years ago | (#39128823)

SQL = Structured Query Language. NoSQL = key/value stores. With SQL, you have a query, the database parses, plans, and executes it. With NoSQL, you have a key (string, number, etc). The database hashes it and finds the previously stored value.

It's dbm or perl tied hashes, updated for the cloud.

this is not correct, yes some noSQL stores work like this, but some don't - look at mongoDB

noSQL = not only SQL are stores which do not follow some of the traditional RDBMS, they can (dont have to) follow ACID, they can (dont have to) use SQL etc...

simply say, these stores give up some RDBMS features in order to gain something else like speed, replication, flexible schema, ...

Re:Why not both? (1)

K. S. Kyosuke (729550) | more than 2 years ago | (#39129355)

Can somebody explain how this NoSQL stuff works? It's a database without SQL, so what replaces it? Is this just the difference between BASIC and C being expanded.

The basic distinguishing feature of RDBMS systems is that they are data-centric, so as to speak. The DB engine takes care of the metadata, access methods, query plan preparation ("SQL compilation") and optimization, transactions, backups, etc. Additionally, the RDBMS can (or is at least supposed to, if it's a proper RDBMS, see Codd's original work) allow you to separate the design of the physical layout of the data from the conceptual design of the data model ("this column is going to be accessed quite a lot, so pretend it's logically in the same table but actually store it physically on this fast new SSD drive, and oh, use this pair of functions to compress it and decompress it during storage") . None of the previous models (network model DBs, hierarchical model DBs etc.) supported this level of abstraction, they were just a thin API over the physical access methods, and I doubt that many of the modern "NoSQL" databases do today.

The net result is that a relational database system allows you to engineer a somewhat universal data model that, in addition to the original application, can be repurposed with minimized risk of project failure. ("We didn't anticipate the need for reverse pointers from items to orders, now we'll have to convert all our data" stuff etc.) So the abstraction costs you a bit in the performance area, compared to specialized storage, but 1) not much, because set-based data models are quite amenable to optimizatons, and 2) that specialized hyper-fast data storage you developed for your app is likely to be useless for the next app the management is going to ask you to deploy anyway, so why bother. If you have 5 TB of company data, you don't want to convert them for each new app you're going to run over it.

If your app is really specialized (Google search engine, a fast caching server for rendered web pages, a DNA database, a real-time trading system...) and performance-hungry, then by all means, go for it and write a specialized data store. It won't be a data-centric, but an application-centric one instead.

Re:Why not both? (5, Informative)

jd (1658) | more than 2 years ago | (#39129359)

NoSQL 1.0 is usually not much more than a hash-accessed flat-table database. GDBM, QDBM and BerkeleyDB are all hash-accessed flat-table databases. The refinements mentioned as being added to NoSQL databases (such as searchable indexes) are simply sequential indexes that associate some indexed parameters with the hash value.

NoSQL generally works by you pushing an item into the database and getting one or more hash values back. You want the item back, you give the database the hash values and you get the item. Object-oriented and object-based NoSQL both work by allowing objects to point to other objects. This gives you inheritance. (Basically you have a hash value that points to another record, where the structure of that other record is fixed rather than chosen at run-time via a join statement.)

Basically, database theory describes all the various forms of database you can have: flat-file, hierarchical, network, relational, object/relational, relational, semi-structured, associative, entity-attribute-value, transactional and star (aka data warehouse). A description of some of these can be found here [unixspace.com] .

This describes how the data is actually laid out, but does NOT necessarily describe how the data is accessed.

Database theory also describes the following underlying methods of accessing data: sequential, indexed, hash. Any combination of these is permitted, so you can have an index that points into sections of a database that are then searched sequentially for example. Or you can have indexes that point to other indexes that in turn point to a hash value. And so on.

SQL is just a meta-language that allows you to apply a restricted form of set theory on the underlying access methods. There were arguments at the time SQL appeared that it should allow all of set theory - and those arguments still go on, with some SQL alternatives using actual set theory notation as opposed to SQL notation.

NoSQL, in some cases, is just direct access to hash tables for directly accessing items. In other cases, it's a lightweight abstraction layer.

In the example advertised in the summary, an object is referenced through a set of indexes. If you have a partial set of indexes, you reference multiple objects but they will be related in some way. There is nothing X.0 about it, it's just a NoSQL database that uses a network database topology rather than a flat-file topology. It is nothing new.

I recognize that marketspeak is what sells things, that calling the systems by what they actually are would not be nearly as impressive to managers. Managers do not, as a rule, read Slashdot. Geeks and Nerds read Slashdot. Geeks and Nerds know Database Theory. (Well, if they don't, they damn well should -- either that, or they can use Google to look the terms up.) The two additions to database theory in the past 30 years have been the Object-Relational and Object-Oriented models.

Re:Why not both? (0)

Anonymous Coward | more than 2 years ago | (#39128477)

Because in the black-and-white duality that is contemporary geekspace "There can be only One" and the others are, you know, evil and stupid or at least very bad.
Kinda silly, yes.

Re:Why not both? (1)

jellomizer (103300) | more than 2 years ago | (#39128835)

You broke the first rule of Slashdot.
Though Shall not embrace two different methodologies.

You shall stick with one methodology until forced to change. When you do change you much embrace this with all your heart.

Re:No SQL, Know SQL (0)

Anonymous Coward | more than 2 years ago | (#39129537)

Web 2.0 running on the cloud serving up HTML5 over HTTP2.0 with a NoSQL2.0 backend.... Dec 2012 really is the end.

No mention of Riak? (1)

platypusfriend (1956218) | more than 2 years ago | (#39127771)

I use Riak in production and, while it is notably-slow, I appreciate its fault-tolerance. I wonder if HyperDex is just as resilient? Also, Riak 1.1 just came out, finally adding management and diagnostics. Secondary indexing may not be as fast as HyperDex, but, depending on the design, "crashproof-ness" can come out the winner.

Re:No mention of Riak? (1)

Anonymous Coward | more than 2 years ago | (#39127885)

Seems to be part of the focus of the research:

Per the abstract:

"Additionally, HyperDex achieves high performance
for simple get/put operations compared to current
state-of-the-art key-value stores, with stronger faulttolerance
and comparable scalability properties."

From the intro:

HyperDex ensures that all data in the
system is replicated to provide a desired level of faulttolerance.
The system employs a value-dependent replication
technique to maintain the desired level of fault
tolerance while providing strong consistency guarantees.
Value-dependent replication ensures a desired number of
copies, provides flexibility in assigning replica nodes to
different regions in hyperspace, and enables replica sets
to be selected from non-fate-sharing nodes. Unlike fixed
replication techniques, the replica sets are dynamically
constructed on the fly depending on the contents of the
object fields as well as the previous contents of the object,
enabling efficient search operations and strong consistency
semantics even in the presence of concurrent updates
and crash failures.

'
emph. mine. Not only do they claim better fault tolerance overall, but also -- to me at least -- this paragraph makes it sound like you can scale your fault tolerance based upon the region of hyperspace (ie type of data), which could be very useful if you're in a situation where data has different levels of importance.

Re:No mention of Riak? (1)

viperidaenz (2515578) | more than 2 years ago | (#39129071)

Hyperspace?

How It Works (1)

Anonymous Coward | more than 2 years ago | (#39127773)

From the paper:

Efficient lookup of fully-specified objects is critical to
object insertion and deletion performance, and requires
a deterministic object to node mapping. Much like in
ring-based key-value stores[3, 17, 38], HyperDex maps
both object coordinates and nodes to the same hyperspace.
Specifically, HyperDex tessellates the hyperspace
into a grid of N regions in space. Zones, which we previously
defined to be mutually exclusive regions belonging
to one or more nodes, are created by assigning nodes
to each of the tessellated regions to be responsible for
all objects which hash to a coordinate within the region.
The zone mapping is disseminated to all clients which
may operate directly on the mapping without any routing
between server nodes.

Berkeley DB? (4, Insightful)

gstoddart (321705) | more than 2 years ago | (#39127775)

This sounds like the old Berkeley DB/Sleepy Cat software.

Key/Value pairs instead of relational stuff. Worked with a product years ago that was built on Berkeley -- offered some pretty useful features that simply didn't map to object-relational stuff.

For some applications, you really do need something that works a little differently than an RDB ... however, there's still loads of things I can't imagine trying to do without one.

Choice is good in technology.

Re:Berkeley DB? (0)

Anonymous Coward | more than 2 years ago | (#39127991)

Yes the NoSQL options are quite similar to BerkeleyDB.

The main difference is that there is a network-enabled server running on-top of your DB, and instead of putting all your data in one file on one machine you can have many servers handling requests from thousands of clients.

But other than that - yes - it's almost exactly the same.

Re:Berkeley DB? (1)

oneiros27 (46144) | more than 2 years ago | (#39128331)

Oh ... so it's more like an LDAP server, instead.

Wasn't OpenLDAP [openldap.org] written on top of BDB?

(Although I don't know how well OpenLDAP handled replication -- the 'many servers' part ... I've only administered the Netscape/SunOne LDAP servers ... which are also key/value stores)

Re:Berkeley DB? (1)

moderatorrater (1095745) | more than 2 years ago | (#39128559)

I don't know how well OpenLDAP handled replication

That's half of the point of NoSQL, or at least Mongo. The point is to have very large data sets that can be accessed quickly and reliably (but not necessarily consistently). Mongo does that in two ways: by simplifying the data store significantly and by providing fast and easy replication and sharding. It's usually as simple as designating which group the server belongs to and then letting mongo take care of the rest.

Re:Berkeley DB? (0)

Anonymous Coward | more than 2 years ago | (#39128119)

however, there's still loads of things I can't imagine trying to do without one.

Do you have any experience doing functional programming? If not, I would pick up a book on SML or Haskell or Scala or -- the paradigm is fundamentally different from OO (programming's equivalent of RDB) and maps very well to key-value data stores. More efficient implementations also tend to be the more natural implementations as well.

Re:Berkeley DB? (1)

jd (1658) | more than 2 years ago | (#39129539)

There are permutations of the concepts in database theory that have not been tried, at least not that I know of, but NoSQL and this NoSQL 2.0 idea are not amongst them.

A short, and not comprehensive, list of underlying database structures [unixspace.com]
Another short list, describing a few others [wikibooks.org]
A short list of some file organization methods (ie: access methods) [toolbox.com]

A truly novel combination of existing structures and access methods would be worthy of a new name. A truly novel form of structure or file organization - assuming it is actually useful - is worthy of not just a new name but a new name writ large on the front page. A repackaged version of an existing combination might want a new marketing name, but it should be recognized that that is all it is - a name to sell by.

wake me in a few years (5, Funny)

joss (1346) | more than 2 years ago | (#39127793)

http://www.youtube.com/watch?v=URJeuxI7kHo [youtube.com]

is the best introduction to this subject I've seen. Until someone can explain the pros of hyperdex with a funny video featuring cute animals I'm sticking with technology that's been tested more thoroughly.

Re:wake me in a few years (4, Informative)

Sarten-X (1102295) | more than 2 years ago | (#39127831)

Decisions based on cute animals and straw-man arguments without any facts... You must be a manager!

Re:wake me in a few years (2)

c0d3g33k (102699) | more than 2 years ago | (#39128099)

It's funny because it's true.

Re:wake me in a few years (1)

royallthefourth (1564389) | more than 2 years ago | (#39128275)

Dealing with managers and coding up job security are certainly linked skills.

Re:wake me in a few years (1)

w_dragon (1802458) | more than 2 years ago | (#39128493)

I thought managers wanted pretty graphs and meaningless statistics? Maybe he's an executive?

Wow! That's some neat Progress! (5, Funny)

VortexCortex (1117377) | more than 2 years ago | (#39127869)

The hashing system is pretty neat. The idea that you could get at records without their specific key via search criterion is astounding.

In the future more advanced hashing systems will allow NoSQL databases to extract a set of records all containing a similar subset of data without keys at all!

Of course we'd need a name for the sections that are matching. Perhaps "Columns", yeah, then each result returned could be called a "Row", makes sense. I bet you could then create even more complex matching patterns for multiple "Columns" against each record in the data-set. If only there was a language to describe query we're sending to the servers... Oh! Server Query Language!

I can't wait to use SQL with NoSQL 3.0!

Re:Wow! That's some neat Progress! (4, Insightful)

Sarten-X (1102295) | more than 2 years ago | (#39128025)

And we'd still be able to have the cluster support, scalability, lax schema, and MapReduce algorithms NoSQL currently provides, right? Sometimes those aspects are vital to the application design, and key to the system's overall performance.

Re:Wow! That's some neat Progress! (1)

sourcerror (1718066) | more than 2 years ago | (#39128297)

Last time I checked Postgres had cluster support. I don't know what you mean by lax schema, or whether I really want it.

Re:Wow! That's some neat Progress! (1)

interval1066 (668936) | more than 2 years ago | (#39128753)

I think the big deal w/NoSQL is scalability. SQL cannot beat NoSQL in this arena. Not even close.

Re:Wow! That's some neat Progress! (1)

rubycodez (864176) | more than 2 years ago | (#39128805)

bwahahahaha, you're kidding right? because sql dbms like db2 or oracle dbms can scale bigger than google's search engine db? n

Re:Wow! That's some neat Progress! (1)

interval1066 (668936) | more than 2 years ago | (#39128913)

Uh, nope. Not kidding, Cassandra [nofluffjuststuff.com]

Re:Wow! That's some neat Progress! (1)

rubycodez (864176) | more than 2 years ago | (#39128991)

oops, my brain read your post as "problem with nosql is scalabilty", the exact opposite of your meaning. guess I should switch my neurons to no-sql.

Re:Wow! That's some neat Progress! (0)

Anonymous Coward | more than 2 years ago | (#39130279)

Don't make this claim. It's not true in all cases. I've proven it. NoSQL is good for mostly read scenarios. It's TERRIBLE for writes. This blind NoSQL is super fast crap is why I've seen some terrible applications designs where they tried to replicate mostly changing data to multiple NoSQL servers. It's just stupid.

There are cases for NoSQL in some types of apps, but sometimes a good memcached would do just as well with perminence is a real database.

Re:Wow! That's some neat Progress! (-1)

sexconker (1179573) | more than 2 years ago | (#39128397)

And we'd still be able to have the cluster support, scalability, lax schema, and MapReduce algorithms NoSQL currently provides, right? Sometimes those aspects are vital to the application design, and key to the system's overall performance.

MS SQL Server has great clustering support.
MS SQL Server has great scalability.
MS SQL Server runs just fine with a lax ("I don't know what I'm doing") schema.
Why would you want MapReduce? For any complex datasets or queries, standard databases perform better.

If you spend 20 minutes to RTFM and think before you start shoveling data at the server, SQL wins every time. There are plenty of implementations of it if you don't like Microsoft. MapReduce is for people who don't know what they're doing.

Re:Wow! That's some neat Progress! (4, Insightful)

Sarten-X (1102295) | more than 2 years ago | (#39128887)

[citation needed], and preferably one that actually covers NoSQL as it's intended for use.

Last time I checked thoroughly (2009), most RDBMSs (MS SQL Server included) could scale across an arbitrarily-large cluster, but for every doubling of the cluster's power, the costs would be around 300% to 400%. When you get to the point of needing billions of rows per table (and yes, there are applications out there that need that, even at relatively small startups), those outpacing costs become prohibitive.

The lax schema isn't about not knowing what you're doing, but about acknowledging that you won't know everything about the data you'll receive. Back when I did server programming, the mantra was "be strict in what you provide, and lax in what you accept". This is that principle applied to databases. Maybe the website you're crawling doesn't have a title, or its address is obviously dynamic. Maybe the medical record's patient has seven different insurance providers. Maybe the passport holder legally doesn't have a surname. When you design a schema for a strict database like an RDBMS, you make certain assumptions about the data you'll get. Those assumptions lead to performance increases if they're accurate, and failure if they're wrong.

MapReduce is the key to performance without assumptions, at lower cost. By moving processing to the data, and replicating the data to multiple nodes, network transfer is reduced greatly. The MapReduce programs are designed to operate on any amount of data they are presented with, so each node in the cluster contributes its available resources, and since the data is spread evenly, most "queries" will be partially processed by every node. Contrast that with RDBMS sharding, where certain servers handle certain shards, and the massive parallelism of the cluster isn't used. Some servers will sit idle while others do all of the work. Note that the parallelism applies generally, to all MapReduce algorithms. This means that you do not need to make as many assumptions about your queries ahead of time, like expecting to only look up a customer by name or phone number (and therefore indexing those).

NoSQL isn't just "not using SQL". It's a different storage paradigm, which comes with its own advantages and disadvantages.

Re:Wow! That's some neat Progress! (1, Insightful)

sexconker (1179573) | more than 2 years ago | (#39129419)

300 - 400%? Lol you're doing it wrong.
Billions of rows? So what? Easily handled by SQL.

Re:Wow! That's some neat Progress! (3, Funny)

binary paladin (684759) | more than 2 years ago | (#39129841)

Another brilliant post on Slashdot!

The quality of responses around here improve every day.

Re:Wow! That's some neat Progress! (1)

Anonymous Coward | more than 2 years ago | (#39130137)

300 - 400%? Lol you're doing it wrong.
Billions of rows? So what? Easily handled even by a single SQL instance

FTFY

Really, most people around here are completely clueless. Billions of rows is simply a matter of proper indexing and having enough storage bandwidth. Problems comes first with increasing concurrent connections. But even in that case it's completely solvable without any NoSQL thingamjic. Data fragmentation, some replication and data depending routing techniques have been around since ages.

I wonder what these sheeps think MS has behind the millions users using Hotmail/Live/Skydrive. Oh, yes, I guess most think they are secretly using some NoSQL engine on Linux. They must be the same that routinely post about MS using Linux web servers...

Re:Wow! That's some neat Progress! (2)

serviscope_minor (664417) | more than 2 years ago | (#39130159)

The lax schema isn't about not knowing what you're doing,

It seems to me that laxness and strictness of schemas is very much like static or dynamic typing. With static typing, certain classes of errors simply cannot happen, but you can deisgn yourself into a corner. With dynamic typing, smooshing things around is a bit easier, but you can get runtime errors if you don't design it properly. Personally, I prefer static typing.

Maybe the medical record's patient has seven different insurance providers. Maybe the passport holder legally doesn't have a surname. When you design a schema for a strict database like an RDBMS, you make certain assumptions about the data you'll get. Those assumptions lead to performance increases if they're accurate, and failure if they're wrong.

I'm really not convinced by your examples. Your processing is going to make those assumptions instead of making them in your schema, since some code somewhere has to know about the number of surnames. Depending on how the assumptions are made, you'll either output a record with the surname of (null), get a NullPointerException (or something similar) ot a whole host of different possible errors.

The lax schema in that case will simply move errors from guaranteed failure at data entry time to some random error at some random point in the future.

It seems that the time to use NoSQL is when your data isn't well represented by a relational model, or (if you are working on a truly immense scale) you have to have the performance.

Re:Wow! That's some neat Progress! (1)

Anonymous Coward | more than 2 years ago | (#39130339)

I'd take MS SQL server any day of the week for large data sets that change. Most Map Reduce implementations are great for searching with static data. It doesn't scale with writes.

You act like NULL columns don't exist. You don't have to write all fields to a database. At some point in the chain you have to interpret the data.. makes sense to store it in a logical way from the beginning rather than trying to guess later AFTER getting a bunch of crap from map reduce.

Re:Wow! That's some neat Progress! (1)

Marillion (33728) | more than 2 years ago | (#39129417)

Google? Yeah, they clearly didn't have a clue what they're doing when they invented MapReduce.

Facetiousness aside, the highly structured storage with tables and columns that a Codd style relational database provides is a better fit for most problems than most of the key/value pair (KVP) databases out there. There is too much R&D invested in that technology to just ignore. Using KVP puts more work on the programmer to organize the data. Storing serialized tuples in JSON or XML or whatever is en vogue at the time isn't structured storage.

Map Reduce is a technique to "pre-cook" summary data. Google uses it's massive farm of computation nodes to precompute facts about their data. You can not effectively create a single Relational DB instance with a thousand machines - as least not without breaking more than a few well established conventions.

TL;DR - Use the right tool for the right problem.

Re:Wow! That's some neat Progress! (0)

Anonymous Coward | more than 2 years ago | (#39128681)

Why, each time I read something similar to "cluster support, scalability, lax schema", it parses as "Can't be bothered to backup, can't be bothered to optimize, can't be bothered to analyze"?

Re:Wow! That's some neat Progress! (0)

Sarten-X (1102295) | more than 2 years ago | (#39128939)

It's because you can't be bothered to learn something different.

Re:Wow! That's some neat Progress! (0)

Anonymous Coward | more than 2 years ago | (#39128083)

Structured Query Language

Re:Wow! That's some neat Progress! (1)

interval1066 (668936) | more than 2 years ago | (#39128741)

I can't wait to use SQL with NoSQL 3.0!

Or to paraphrase Vortex: "I can't wait to build a hammer with a hypersonic toothbrush!"

Re:Wow! That's some neat Progress! (0)

Anonymous Coward | more than 2 years ago | (#39130429)

> Perhaps "Columns", yeah, then each result returned could be called a "Row", makes sense.

They're called "Attributes" and "Tuples" you insensitive clod!

Great for Perl aficionado... (5, Interesting)

Spectre (1685) | more than 2 years ago | (#39127881)

Many of the key-value pair DBs supply a Perl library that let you tie a Perl hash (%Variable) to the DB directly, giving you persistent hashes.

Makes database storage virtually a native feature of the language. Anybody who uses Perl is probably already a hash buff, so it is a win-win if you and your app already use Perl.

Disclaimer: I run a 10yo web "app" (Perl/CGI/Apache), so I'm a bit biased. But, the thing is rock-solid, so I'm not going to be too apologetic.

Re:Great for Perl aficionado... (1)

interval1066 (668936) | more than 2 years ago | (#39128803)

Many of the key-value pair DBs supply a Perl library...

Did somebody say MongoDB + PERL? [mongodb.org]

Re:Great for Perl aficionado... (1)

idontgno (624372) | more than 2 years ago | (#39128843)

Anybody who uses Perl is probably already a hash buff,

Not just hash, in truth; hallucinogens and sedatives are also widely popular with Perl-heads.

Branding (2)

slasho81 (455509) | more than 2 years ago | (#39127939)

So, at what point do we all admit that a NoSQL database is basically a glorified file-system over a network and start calling it a file-system again?

Re:Branding (2)

vlm (69642) | more than 2 years ago | (#39128011)

Some nosql "db" support 256 bit keys and everyone knows filesystems can only support 8.3 filenames, so at 8 characters of 7 bit ascii thats only something like 56 bits. If only microsoft had a filesystem supporting longer filenames... maybe next decade.

(note I'm intentionally avoiding the idea of a 256 directory deep filesystem, each directory containing a subdirectory 0 or 1, because that is just ... illness)

Re:Branding (1)

slasho81 (455509) | more than 2 years ago | (#39128143)

I didn't say NoSQL dbs aren't great file-systems, but they are file-systems nonetheless...

Re:Branding (4, Insightful)

donscarletti (569232) | more than 2 years ago | (#39128261)

It's great branding.

Previously, I was developing MMO backend software that uses MySQL for a data storage. The fit to the model was completely inappropriate, there was just no applications of the relational model, since we were just checking in and out large blobs of data, not actually performing read/update transactions. Storing records (persistent game entities) as files in a directory would have worked far better than forcing that stuff into a relational DB. But customers know that Databases are what professionals use, so we did it anyway. Clients can buy it, realise they need the flat files and turn them on after benchmarking, we get the sale, they get a good product in the end, win win, but a bit of wasted effort.

Now NoSQL is what professionals use, relational DBs can be used for what they are good at and NoSQL gives us marketing hype for doing certain things in the right way that could have been done using filesystems all along. I couldn't be happier. Furthermore we get this nice application level distributed data store with map-reduce stuff built in if we can be bothered using it.

Here's what most geeks don't get about marketing: it's not just about being smarter than the other guy, you've got to be smarter than him and make him give you his money. Money is good, it buys freedom and power and if branding makes sure that you have more of this freedom and power than the fool who falls for it, then the world will be a better place.

Re:Branding (2)

slasho81 (455509) | more than 2 years ago | (#39129031)

You're proving my point, which is that NoSQL is a marketing distinction, not a technical one. The problem with this is any technical discussion about the subject is biased and therefore flawed.

Use NoNoSQL like me! (0)

Anonymous Coward | more than 2 years ago | (#39128051)

Use NoNoSQL like me! Otherwise known as SQL.

Keys and values? (0)

Anonymous Coward | more than 2 years ago | (#39128089)

Isn't that what XML is for? XML files are also compatible across systems.

Re:Keys and values? (3, Informative)

Korin43 (881732) | more than 2 years ago | (#39128161)

Isn't that what XML is for? XML files are also compatible across systems.

XML is more useful for transferring data between systems. For storing data is kind of sucks, since there's no indexes (not the kind we need for fast lookups anyway) and it's extremely verbose.

Re:Keys and values? (1)

rubycodez (864176) | more than 2 years ago | (#39128847)

there are indexed xml retrieval systems with their own query languages to boot. Oracle XML DB is one (built on top its sql dbms)

Locally sensitive hashing (5, Informative)

Animats (122034) | more than 2 years ago | (#39128091)

This is a type of index, not a type of database. See locally sensitive hashing. [wikipedia.org] It's an efficient way to find keys which are "near" the search key in some sense.

Such a mechanism could be provided in a key/value store or an SQL database. It's even possible to do it on top of an SQL database. [compgeom.com] It's more powerful in a database that can do joins, because you can ask questions with several approximate keys.

This is an area of active research. Many machine-learning algorithms are scaled up by locally sensitive hashing, so they can work on big data.

Re:Locally sensitive hashing (0)

Anonymous Coward | more than 2 years ago | (#39129129)

Why would you link to a useless paper from a bunch of Microsoft cultists? FSU is not a good school, and their worship of broken crap from Microsoft just makes them even more useless. Are you one of them? Is that why you're spamming that garbage?

LOL (0)

Anonymous Coward | more than 2 years ago | (#39128127)

Who needs to be afraid of ACTA, SOPA and PIPA if his database is gone after a reboot?

This is retarded like everything I read on Slashdot. I come here for the retarded bullshit.

http://www.youtube.com/watch?v=URJeuxI7kHo [youtube.com]

The days are numbered. (0)

Anonymous Coward | more than 2 years ago | (#39128175)

You know what's better than NoSQL? - modern distributed relational SQL implementations like Clustrix http://www.clustrix.com/ and Volt http://en.wikipedia.org/wiki/VoltDB to mention two.

NoSQL 2.0 (1)

pankkake (877909) | more than 2 years ago | (#39128361)

Because NoSQL wasn't hipster enough.

Re:NoSQL 2.0 (4, Funny)

Sez Zero (586611) | more than 2 years ago | (#39128557)

Yeah, all the hipsters will be calling it "No2SQL", which is way more righteous.

Re:NoSQL 2.0 (1)

dreemernj (859414) | more than 2 years ago | (#39128885)

Why would you say that? That's actually going to be a thing now.

Re:NoSQL 2.0 (1)

flimflammer (956759) | more than 2 years ago | (#39130531)

Oh my god. Do you know what you've just done?!

i was all excited... (1)

Anonymous Coward | more than 2 years ago | (#39128435)

until I noticed that there seems to be a single point of failure in this system. from the site:

The HyperDex coordinator maintains the "hyperspace." This involves making sure that servers are up, detecting failed or slow nodes, taking them out of the system, and replacing them where necessary. The coordinator maintains a critical data structure, the hyperspace map, that establishes the mapping between the hyperspace and servers. Clients use this map to locate the servers they need to contact, while servers use it to perform object propagation and replication to achieve the application's desired goals.

How can people call a system "fault tolerant" and "distributed" when it might as well be running off a single box?

Re:i was all excited... (2)

rescrv (2575031) | more than 2 years ago | (#39128517)

Although the coordinator is logically centralized, we've got a version in the works that uses Paxos (a consensus algorithm) to distribute the coordinator as well. For more information check out http://openreplica.org/ [openreplica.org]

multidimensional hash, no point (0)

Anonymous Coward | more than 2 years ago | (#39128541)

Just append your keys together with a joining symbol. The progammer should be able to produce his own hash in
how ever many dimensions, better than anything predefined in the database code.

Re:multidimensional hash, no point (1)

rescrv (2575031) | more than 2 years ago | (#39128659)

When keys are concatenated with a joining symbol, objects can only be retrieved when one posesses all of the joining keys. Hyperspace hashing allows object retrieval when only a subset of the attributes are available.

Wait... What? (0)

Anonymous Coward | more than 2 years ago | (#39128667)

NoSQL2.0...
Read "NoSequel... the sequel"
So, if the Sequel to NoSQL does exist, then it is itself a paradox.

NoSQL (1)

M0j0_j0j0 (1250800) | more than 2 years ago | (#39128959)

Facebook, Google and friends wouldn't need such databases if they respected privacy, solve the privacy issues and a MyISAM will be enough to everyone. And for the marketing, just send pregnancy coupons to everyone, youll get em.

Why hashing? (1)

dtoader (1104557) | more than 2 years ago | (#39129021)

In using Oracle RDBMS, I see that for very large data set queries, using a hash join causes lots of disk activity (lots of paging, going to swap) Though hash functions are fast, this performance is from scanning though a hash table that's fully mapped in memory. Once your hash table gets too big for the available memory, you start using disk space (unindexed, sequential full reads) Isn't this a bottleneck in a distributed database that relies on hash functions? Wouldn't you want to have a distributed DB based on a distributed version of a B-Tree descendant (B+Tree, B*Tree,B**Tree) that would use memory AND storage and scale out more than just the available memory on all your nodes? Not only that, but you'd likely have better performance on range scans. Just thinking...

Re:Why hashing? (0)

Anonymous Coward | more than 2 years ago | (#39129121)

I don't think you're thinking how you should think about this.

Just a thought.

No. (2)

eternaldoctorwho (2563923) | more than 2 years ago | (#39129059)

Whenever a /. headline asks a question, the answer is always No.

FTFY (0)

Anonymous Coward | more than 2 years ago | (#39129073)

"Key-value stores have been supplementing traditional databases..."

Most successful NoSQL in history: Windows Registry (0)

Anonymous Coward | more than 2 years ago | (#39129485)

Well, the most successful NoSQL product in history is the Windows Registry, which is like one giant Java properties file. Not sure if that's an argument for or against NoSQL.

Only 2.0? (2)

alphred (1920232) | more than 2 years ago | (#39129769)

Hand it over to Mozilla. We could then have NoSQL 5.0 by the end of summer.

Re:Only 2.0? (3, Funny)

kangsterizer (1698322) | more than 2 years ago | (#39130147)

Or NoSQL 16 if that was Google. What a great joke.

Is it time for the NoIP Internet? (1)

WaffleMonster (969671) | more than 2 years ago | (#39130107)

Why does a new product operating in the very same space as other keyvalue stores warrant an increment of the buzzword version number?

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?