Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Enthusiasts Convene To Say No To SQL, Hash Out New DB Breed

timothy posted more than 4 years ago | from the sql-like-a-pig dept.

Data Storage 423

ericatcw writes "The inaugural NoSQL meet-up in San Francisco during last month's Yahoo! Apache Hadoop Summit had a whiff of revolution about it, like a latter-day techie version of the American Patriots planning the Boston Tea Party. Like the Patriots, who rebelled against Britain's heavy taxes, NoSQLers came to share how they had overthrown the tyranny of burdensome, expensive relational databases in favor of more efficient and cheaper ways of managing data, reports Computerworld."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered


Quit Whining (5, Funny)

KingPin27 (1290730) | more than 4 years ago | (#28565105)

Just use flat text files --- no need for expensive db's .... think of the freedom!

Re:Quit Whining (4, Insightful)

Anonymous Coward | more than 4 years ago | (#28565157)

The horrible lag I get when using address completion in Firefox 3 makes me wish more people thought that way!

Re:Quit Whining (1)

phantomfive (622387) | more than 4 years ago | (#28565553)

This is one of the main objectives of ReiserFS, to make such things easy, a project which unfortunately has run into some difficulty of late.

Re:Quit Whining (0)

Anonymous Coward | more than 4 years ago | (#28565591)

Like his wife?

Re:Quit Whining (1, Funny)

Anonymous Coward | more than 4 years ago | (#28565655)

She was screwing Mark Sanford!

Re:Quit Whining (2, Funny)

MichaelSmith (789609) | more than 4 years ago | (#28565727)

This is one of the main objectives of ReiserFS, to make such things easy, a project which unfortunately has run into some difficulty of late.

I wonder if I could sneak Hans an eeepc inside a birthday cake...

Re:Quit Whining (1)

rubycodez (864176) | more than 4 years ago | (#28565775)

I"ve lost data in two filesystems thanks to the Slasher's shoddy work.

Re:Quit Whining (4, Funny)

Paradise Pete (33184) | more than 4 years ago | (#28566195)

I"ve lost data in two filesystems thanks to the Slasher's shoddy work.

Have you looked near Redwood Regional Park? On the side of a hill?

A time and place for everything (3, Insightful)

Marillion (33728) | more than 4 years ago | (#28565143)

There is a time and place for SQL. There is a time and place to avoid SQL.
SQL is great for financial data. SQL is terrible for genetic data.

Re:A time and place for everything (0, Offtopic)

snl2587 (1177409) | more than 4 years ago | (#28565263)

It was the best of times, it was the worst of times,
it was the age of wisdom, it was the age of foolishness,
it was the epoch of belief, it was the epoch of incredulity,
it was the season of Light, it was the season of Darkness,
it was the spring of hope, it was the winter of despair,
we had everything before us, we had nothing before us,
we were all going direct to heaven, we were all going direct the other way

Re:A time and place for everything (3, Insightful)

Bromskloss (750445) | more than 4 years ago | (#28565367)

It would be interesting to hear why this is.

Re:A time and place for everything (0)

SendBot (29932) | more than 4 years ago | (#28565467)

I'm just throwing a half-educated guess out there, but genetic algorithms have so many outputs tied back into its inputs, all changing around quite frequently, such that an sql implementation would be painfully contorted.

But then, I don't quite see how neural network programs need mass replication the way db's do.

or would they.... ?

this is an interesting issue!

Re:A time and place for everything (3, Insightful)

Carewolf (581105) | more than 4 years ago | (#28565521)

Design an efficient table relating a tree structure. Then design queries to answer questions such as:
* Find the nodes in the subtree under B.
* Find all ancesters of G
* Find the nearest common ancestor of D and H

Trees is a wellknown problem of SQL, but the fact is that SQL can't handle most datastructures and complex relations, only very simple one dimensional ones.

Re:A time and place for everything (1)

Threni (635302) | more than 4 years ago | (#28566007)

One dimensional? So my really-fast snowflake design doesn't exist in your world? 2D tables no good? Sigh...and I was so happy with them.

Re:A time and place for everything (5, Interesting)

E IS mC(Square) (721736) | more than 4 years ago | (#28566043)

>> Trees is a wellknown problem of SQL, but the fact is that SQL can't handle most datastructures and complex relations, only very simple one dimensional ones.

Sorry, that's not true. Have you tried analytical functions? You would be amazed how complex scenarios can be handled easily with them. And they are part of ANSI SQL standards. And db providers (Oracle etc) have taken the concept and improved a lot on it.

I think the anti-sql 'movement' has more to do with new (internet era) languages and their developers than so called 'lack' of features. In my limited experience, I have observed people coming from C (and such) background have no problem with sql, while java developers (and this is probably true for most developers working on web-based applications) are the worst kind when it comes to understanding even basics of sql. All they want is their objects.

I strongly believe that a competent programmer designing/developing system which includes data and data-storage should at least know normalization, indexes, and what does it mean by 3NF. Programming language is one thing, database is another, and knowledge of both is required to build a decent system.

Re:A time and place for everything (2, Insightful)

MichaelSmith (789609) | more than 4 years ago | (#28565531)

It would be interesting to hear why this is.

My guess would be that because SQL is a Structured Query Language it is best used for handling structured data. If you have serial, unstructured data you have to invent your own format for it to use inside the database, and then the query language isn't helping you.

Re:A time and place for everything (3, Insightful)

Marillion (33728) | more than 4 years ago | (#28565879)

Right, I went into a little detail on another post. When I said, "genetic," I mean genes - DNA. There are four main Nucleic Acid types in DNA: Adenosine, Cytosine, Guanine, and Thymidine. Abbreviated ACGT. So you could store a gene sequence as ACGCCTGCAATC. But in other populations, Asian for example, the same gene is more commonly found as ACTCCTGCAATC. (The third nucleotide is different) Exact string matches won't find matches between different population groups. So they create wild-card letters that represent either G or T -> K. So ACKCCTGCAATC would match either the both of sequences commonly found in western and eastern populations. Data of this nature has no business being in a relational database. For that matter, it doesn't belong in these pseudo databases either.

Re:A time and place for everything (2, Interesting)

Marillion (33728) | more than 4 years ago | (#28565613)

Genetic sequences are long strings alphabetic characters. One of the most common representations is the FASTA [wikipedia.org] which deals with the most common type of nucleotide polymorphisms. You can't use exact string searching to find a match which makes BLOBS and CLOBS useless. That said, the meta-data of genetic data is reasonably structured and does load into relational databases fairly well.

Re:A time and place for everything (0)

Anonymous Coward | more than 4 years ago | (#28565615)

See, I don't think there is ever a good time or place for SQL. Anyone who says so has never had to use it. I like to compare it with JavaScript. It's a language that is difficult to refactor, maintain, and while it's a standard, the standard is so vague that it's useless. Like JavaScript, people are trying to build other languages on top of it to hide its shortcomings -- for javascript you have tools like GWT, and for SQL you have HQL, Linq, etc.

Not to say that there is anything wrong with relational databases, we just lack a good tool to interface with them.

This is what happens (4, Funny)

Anonymous Coward | more than 4 years ago | (#28565145)

When you get a lot of morbidly obese nerds with no life to program for you.

Meanwhile SQL users get laid.

Re:This is what happens (5, Funny)

Anonymous Coward | more than 4 years ago | (#28565323)

It's true. I do a lot of INNER JOINing. Often with multiple tables.

Re:This is what happens (1)

Anonymous Coward | more than 4 years ago | (#28565971)

Jocks get to SELECT * FROM sys.tables, so they always get the tables with the lovely columns and big BLOB's. The ones we can access have a lot of constraints, but also integrity.

Don't Like Traditional Relational Databases? (3, Funny)

ChoboMog (917656) | more than 4 years ago | (#28565149)

Go fork yourself!

Re:Don't Like Traditional Relational Databases? (2, Funny)

CorporateSuit (1319461) | more than 4 years ago | (#28565931)

It seems an idiot has modded you down because they don't understand very basic database expressions.

No need to get mad at Slashdot's mod point system, because, after all, if they outlaw giving mod points to stupids, then only stupid outlaws will have mod points... or something like that.

Tilting at windmills (5, Insightful)

Anonymous Coward | more than 4 years ago | (#28565151)

Seems to be a silly thing to be against. Relational databases and the stuctured query language may not be perfect, but I bet these people could die in their 90's and people will still be using relational dbs and sql.

If you want to tout open or cheap dbs and more lightweight types of storage/db servers, then they might have some points, but being against sql is just plain dumb.

Re:Tilting at windmills (5, Insightful)

Qzukk (229616) | more than 4 years ago | (#28565391)

SQL isn't the only way possible to query relational databases. It's nice and does a really good job for even mildly complex queries and I would not want to ditch it just yet, but seriously... who hasn't had a business need for multiple levels of aggregates (eg averages of sums across multiple groupings, say "average across all customers' total balances") As it is, you end up splitting the logic between the database and the application, or creating a view of the first level of aggregation, then querying against that and hoping that the performance doesn't suck total ass.

Re:Tilting at windmills (2, Insightful)

profplump (309017) | more than 4 years ago | (#28565579)

I agree, there are problems SQL doesn't solve well. But I think it's unlikely that other, better solutions to those problems will also be superior to SQL where it *does* perform well. As such, "no SQL" is probably not the right plan any more than "SQL only".

Re:Tilting at windmills (0)

Anonymous Coward | more than 4 years ago | (#28565587)

As it is, you end up splitting the logic between the database and the application

I always thought that was intentional. The DBMS is for data integrity and access. Business logic belongs in the applications.

RDBMS and application logic (4, Insightful)

gd2shoe (747932) | more than 4 years ago | (#28566053)

That is one view. It's nice and all, but incomplete. The issue is performance.

Any time you're dealing with a large quantity of data, it's always easiest to process or filter where it's located. Transmitting it, processing it, and transmitting back changes adds an unreasonable amount of overhead. Hence, SQL is a "Query" language. In other words, you have the RDBMS do reasonable data processing and filtering of records for you. Your application should only need to specify the operations performed, and should only process data if your computation is particularly unusual. This makes feasible computations that would otherwise be entirely unreasonable. (note that an application working on the same machine generally has the same issue as one working on a separate system. SQL servers present the application with a stream of data - pipe, socket, etc)

My opinion: SQL is horrendous. It's a pain to use, and many basic data transforms cannot be described in that language (at least without some huge, awful, convoluted command == maintenance nightmare).

Re:Tilting at windmills (0)

Anonymous Coward | more than 4 years ago | (#28565735)

(...) who hasn't had a business need for multiple levels of aggregates (eg averages of sums across multiple groupings, say "average across all customers' total balances") (...)

I think it's called partitioning. Recent versions of PostgreSQL have it and i think Oracle has it too.

Re:Tilting at windmills (3, Informative)

Strudelkugel (594414) | more than 4 years ago | (#28565861)

OLAP [wikipedia.org] was designed to answer that type of question. MDX [wikipedia.org] is the language used to perform multi-dimensional queries.

Re:Tilting at windmills (1)

quantum bit (225091) | more than 4 years ago | (#28565959)

who hasn't had a business need for multiple levels of aggregates (eg averages of sums across multiple groupings, say "average across all customers' total balances")

Funny you should mention that. Window functions in the SQL:2003 standard address that need, and there was an article on Slashdot earlier today about PostgreSQL 8.4 being released with support for them. Oracle has for a while now.

Re:Tilting at windmills (1)

CrashandDie (1114135) | more than 4 years ago | (#28565749)

Cheaper and more lightweight than Oracle?

Next thing we're going to hear people wanting a free DBMS...

Protest taxes, not databases (0, Troll)

Anonymous Coward | more than 4 years ago | (#28565167)

Too bad they can't protest the current regimes taxes with as much enthusiasm. At least it would be a protest against something that actually matters.

Flat Earth (3, Insightful)

Seumas (6865) | more than 4 years ago | (#28565199)

I've seen strong reactions from various camps with regard to concern over saying no to SQL. I'm not sure why people freak out over it. First, you have to strike out toward new things if you want to progress the world. Second, SQL hasn't caused people to stop using spreadsheets or Access databases. Third, there are groups that get together to dispute that the earth is round; insisting that it is flat. Or that gray aliens are visiting earth regularly and probing our anuses.

Bring on the next fascinating data technology. SQL will continue to have a major place for many years to come, no matter what happens.

Re:Flat Earth (3, Interesting)

syzler (748241) | more than 4 years ago | (#28565321)

I've seen strong reactions from various camps with regard to concern over saying no to SQL.. Third, there are groups that get together to dispute that the earth is round; insisting that it is flat.

Corporations represented in this group included the likes of Google, Last.fm, Amazon, and Facebook. Hardly the same caliber of people who claim the earth is still flat. I'm inclined to listen to engineers from these companies if they say that an SQL database does not scale well for vast amounts of data.

Re:Flat Earth (5, Insightful)

MightyMartian (840721) | more than 4 years ago | (#28565385)

And yet where the other corporations; the oil companies, the banks, large merchant conglomerates. In IT we seem to have this sort of myopic view that if it isn't an IT company of some kind, it doesn't exist. Google, as compared to the huge companies that use tools like Oracle, is a bit player. I know that's hard for all of us who have sucked at the teat of silicon valley for so long have a hard time dealing with, but a significant amount of data that has nothing to do with social networking and finding pr0n goes on and does use tools like SQL.

Re:Flat Earth (1)

hachete (473378) | more than 4 years ago | (#28565659)

I work for a financial company and if the rest use their bright shiny oracle databases like we do - and I don't think we're atypical - then, no, they have no idea how to use a database. Or build applications. At all. Not a clue.I can't begin to describe the inability, the sheer awesome crap-ness of what they do. The amount of work-arounds that the programmers implement to short-circuit the crap-ness. Really, you have no idea what you're talking about.

Re:Flat Earth (3, Informative)

Vellmont (569020) | more than 4 years ago | (#28565713)

And so you're saying this is all the fault of the relational database, and would all be solved by using some sort of object based database? That's the topic at hand here, not developers dealing with legacy systems patched together.

Re:Flat Earth (1)

kraut (2788) | more than 4 years ago | (#28565673)

Actually, the oil companies almost certainly have huge amounts of non-SQL data; I'm not sure whether seismology data comes in HDF, but it certainly doesn't come in SQL ;) Ditto banks have enormous amounts of non-SQL market data in specialised tick databases. That doesn't stop them from also having other important systems using SQL.

Vice versa, I'm pretty sure that while Google doesn't store its petabytes of web indexing info in a relational database (why on earth would you?), I'm equally sure that its billing, accounting and HR systems use relational databases; why on earth wouldn't they? Same thing applies to Amazon.

Horses for course may be an old saying, but it's still true.

Re:Flat Earth (1)

syzler (748241) | more than 4 years ago | (#28565677)

In IT we seem to have this sort of myopic view that if it isn't an IT company of some kind, it doesn't exist.

I understand that not all companies that maintain large data sets are technology companies. My only point was that when a group of companies known to manage large sets of data say that SQL does not always fit the bill, then I am inclined to listen rather than calling them nuts.

Re:Flat Earth (1, Interesting)

Anonymous Coward | more than 4 years ago | (#28565693)

You sure there's absolutely no difference between the nature of a bank and the nature of a massive search engine?

And how sure are you that a bank's IT staff are on the leading edge of innovative technologies? If anything, they lag behind because it's "safer" than risking the untested new thing.

Try a few of the Post-Relational databases, read up on the CAP Theorem, understand the -nature- of the problem you're talking about, and then come back.

Or I'll save you some time. RDBMS systems focus on Consistency, and trade Availability for it. Your bank's computer can be down for an hour... inconvenient, but acceptable. But they cannot, under ANY circumstances, be incorrect. Period. Google, on the other hand, can handle some slightly incorrect data... but being offline is totally unacceptable.

Amazon's CTO gave a great example. He talked about how a Shopping Cart must have Availability, and slight inconsistencies in the data as that data propagates a network are acceptable. In the end, the data is eventually consistent anyways, and you NEVER want your customer to not be able to add a cart item. The checkout, however, is financial, and heavily needs Consistency. Alternatively, after the order is done, the list of past transactions again can lose consistency a tiny bit (since it's read-mostly anyways) in exchange for always being up.

Hmm... more to the issue than you thought? XD

Re:Flat Earth (1)

fabs64 (657132) | more than 4 years ago | (#28565813)

I look to the oil companies to innovate in drilling technologies.
I look to financial companies to hopefully not innovate too much anywhere :-)
I look to IT companies to innovate in IT.

I dunno about you, but I've seen an incredible amount of money spent in the last 10 years or so attempting to change those massive relational databases into formats that can be reported on, as well as huge amounts of energy put into moving from one relational schema to another.

Pretending the big conglomerates present the best answer just because they're big is a recipe for non-movement.

Re:Flat Earth (1)

Vellmont (569020) | more than 4 years ago | (#28565645)

I'm inclined to listen to engineers from these companies if they say that an SQL database does not scale well for vast amounts of data.

This statement, taken as a whole is pure nonsense. "Databases" scale quite well for "vast" amounts of data. There's retailers that store millions of transactions a day on relational databases that would be out of business if they didn't.

If I had to guess, I'd say that relational databases might not be a great solution for a quickly evolving web company with possibly constantly changing data structures and new requirements being added. Doing all that glue code sucks, and patchwork solutions like Hibernate aren't much better (and IMO worse).

It shouldn't be surprising that a tool developed for one purpose isn't well suited to all purposes. Creating some kind of "movement" out of it is about as stupid as being against hammers in favor of screwdrivers. Down with hammers! Yeah screwdrivers!!

Re:Flat Earth (1)

leenks (906881) | more than 4 years ago | (#28565969)

There's retailers that store millions of transactions a day on relational databases that would be out of business if they didn't.

He said vast...

Re:Flat Earth (2, Interesting)

MightyMartian (840721) | more than 4 years ago | (#28565345)

The whole thing is just reactionary mumbo-jumbo. There are kinds of data that relational databases are fantastic for, and kinds of data they're not, and sometimes none of it is exactly perfect. SQL is actually a pretty damned good, single-purpose language. It's not hard to learn, and once you learn it, the differences between RDBMS implementations becomes a little like Javascript, just something you have to put up with, not that a lot of people actually have to worry all that much about writing fully-portable SQL queries.

Re:Flat Earth (1)

moderatorrater (1095745) | more than 4 years ago | (#28565585)

Agreed. SQL is a generalized solution that works well for a lot of different things and works extremely well for a subset of those thing. For other applications (like indexing the internet), more specialized solutions are going to kick its ass. It's the same way as any programming you do: the easier and more general the tool, the more you sacrifice for it in terms of speed, efficiency, scalability, whatever.

Re:Flat Earth (2, Informative)

Threni (635302) | more than 4 years ago | (#28566045)

> Second, SQL hasn't caused people to stop using spreadsheets or Access databases

If if weren't for SQL there wouldn't be any Access databases...

SQL.... (-1)

Darkness404 (1287218) | more than 4 years ago | (#28565227)

SQL has some great uses that its meant for. However, like all OSS tools it can get to where it is used enough for a certain purpose people will try to reuse it to varying success. SQL is great for financial data, however for some of the places it is in, SQL just doesn't do the job well.

Re:SQL.... (1)

rjstanford (69735) | more than 4 years ago | (#28565997)

SQL is an OSS tool?

Why'd I pay so much to Informix around 1990 then I wonder?

For that matter... SQL is a thing? I always thought it was a spec and a language :)

I've been using text files and Excel (0)

Anonymous Coward | more than 4 years ago | (#28565235)

I keep track of all my car bills and cat names with Notepad and Excel. I don't know why anyone would need anything more than that. If I need to sort my text file, I go to this thing called the command line and use the "SORT" command. If I need to find something in my text file, likewise, I use the command line and the "FIND" command


Anonymous Coward | more than 4 years ago | (#28565237)

THE GREAT AMERICAN BUBBLE MACHINE - By MATT TAIBBI - Rolling Stone (Current Issue, July (9-23) 2009) - and two other articles by Matt Taibbi.

"It's a gangster state, running on gangster economics, and even prices can't be trusted anymore; there are hidden taxes in every buck you pay"

From tech stocks to high gas prices, Goldman Sachs has engineered every major market manipulation since the Great Depression - and they're about to do it again

The first thing you need to know about Goldman Sachs is that it's everywhere. The world's most powerful investment bank is a great vampire squid wrapped around the face of humanity, relentlessly jamming its blood funnel into anything that smells like money. In fact, the history of the recent financial crisis, which doubles as a history of the rapid decline and fall of the suddenly swindled-dry American empire, reads like a Who's Who of Goldman Sachs graduates.

By now, most of us know the major players. As George Bush's last Treasury secretary, former Goldman CEO Henry Paulson was the architect of the bailout, a suspiciously self-serving plan to funnel trillions of Your Dollars to a handful of his old friends on Wall Street. Robert Rubin, Bill Clinton's former Treasury secretary, spent 26 years at Goldman before becoming chairman of Citigroup - which in turn got a $300 billion taxpayer bailout from Paulson. There's John Thain, the rear end in a top hat chief of Merrill Lynch who bought an $87,000 area rug for his office as his company was imploding; a former Goldman banker, Thain enjoyed a multibillion-dollar handout from Paulson, who used billions in taxpayer funds to help Bank of America rescue Thain's sorry company. And Robert Steel, the former Goldmanite head of Wachovia, scored himself and his fellow executives $225 million in golden parachute payments as his bank was self-destructing. There's Joshua Bolten, Bush's chief of staff during the bailout, and Mark Patterson, the current Treasury chief of staff, who was a Goldman lobbyist just a year ago, and Ed Liddy, the former Goldman director whom Paulson put in charge of bailed-out insurance giant AIG, which forked over $13 billion to Goldman after Liddy came on board. The heads of the Canadian and Italian national banks are Goldman alums, as is the head of the World Bank, the head of the New York Stock Exchange, the last two heads of the Federal Reserve Bank of New York - which, incidentally, is now in charge of overseeing Goldman - not to mention ...

But then, any attempt to construct a narrative around all the former Goldmanites in influential positions quickly becomes an absurd and pointless exercise, like trying to make a list of everything. What you need to know is the big picture: If America is circling the drain, Goldman Sachs has found a way to be that drain - an extremely unfortunate loophole in the system of Western democratic capitalism, which never foresaw that in a society governed passively by free markets and free elections, organized greed always defeats disorganized democracy.

The bank's unprecedented reach and power have enabled it to turn all of America into a giant pump-and-dump scam, manipulating whole economic sectors for years at a time, moving the dice game as this or that market collapses, and all the time gorging itself on the unseen costs that are breaking families everywhere - high gas prices, rising consumer-credit rates, half-eaten pension funds, mass layoffs, future taxes to pay off bailouts. All that money that you're losing, it's going somewhere, and in both a literal and a figurative sense, Goldman Sachs is where it's going: The bank is a huge, highly sophisticated engine for converting the useful, deployed wealth of society into the least useful, most wasteful and insoluble substance on Earth - pure profit for rich individuals.

They achieve this using the same playbook over and over again. The formula is relatively simple: Goldman positions itself in the middle of a speculative bubble, selling investments they know are crap. Then they hoover up vast sums from the middle and lower floors of society with the aid of a crippled and corrupt state that allows it to rewrite the rules in exchange for the relative pennies the bank throws at political patronage. Finally, when it all goes bust, leaving millions of ordinary citizens broke and starving, they begin the entire process over again, riding in to rescue us all by lending us back our own money at interest, selling themselves as men above greed, just a bunch of really smart guys keeping the wheels greased. They've been pulling this same stunt over and over since the 1920s - and now they're preparing to do it again, creating what may be the biggest and most audacious bubble yet. ...


Goldman wasn't always a too-big-to-fail Wall Street behemoth, the ruthless face of kill-or-be-killed capitalism on steroids - just almost always. The bank was actually founded in 1869 by a German immigrant named Marcus Goldman, who built it up with his son-in-law Samuel Sachs. They were pioneers in the use of commercial paper, which is just a fancy way of saying they made money lending out short-term IOUs to small-time vendors in downtown Manhattan.

You can probably guess the basic plotline of Goldman's first 100 years in business: plucky, immigrant-led investment bank beats the odds, pulls itself up by its bootstraps, makes shitloads of money. In that ancient history there's really only one episode that bears scrutiny now, in light of more recent events: Goldman's disastrous foray into the speculative mania of pre-crash Wall Street in the late 1920s.

This great Hindenburg of financial history has a few features that might sound familiar. Back then, the main financial tool used to bilk investors was called an "investment trust." Similar to modern mutual funds, the trusts took the cash of investors large and small and (theoretically, at least) invested it in a smorgasbord of Wall Street securities, though the securities and amounts were often kept hidden from the public. So a regular guy could invest $10 or $100 in a trust and feel like he was a big player. Much as in the 1990s, when new vehicles like day trading and e-trading attracted reams of new suckers from the sticks who wanted to feel like big shots, investment trusts roped a new generation of regular-guy investors into the speculation game.

Beginning a pattern that would repeat itself over and over again, Goldman got into the investment-trust game late, then jumped in with both feet and went hog-wild. The first effort was the Goldman Sachs Trading Corporation; the bank issued a million shares at $100 apiece, bought all those shares with its own money and then sold 90 percent of them to the hungry public at $104. The trading corporation then relentlessly bought shares in itself, bidding the price up further and further. Eventually it dumped part of its holdings and sponsored a new trust, the Shenandoah Corporation, issuing millions more in shares in that fund - which in turn sponsored yet another trust called the Blue Ridge Corporation. In this way, each investment trust served as a front for an endless investment pyramid: Goldman hiding behind Goldman hiding behind Goldman. Of the 7,250,000 initial shares of Blue Ridge, 6,250,000 were actually owned by Shenandoah - which, of course, was in large part owned by Goldman Trading.

The end result (ask yourself if this sounds familiar) was a daisy chain of borrowed money, one exquisitely vulnerable to a decline in performance anywhere along the line ....

Fast-Forward about 65 years. Goldman not only survived the crash that wiped out so many of the investors it duped, it went on to become the chief underwriter to the country's wealthiest and most powerful corporations. Thanks to Sidney Weinberg, who rose from the rank of janitor's assistant to head the firm, Goldman became the pioneer of the initial public offering, one of the principal and most lucrative means by which companies raise money. During the 1970s and 1980s, Goldman may not have been the planet-eating Death Star of political influence it is today, but it was a top-drawer firm that had a reputation for attracting the very smartest talent on the Street.

It also, oddly enough, had a reputation for relatively solid ethics and a patient approach to investment that shunned the fast buck; its executives were trained to adopt the firm's mantra, "long-term greedy." One former Goldman banker who left the firm in the early Nineties recalls seeing his superiors give up a very profitable deal on the grounds that it was a long-term loser. "We gave back money to 'grownup' corporate clients who had made bad deals with us," he says. "Everything we did was legal and fair - but 'long-term greedy' said we didn't want to make such a profit at the clients' collective expense that we spoiled the marketplace." ...

But then, something happened. It's hard to say what it was exactly; it might have been the fact that Goldman's co-chairman in the early Nineties, Robert Rubin, followed Bill Clinton to the White House, where he directed the National Economic Council and eventually became Treasury secretary. ...

Rubin was the prototypical Goldman banker. He was probably born in a $4,000 suit, he had a face that seemed permanently frozen just short of an apology for being so much smarter than you, and he exuded a Spock-like, emotion-neutral exterior; the only human feeling you could imagine him experiencing was a nightmare about being forced to fly coach. It became almost a national cliche that whatever Rubin thought was best for the economy - a phenomenon that reached its apex in 1999, when Rubin appeared on the cover of Time with his Treasury deputy, Larry Summers, and Fed chief Alan Greenspan under the headline THE COMMITTEE TO SAVE THE WORLD. And "what Rubin thought," mostly, was that the American economy, and in particular the financial markets, were over-regulated and needed to be set free. ...

The basic scam in the Internet Age is pretty easy even for the financially illiterate to grasp. Companies that weren't much more than pot-fueled ideas scrawled on napkins by up-too-late bong-smokers were taken public via IPOs, hyped in the media and sold to the public for megamillions. It was as if banks like Goldman were wrapping ribbons around watermelons, tossing them out 50-story windows and opening the phones for bids. In this game you were a winner only if you took your money out before the melon hit the pavement.

It sounds obvious now, but what the average investor didn't know at the time was that the banks had changed the rules of the game, making the deals look better than they actually were. They did this by setting up what was, in reality, a two-tiered investment system - one for the insiders who knew the real numbers, and another for the lay investor who was invited to chase soaring prices the banks themselves knew were irrational. While Goldman's later pattern would be to capitalize on changes in the regulatory environment, its key innovation in the Internet years was to abandon its own industry's standards of quality control.

"Since the Depression, there were strict underwriting guidelines that Wall Street adhered to when taking a company public," says one prominent hedge-fund manager. "The company had to be in business for a minimum of five years, and it had to show profitability for three consecutive years. But Wall Street took these guidelines and threw them in the trash." Goldman completed the snow job by pumping up the sham stocks: "Their analysts were out there saying Bullshit.com is worth $100 a share."

The problem was, nobody told investors that the rules had changed. "Everyone on the inside knew," the manager says. "Bob Rubin sure as hell knew what the underwriting standards were. They'd been intact since the 1930s." ...

Goldman has denied that it changed its underwriting standards during the Internet years, but its own statistics belie the claim. Just as it did with the investment trust in the 1920s, Goldman started slow and finished crazy in the Internet years. After it took a little-known company with weak financials called Yahoo! public in 1996, once the tech boom had already begun, Goldman quickly became the IPO king of the Internet era. Of the 24 companies it took public in 1997, a third were losing money at the time of the IPO. In 1999, at the height of the boom, it took 47 companies public, including stillborns like Webvan and eToys, investment offerings that were in many ways the modern equivalents of Blue Ridge and Shenandoah.. The following year, it underwrote 18 companies in the first four months, 14 of which were money losers at the time. As a leading underwriter of Internet stocks during the boom, Goldman provided profits far more volatile than those of its competitors: In 1999, the average Goldman IPO leapt 281 percent above its offering price, compared to the Wall Street average of 181 percent.

How did Goldman achieve such extraordinary results? One answer is that they used a practice called "laddering," which is just a fancy way of saying they manipulated the share price of new offerings. Here's how it works: Say you're Goldman Sachs, and Bullshit.com comes to you and asks you to take their company public. You agree on the usual terms: You'll price the stock, determine how many shares should be released and take the Bullshit.com CEO on a "road show" to schmooze investors, all in exchange for a substantial fee (typically six to seven percent of the amount raised). You then promise your best clients the right to buy big chunks of the IPO at the low offering price - let's say Bullshit.com's starting share price is $15 - in exchange for a promise that they will buy more shares later on the open market. That seemingly simple demand gives you inside knowledge of the IPO's future, knowledge that wasn't disclosed to the day-trader schmucks who only had the prospectus to go by: You know that certain of your clients who bought X amount of shares at $15 are also going to buy Y more shares at $20 or $25, virtually guaranteeing that the price is going to go to $25 and beyond. In this way, Goldman could artificially jack up the new company's price, which of course was to the bank's benefit - a six percent fee of a $500 million IPO is serious money.

Goldman was repeatedly sued by shareholders for engaging in laddering in a variety of Internet IPOs, including Webvan and NetZero. The deceptive practices also caught the attention of Nichol as Maier, the syndicate manager of Cramer & Co., the hedge fund run at the time by the now-famous chattering television rear end in a top hat Jim Cramer, himself a Goldman alum. ...

"Goldman, from what I witnessed, they were the worst perpetrator," Maier said. "They totally fueled the bubble. And it's specifically that kind of behavior that has caused the market crash. They built these stocks upon an illegal foundation - manipulated up - and ultimately, it really was the small person who ended up buying in." In 2005, Goldman agreed to pay $40 million for its laddering violations - a puny penalty relative to the enormous profits it made. (Goldman, which has denied wrongdoing in all of the cases it has settled, refused to respond to questions for this story.)

Another practice Goldman engaged in during the Internet boom was "spinning," better known as bribery. Here the investment bank would offer the executives of the newly public company shares at extra-low prices, in exchange for future underwriting business. Banks that engaged in spinning would then undervalue the initial offering price - ensuring that those "hot" opening price shares it had handed out to insiders would be more likely to rise quickly, supplying bigger first-day rewards for the chosen few. So instead of Bullshit.com opening at $20, the bank would approach the Bullshit.com CEO and offer him a million shares of his own company at $18 in exchange for future business - effectively robbing all of Bullshit's new shareholders by diverting cash that should have gone to the company's bottom line into the private bank account of the company's CEO. ...

Such practices conspired to turn the Internet bubble into one of the greatest financial disasters in world history: Some $5 trillion of wealth was wiped out on the NASDAQ alone. But the real problem wasn't the money that was lost by shareholders, it was the money gained by investment bankers, who received hefty bonuses for tampering with the market. Instead of teaching Wall Street a lesson that bubbles always deflate, the Internet years demonstrated to bankers that in the age of freely flowing capital and publicly owned financial companies, bubbles are incredibly easy to inflate, and individual bonuses are actually bigger when the mania and the irrationality are greater.


Nowhere was this truer than at Goldman. Between 1999 and 2002, the firm paid out $28.5 billion in compensation and benefits - an average of roughly $350,000 a year per employee. Those numbers are important because the key legacy of the Internet boom is that the economy is now driven in large part by the pursuit of the enormous salaries and bonuses that such bubbles make possible. Goldman's mantra of "long-term greedy" vanished into thin air as the game became about getting your check before the melon hit the pavement.

The market was no longer a rationally managed place to grow real, profitable businesses: It was a huge ocean of Someone Else's Money where bankers hauled in vast sums through whatever means necessary and tried to convert that money into bonuses and payouts as quickly as possible. If you laddered and spun 50 Internet IPOs that went bust within a year, so what? By the time the Securities and Exchange Commission got around to fining your firm $110 million, the yacht you bought with your IPO bonuses was already six years old. Besides, you were probably out of Goldman by then, running the U.S. Treasury or maybe the state of New Jersey. (One of the truly comic moments in the history of America's recent financial collapse came when Gov. Jon Corzine of New Jersey, who ran Goldman from 1994 to 1999 and left with $320 million in IPO-fattened stock, insisted in 2002 that "I've never even heard the term 'laddering' before.")

For a bank that paid out $7 billion a year in salaries, $110 million fines issued half a decade late were something far less than a deterrent - they were a joke. Once the Internet bubble burst, Goldman had no incentive to reassess its new, profit-driven strategy; it just searched around for another bubble to inflate. As it turns out, it had one ready, thanks in large part to Rubin.

Goldman's role in the sweeping disaster that was the housing bubble is not hard to trace. Here again, the basic trick was a decline in underwriting standards, although in this case the standards weren't in IPOs but in mortgages. ...

None of that would have been possible without investment bankers like Goldman, who created vehicles to package those lovely mortgages and sell them en masse to unsuspecting insurance companies and pension funds. This created a mass market for toxic debt that would never have existed before; in the old days, no bank would have wanted to keep some addict ex-con's mortgage on its books, knowing how likely it was to fail. You can't write these mortgages, in other words, unless you can sell them to someone who doesn't know what they are..

Goldman used two methods to hide the mess they were selling. First, they bundled hundreds of different mortgages into instruments called Collateralized Debt Obligations. Then they sold investors on the idea that, because a bunch of those mortgages would turn out to be OK, there was no reason to worry so much about the lovely ones: The CDO, as a whole, was sound. Thus, junk-rated mortgages were turned into AAA-rated investments. Second, to hedge its own bets, Goldman got companies like AIG to provide insurance - known as credit-default swaps - on the CDOs. The swaps were essentially a racetrack bet between AIG and Goldman: Goldman is betting the ex-cons will default, AIG is betting they won't.

There was only one problem with the deals: All of the wheeling and dealing represented exactly the kind of dangerous speculation that federal regulators are supposed to rein in. Derivatives like CDOs and credit swaps had already caused a series of serious financial calamities: Procter & Gamble and Gibson Greetings both lost fortunes, and Orange County, California, was forced to default in 1994. A report that year by the Government Accountability Office recommended that such financial instruments be tightly regulated - and in 1998, the head of the Commodity Futures Trading Commission, a woman named Brooksley Born, agreed. That May, she circulated a letter to business leaders and the Clinton administration suggesting that banks be required to provide greater disclosure in derivatives trades, and maintain reserves to cushion against losses. ...

Clinton's reigning economic foursome - "especially Rubin," according to Greenberger - called Born in for a meeting and pleaded their case. She refused to back down, however, and continued to push for more regulation of the derivatives. Then, in June 1998, Rubin went public to denounce her move, eventually recommending that Congress strip the CFTC of its regulatory authority. In 2000, on its last day in session, Congress passed the now-notorious Commodity Futures Modernization Act, which had been inserted into an 1l,000-page spending bill at the last minute, with almost no debate on the floor of the Senate. Banks were now free to trade default swaps with impunity.

But the story didn't end there. AIG, a major purveyor of default swaps, approached the New York State Insurance Department in 2000 and asked whether default swaps would be regulated as insurance. At the time, the office was run by one Neil Levin, a former Goldman vice president, who decided against regulating the swaps. Now freed to underwrite as many housing-based securities and buy as much credit-default protection as it wanted, Goldman went berserk with lending lust. By the peak of the housing boom in 2006, Goldman was underwriting $76.5 billion worth of mortgage-backed securities - a third of which were subprime - much of it to institutional investors like pensions and insurance companies. And in these massive issues of real estate were vast swamps of crap.

Take one $494 million issue that year, GSAMP Trust 2006-S3. Many of the mortgages belonged to second-mortgage borrowers, and the average equity they had in their homes was 0.71 percent. Moreover, 58 percent of the loans included little or no documentation - no names of the borrowers, no addresses of the homes, just zip codes. Yet both of the major ratings agencies, Moody's and Standard & Poor's, rated 93 percent of the issue as investment grade. Moody's projected that less than 10 percent of the loans would default. In reality, 18 percent of the mortgages were in default within 18 months.

Not that Goldman was personally at any risk. The bank might be taking all these hideous, completely irresponsible mortgages from beneath-gangster-status firms like Countrywide and selling them off to municipalities and pensioners - old people, for God's sake - pretending the whole time that it wasn't grade-D horseshit. But even as it was doing so, it was taking short positions in the same market, in essence betting against the same crap it was selling. Even worse, Goldman bragged about it in public. "The mortgage sector continues to be challenged," David Viniar, the bank's chief financial officer, boasted in 2007. "As a result, we took significant markdowns on our long inventory positions .... However, our risk bias in that market was to be short, and that net short position was profitable." In other words, the mortgages it was selling were for chumps. The real money was in betting against those same mortgages.

"That's how audacious these assholes are," says one hedge-fund manager. "At least with other banks, you could say that they were just dumb - they believed what they were selling, and it blew them up. Goldman knew what it was doing." I ask the manager how it could be that selling something to customers that you're actually betting against - particularly when you know more about the weaknesses of those products than the customer - doesn't amount to securities fraud.

"It's exactly securities fraud," he says. "It's the heart of securities fraud."

Eventually, lots of aggrieved investors agreed. In a virtual repeat of the Internet IPO craze, Goldman was hit with a wave of lawsuits after the collapse of the housing bubble, many of which accused the bank of withholding pertinent information about the quality of the mortgages it issued. .... But once again, Goldman got off virtually scot-free, staving off prosecution by agreeing to pay a paltry $60 million - about what the bank's CDO division made in a day and a half during the real estate boom.

The effects of the housing bubble are well known - it led more or less directly to the collapse of Bear Stearns, Lehman Brothers and AIG, whose toxic portfolio of credit swaps was in significant part composed of the insurance that banks like Goldman bought against their own housing portfolios. In fact, at least $13 billion of the taxpayer money given to AIG in the bailout ultimately went to Goldman, meaning that the bank made out on the housing bubble twice: It hosed the investors who bought their horseshit CDOs by betting against its own crappy product, then it turned around and hosed the taxpayer by making him payoff those same bets.

And once again, while the world was crashing down all around the bank, Goldman made sure it was doing just fine in the compensation department. In 2006, the firm's payroll jumped to $16.5 billion - an average of $622,000 per employee. As a Goldman spokesman explained, "We work very hard here."

But the best was yet to come. While the collapse of the housing bubble sent most of the financial world fleeing for the exits, or to jail, Goldman boldly doubled down - and almost single-handedly created yet another bubble, one the world still barely knows the firm had anything to do with.

By the beginning of 2008, the financial world was in turmoil. Wall Street had spent the past two and a half decades producing one scandal after another, which didn't leave much to sell that wasn't tainted. The terms junk bond, IPO, subprime mortgage and other once-hot financial fare were now firmly associated in the public's mind with scams; the terms credit swaps and CDOs were about to join them. The credit markets were in crisis, and the mantra that had sustained the fantasy economy throughout the Bush years - the notion that housing prices never go down - was now a fully exploded myth, leaving the Street clamoring for a new bullshit paradigm to sling.

Where to go? With the public reluctant to put money in anything that felt like a paper investment, the Street quietly moved the casino to the physical-commodities market - stuff you could touch: corn, coffee, cocoa, wheat and, above all, energy commodities, especially oil. In conjunction with a decline in the dollar, the credit crunch and the housing crash caused a "flight to commodities." Oil futures in particular skyrocketed, as the price of a single barrel went from around $60 in the middle of 2007 to a high of $147 in the summer of 2008.

That summer, as the presidential campaign heated up, the accepted explanation for why gasoline had hit $4.11 a gallon was that there was a problem with the world oil supply. In a classic example of how Republicans and Democrats respond to crises by engaging in fierce exchanges of moronic irrelevancies, John McCain insisted that ending the moratorium on offshore drilling would be "very helpful in the short term," while Barack Obama in typical liberal-arts yuppie style argued that federal investment in hybrid cars was the way out.


But it was all a lie. While the global supply of oil will eventually dry up, the short-term flow has actually been increasing. In the six months before prices spiked, according to the U.S. Energy Information Administration, the world oil supply rose from 85.24 million barrels a day to 85.72 million. Over the same period, world oil demand dropped from 86.82 million barrels a day to 86.07 million. Not only was the short-term supply of oil rising, the demand for it was falling - which, in classic economic terms, should have brought prices at the pump down.

So what caused the huge spike in oil prices? Take a wild guess. Obviously Goldman had help - there were other players in the physical-commodities market - but the root cause had almost everything to do with the behavior of a few powerful actors determined to turn the once-solid market into a speculative casino. Goldman did it by persuading pension funds and other large institutional investors to invest in oil futures - agreeing to buy oil at a certain price on a fixed date. The push transformed oil from a physical commodity, rigidly subject to supply and demand, into something to bet on, like a stock. Between 2003 and 2008, the amount of speculative money in commodities grew from $13 billion to $317 billion, an increase of 2,300 percent. By 2008, a barrel of oil was traded 27 times, on average, before it was actually delivered and consumed.

As is so often the case, there had been a Depression-era law in place designed specifically to prevent this sort of thing. ... In 1936, Congress recognized that there should never be more speculators in the market than real producers and consumers. If that happened, prices would be affected by something other than supply and demand, and price manipulations would ensue. A new law empowered the Commodity Futures Trading Commission - the very same body that would later try and fail to regulate credit swaps - to place limits on speculative trades in commodities. As a result of the CFTC's oversight, peace and harmony reigned in the commodities markets for more than 50 years.

All that changed in 1991 when, unbeknownst to almost everyone in the world, a Goldman-owned commodities-trading subsidiary called J. Aron wrote to the CFTC and made an unusual argument. Farmers with big stores of corn, Goldman argued, weren't the only ones who needed to hedge their risk against future price drops - Wall Street dealers who made big bets on oil prices also needed to hedge their risk, because, well, they stood to lose a lot too.

This was complete and utter crap - the 1936 law, remember, was specifically designed to maintain distinctions between people who were buying and selling real tangible stuff and people who were trading in paper alone. But the CFTC, amazingly, bought Goldman's argument. It issued the bank a free pass, called the "Bona Fide Hedging" exemption, allowing Goldman's subsidiary to call itself a physical hedger and escape virtually all limits placed on speculators. In the years that followed, the commission would quietly issue 14 similar exemptions to other companies.

Now Goldman and other banks were free to drive more investors into the commodities markets, enabling speculators to place increasingly big bets. That 1991 letter from Goldman more or less directly led to the oil bubble in 2008, when the number of speculators in the market - driven there by fear of the falling dollar and the housing crash - finally overwhelmed the real physical suppliers and consumers. By 2008, at least three quarters of the activity on the commodity exchanges was speculative, according to a congressional staffer who studied the numbers - and that's likely a conservative estimate. By the middle of last summer, despite rising supply and a drop in demand, we were paying $4 a gallon every time we pulled up to the pump.

What is even more amazing is that the letter to Goldman, along with most of the other trading exemptions, was handed out more or less in secret. "I was the head of the division of trading and markets, and Brooksley Born was the chair of the CFTC," says Greenberger, "and neither of us knew this letter was out there." In fact, the letters only came to light by accident. Last year, a staffer for the House Energy and Commerce Committee just happened to be at a briefing when officials from the CFTC made an offhand reference to the exemptions.

"1 had been invited to a briefing the commission was holding on energy," the staffer recounts. "And suddenly in the middle of it, they start saying, 'Yeah, we've been issuing these letters for years now.' I raised my hand and said, 'Really? You issued a letter? Can I see it?' And they were like, 'Duh, duh.' So we went back and forth, and finally they said, 'We have to clear it with Goldman Sachs.' I'm like, 'What do you mean, you have to clear it with Goldman Sachs?'" ... [I]n a classic example of how complete Goldman's capture of government is, the CFTC waited until it got clearance from the bank before it turned the letter over.

Armed with the semi-secret government exemption, Goldman had become the chief designer of a giant commodities betting parlor. Its Goldman Sachs Commodities Index - which tracks the prices of 24 major commodities but is overwhelmingly weighted toward oil - became the place where pension funds and insurance companies and other institutional investors could make massive long-term bets on commodity prices. Which was all well and good, except for a couple of things. One was that index speculators are mostly "long only" bettors, who seldom if ever take short positions - meaning they only bet on prices to rise. While this kind of behavior is good for a stock market, it's terrible for commodities, because it continually forces prices upward. "If index speculators took short positions as well as long ones, you'd see them pushing prices both up and down," says Michael Masters, a hedge-fund manager who has helped expose the role of investment banks in the manipulation of oil prices. "But they only push prices in one direction: up."

Complicating matters even further was the fact that Goldman itself was cheerleading with all its might for an increase in oil prices. In the beginning of 2008, Arjun Murti, a Goldman analyst, hailed as an "oracle of oil" by The New York Times, predicted a "super spike" in oil prices, forecasting a rise to $200 a barrel. At the time Goldman was heavily invested in oil through its commodities-trading subsidiary, J. Aron; it also owned a stake in a major oil refinery in Kansas, where it warehoused the crude it bought and sold. Even though the supply of oil was keeping pace with demand, Murti continually warned of disruptions to the world oil supply, going so far as to broadcast the fact that he owned two hybrid cars. High prices, the bank insisted, were somehow the fault of the piggish American consumer; in 2005, Goldman analysts insisted that we wouldn't know when oil prices would fall until we knew "when American consumers will stop buying gas-guzzling sport utility vehicles and instead seek fuel-efficient alternatives."

But it wasn't the consumption of real oil that was driving up prices - it was the trade in paper oil. By the summer of2008, in fact, commodities speculators had bought and stockpiled enough oil futures to fill 1.1 billion barrels of crude, which meant that speculators owned more future oil on paper than there was real, physical oil stored in all of the country's commercial storage tanks and the Strategic Petroleum Reserve combined. It was a repeat of both the Internet craze and the housing bubble, when Wall Street jacked up present-day profits by selling suckers shares of a fictional fantasy future of endlessly rising prices.

In what was by now a painfully familiar pattern, the oil-commodities melon hit the pavement hard in the summer of 2008, causing a massive loss of wealth; crude prices plunged from $147 to $33. Once again the big losers were ordinary people. The pensioners whose funds invested in this crap got massacred: CalPERS, the California Public Employees' Retirement System, had $1.1 billion in commodities when the crash came.. And the damage didn't just come from oil. Soaring food prices driven by the commodities bubble led to catastrophes across the planet, forcing an estimated 100 million people into hunger and sparking food riots throughout the Third World. ...

After the oil bubble collapsed last fall, there was no new bubble to keep things humming - this time, the money seems to be really gone, like worldwide-depression gone. So the financial safari has moved elsewhere, and the big game in the hunt has become the only remaining pool of dumb, unguarded capital left to feed upon: taxpayer money. Here, in the biggest bailout in history, is where Goldman Sachs really started to flex its muscle.

It began in September of last year, when then-Treasury secretary Paulson made a momentous series of decisions. Although he had already engineered a rescue of Bear Stearns a few months before and helped bail out quasi-private lenders Fannie Mae and Freddie Mac, Paulson elected to let Lehman Brothers - one of Goldman's last real competitors - collapse without intervention. ("Goldman's superhero status was left intact," says market analyst Eric Salzman, "and an investment-banking competitor, Lehman, goes away.") The very next day, Paulson greenlighted a massive, $85 billion bailout of AIG, which promptly turned around and repaid $13 billion it owed to Goldman. Thanks to the rescue effort, the bank ended up getting paid in full for its bad bets: By contrast, retired auto workers awaiting the Chrysler bailout will be lucky to receive 50 cents for every dollar they are owed.

Immediately after the AIG bailout, Paulson announced his federal bailout for the financial industry, a $700 billion plan called the Troubled Asset Relief Program, and put a heretofore unknown 35-year-old Goldman banker named Neel Kashkari in charge of administering the funds. In order to qualify for bailout monies, Goldman announced that it would convert from an investment bank to a bankholding company, a move that allows it access not only to $10 billion in TARP funds, but to a whole galaxy of less conspicuous, publicly backed funding - most notably, lending from the discount window of the Federal Reserve. By the end of March, the Fed will have lent or guaranteed at least $8.7 trillion under a series of new bailout programs - and thanks to an obscure law allowing the Fed to block most congressional audits, both the amounts and the recipients of the monies remain almost entirely secret.

Converting to a bank-holding company has other benefits as well: Goldman's primary supervisor is now the New York Fed, whose chairman at the time of its announcement was Stephen Friedman, a former co-chairman of Goldman Sachs. Friedman was technically in violation of Federal Reserve policy by remaining on the board of Goldman even as he was supposedly regulating the bank; in order to rectify the problem, he applied for, and got, a conflict-of-interest waiver from the government. Friedman was also supposed to divest himself of his Goldman stock after Goldman became a bank-holding company, but thanks to the waiver, he was allowed to go out and buy 52,000 additional shares in his old bank, leaving him $3 million richer. Friedman stepped down in May, but the man now in charge of supervising Goldman - New York Fed president William Dudley - is yet another former Goldmanite.

The collective message of all this - the AIG bailout, the swift approval for its bank-holding conversion, the TARP funds - is that when it comes to Goldman Sachs, there isn't a free market at all. The government might let other players on the market die, but it simply will not allow Goldman to fail under any circumstances. Its edge in the market has suddenly become an open declaration of supreme privilege. "In the past it was an implicit advantage," says Simon Johnson, an economics professor at MIT and former official at the International Monetary Fund, who compares the bailout to the crony capitalism he has seen in Third World countries. "Now it's more of an explicit advantage." ...

And here's the real punch line. After playing an intimate role in four historic bubble catastrophes, after helping $5 trillion in wealth disappear from the NASDAQ, after pawning off thousands of toxic mortgages on pensioners and cities, after helping to drive the price of gas up to $4 a gallon and to push 100 million people around the world into hunger, after securing tens of billions of taxpayer dollars through a series of bailouts overseen by its former CEO, what did Goldman Sachs give back to the people of the United States in 2008?

Fourteen million dollars.

That is what the firm paid in taxes in 2008, an effective tax rate of exactly one, read it, one percent. The bank paid out $10 billion in compensation and benefits that same year and made a profit of more than $2 billion - yet it paid the Treasury less than a third of what it forked over to CEO Lloyd Blankfein, who made $42.9 million last year.

How is this possible? According to Goldman's annual report, the low taxes are due in large part to changes in the bank's "geographic earnings mix." In other words, the bank moved its money around so that most of its earnings took place in foreign countries with low tax rates. Thanks to our completely hosed corporate tax system, companies like Goldman can ship their revenues offshore and defer taxes on those revenues indefinitely, even while they claim deductions upfront on that same untaxed income. This is why any corporation with an at least occasionally sober accountant can usually find a way to zero out its taxes. A GAO report, in fact, found that between 1998 and 2005, roughly two-thirds of all corporations operating in the U.S. paid no taxes at all.

This should be a pitchfork-level outrage - but somehow, when Goldman released its post-bailout tax profile, hardly anyone said a word. One of the few to remark on the obscenity was Rep. Lloyd Doggett, a Democrat from Texas who serves on the House Ways and Means Committee. "With the right hand out begging for bailout money," he said, "the left is hiding it offshore."

Fast-Forward to today. It's early June in Washington, D.C. Barack Obama, a popular young politician whose leading private campaign donor was an investment bank called Goldman Sachs - its employees paid some $981,000 to his campaign - sits in the White House. Having seamlessly navigated the political minefield of the bailout era, Goldman is once again back to its old business, scouting out loopholes in a new government-created market with the aid of a new set of alumni occupying key government jobs.


Gone are Hank Paulson and Neel Kashkari; in their place are Treasury chief of staff Mark Patterson and CFTC chief Gary Gensler, both former Goldmanites. (Gensler was the firm's co-head of finance) And instead of credit derivatives or oil futures or mortgage-backed CDOs, the new game in town, the next bubble, is in carbon credits - a booming trillion-dollar market that barely even exists yet, but will if the Democratic Party that it gave $4,452,585 to in the last election manages to push into existence a groundbreaking new commodities bubble, disguised as an "environmental plan," called cap-and-trade.

The new carbon-credit market is a virtual repeat of the commodities-market casino that's been kind to Goldman, except it has one delicious new wrinkle: If the plan goes forward as expected, the rise in prices will be government-mandated. Goldman won't even have to rig the game. It will be rigged in advance.

Here's how it works: If the bill passes; there will be limits for coal plants, utilities, natural-gas distributors and numerous other industries on the amount of carbon emissions (a.k.a. greenhouse gases) they can produce per year. If the companies go over their allotment, they will be able to buy "allocations" or credits from other companies that have managed to produce fewer emissions. President Obama conservatively estimates that about $646 billions worth of carbon credits will be auctioned in the first seven years; one of his top economic aides speculates that the real number might be twice or even three times that amount.

The feature of this plan that has special appeal to speculators is that the "cap" on carbon will be continually lowered by the government, which means that carbon credits will become more and more scarce with each passing year. Which means that this is a brand-new commodities market where the main commodity to be traded is guaranteed to rise in price over time. The volume of this new market will be upwards of a trillion dollars annually; for comparison's sake, the annual combined revenues of an electricity suppliers in the U.S. total $320 billion.

Goldman wants this bill. The plan is (1) to get in on the ground floor of paradigm-shifting legislation, (2) make sure that they're the profit-making slice of that paradigm and (3) make sure the slice is a big slice. Goldman started pushing hard for cap-and-trade long ago, but things really ramped up last year when the firm spent $3.5 million to lobby climate issues. (One of their lobbyists at the time was none other than Patterson, now Treasury chief of staff.) Back in 2005, when Hank Paulson was chief of Goldman, he personally helped author the bank's environmental policy, a document that contains some surprising elements for a firm that in all other areas has been consistently opposed to any sort of government regulation. Paulson's report argued that "voluntary action alone cannot solve the climate-change problem." A few years later, the bank's carbon chief, Ken Newcombe, insisted that cap-and-trade alone won't be enough to fix the climate problem and called for further public investments in research and development. Which is convenient, considering that 'Goldman made early investments in wind power (it bought a subsidiary called Horizon Wind Energy), renewable diesel (it is an investor in a firm called Changing World Technologies) and solar power (it partnered with BP Solar), exactly the kind of deals that will prosper if the government forces energy producers to use cleaner energy. As Paulson said at the time, "We're not making those investments to lose money."

The bank owns a 10 percent stake in the Chicago Climate Exchange, where the carbon credits will be traded. Moreover, Goldman owns a minority stake in Blue Source LLC, a Utah-based firm that sells carbon credits of the type that will be in great demand if the bill passes. Nobel Prize winner Al Gore, who is intimately involved with the planning of cap-and-trade, started up a company called Generation Investment Management with three former bigwigs from Goldman Sachs Asset Management, David Blood, Mark Ferguson and Peter Harris. Their business? Investing in carbon offsets. There's also a $500 million Green Growth Fund set up by a Goldmanite to invest in green-tech ... the list goes on and on. Goldman is ahead of the headlines again, just waiting for someone to make it rain in the right spot. Will this market be bigger than the energy-futures market?

"Oh, it'll dwarf it," says a former staffer on the House energy committee. ....

"If it's going to be a tax, I would prefer that Washington set the tax and collect it," says Michael Masters, the hedge fund director who spoke out against oil-futures speculation. "But we're saying that Wall Street can set the tax, and Wall Street can collect the tax. That's the last thing in the world I want. It's just asinine."

Cap-and-trade is going to happen. Or, if it doesn't, something like it will. The moral is the same as for all the other bubbles that Goldman helped create, from 1929 to 2009. In almost every case, the very same bank that behaved recklessly for years, weighing down the system with toxic loans and predatory debt, and accomplishing nothing but massive bonuses for a few bosses, has been rewarded with mountains of virtually free money and government guarantees - while the actual victims in this mess, ordinary taxpayers, are the ones paying for it.

It's not always easy to accept the reality of what we now routinely allow these people to get away with; there's a kind of collective denial that kicks in when a country goes through what America has gone through lately, when a people lose as much prestige and status as we have in the past few years. You can't really register the fact that you're no longer a citizen of a thriving first-world democracy, that you're no longer above getting robbed in broad daylight, because like an amputee, you can still sort of feel things that are no longer there.

But this is it. This is the world we live in now. And in this world, some of us have to play by the rules, while others get a note from the principal excusing them from homework till the end of time, plus 10 billion free dollars in a paper bag to buy lunch. It's a gangster state, running on gangster economics, and even prices can't be trusted anymore; there are hidden taxes in every buck you pay. And maybe we can't stop it, but we should at least know where it's all going.

The Big Takeover
The global economic crisis isn't about money - it's about power. How Wall Street insiders are using the bailout to stage a revolution

It's over â" we're officially, royally fucked. No empire can survive being rendered a permanent laughingstock, which is what happened as of a few weeks ago, when the buffoons who have been running things in this country finally went one step too far. It happened when Treasury Secretary Timothy Geithner was forced to admit that he was once again going to have to stuff billions of taxpayer dollars into a dying insurance giant called AIG, itself a profound symbol of our national decline â" a corporation that got rich insuring the concrete and steel of American industry in the country's heyday, only to destroy itself chasing phantom fortunes at the Wall Street card tables, like a dissolute nobleman gambling away the family estate in the waning days of the British Empire.

The latest bailout came as AIG admitted to having just posted the largest quarterly loss in American corporate history â" some $61.7 billion. In the final three months of last year, the company lost more than $27 million every hour. That's $465,000 a minute, a yearly income for a median American household every six seconds, roughly $7,750 a second. And all this happened at the end of eight straight years that America devoted to frantically chasing the shadow of a terrorist threat to no avail, eight years spent stopping every citizen at every airport to search every purse, bag, crotch and briefcase for juice boxes and explosive tubes of toothpaste. Yet in the end, our government had no mechanism for searching the balance sheets of companies that held life-or-death power over our society and was unable to spot holes in the national economy the size of Libya (whose entire GDP last year was smaller than AIG's 2008 losses).

So it's time to admit it: We're fools, protagonists in a kind of gruesome comedy about the marriage of greed and stupidity. And the worst part about it is that we're still in denial â" we still think this is some kind of unfortunate accident, not something that was created by the group of psychopaths on Wall Street whom we allowed to gang-rape the American Dream. When Geithner announced the new $30 billion bailout, the party line was that poor AIG was just a victim of a lot of shitty luck â" bad year for business, you know, what with the financial crisis and all. Edward Liddy, the company's CEO, actually compared it to catching a cold: "The marketplace is a pretty crummy place to be right now," he said. "When the world catches pneumonia, we get it too." In a pathetic attempt at name-dropping, he even whined that AIG was being "consumed by the same issues that are driving house prices down and 401K statements down and Warren Buffet's investment portfolio down."

Liddy made AIG sound like an orphan begging in a soup line, hungry and sick from being left out in someone else's financial weather. He conveniently forgot to mention that AIG had spent more than a decade systematically scheming to evade U.S. and international regulators, or that one of the causes of its "pneumonia" was making colossal, world-sinking $500 billion bets with money it didn't have, in a toxic and completely unregulated derivatives market.

Nor did anyone mention that when AIG finally got up from its seat at the Wall Street casino, broke and busted in the afterdawn light, it owed money all over town â" and that a huge chunk of your taxpayer dollars in this particular bailout scam will be going to pay off the other high rollers at its table. Or that this was a casino unique among all casinos, one where middle-class taxpayers cover the bets of billionaires.

People are pissed off about this financial crisis, and about this bailout, but they're not pissed off enough. The reality is that the worldwide economic meltdown and the bailout that followed were together a kind of revolution, a coup d'état. They cemented and formalized a political trend that has been snowballing for decades: the gradual takeover of the government by a small class of connected insiders, who used money to control elections, buy influence and systematically weaken financial regulations.

The crisis was the coup de grÃce: Given virtually free rein over the economy, these same insiders first wrecked the financial world, then cunningly granted themselves nearly unlimited emergency powers to clean up their own mess. And so the gambling-addict leaders of companies like AIG end up not penniless and in jail, but with an Alien-style death grip on the Treasury and the Federal Reserve â" "our partners in the government," as Liddy put it with a shockingly casual matter-of-factness after the most recent bailout.

The mistake most people make in looking at the financial crisis is thinking of it in terms of money, a habit that might lead you to look at the unfolding mess as a huge bonus-killing downer for the Wall Street class. But if you look at it in purely Machiavellian terms, what you see is a colossal power grab that threatens to turn the federal government into a kind of giant Enron â" a huge, impenetrable black box filled with self-dealing insiders whose scheme is the securing of individual profits at the expense of an ocean of unwitting involuntary shareholders, previously known as taxpayers.

Candidates for Sale
What do Obama and McCain have in common? The same big donors, who will expect to have their way no matter who wins


Posted Aug 21, 2008 9:42 AM

Remember the total, hideous, inexcusable absence of oversight that has been the great hallmark of George Bush's America for almost eight years now? Well, now we're getting to see that same regulatory malfeasance applied to yet another cornerstone of our political system. The Federal Election Commission â" the body that supposedly enforces campaign-finance laws in this country â" has been out of business for more than six months. That's because Congress was dragging its feet over confirmation hearings for new FEC commissioners, leaving the agency without a quorum. The commission just started work again for the first time on July 10th under its new chairman, Donald McGahn, a classic Republican Party yahoo whose chief qualifications include representing Tom DeLay, the corrupt ex-speaker of the House, in matters of campaign finance.

Apart from the obvious absurdity of not having a functioning election-policing mechanism in an election year in the world's richest democracy, the late start by the FEC makes it almost impossible for the agency to do its job. The commission has a long-standing reluctance to take action in the last months before a vote, a policy designed to help prevent federal regulators from influencing election outcomes. Normally, the FEC tries to root out infractions and loopholes â" fining campaigns for incomplete reporting, or for taking shortcuts around spending limits â" in the early months of a campaign season. But that ship sailed way too long ago to take the stink off the 2008 race.

"The time for setting the ground rules was earlier," says Craig Holman, a lobbyist with the watchdog group Public Citizen. "There isn't time to do much now."

That's especially true given the magnitude of what we're dealing with here: the biggest pile of political contributions in the history of free elections, nearly a billion dollars given to presidential candidates in this season alone. Because the FEC has been dead in the water for so long, it's likely that we'll still be in the dark about a large chunk of this record manure pile of campaign contributions when we go to vote in November.

But that doesn't mean that a little sifting through campaign records doesn't tell us quite a lot about who's backing whom in these races. The truth is that the campaigns of both Barack Obama and John McCain are being inundated with cash from more or less exactly the same gorgons of the corporate scene. From Wall Street to the Big Oil powerhouses to the military-industrial complex, America's fat-cat business leaders know that the Animal House-style party of the last eight years that made almost all of them rich with bonuses, government contracts and bubble profits is about to come to an end, and someone is going to have to pay to clean up the mess. They want that someone to be you, not them, and they've spared no expense to make sure both presidential candidates will be there to bail them out next year.

They're succeeding. Both would-be presidents have already sold us out. They've taken the money and run â" completing the cyclical transformation of the American political narrative from one of monopolistic Republican iniquity to an even more depressing tale about the overweening power of corporate money and the essentially fictitious nature of our two-party system.

In layman's terms, we've gone from being screwed to being fucked. Who knows â" maybe Barack Obama will surprise us if he wins the election. But if you look at the money, it doesn't look good.

Thanks in part to the dormant FEC, corporate America has had even easier access to the candidates than usual in its effort to buy off the next government before the crash. In fact, this election has seen some excellent new innovations in the area of campaign-fundraising atrocities. Chief among them is the rise of so-called "joint committees."

It used to be that campaigns could raise a maximum of $2,300 from each individual.. Now, both candidates â" but especially McCain, who far outstrips Obama in this area â" routinely hold fundraisers in which individuals can give far more to a joint committee. Technically, the candidate still pockets only $2,300 in contributions. The bulk of the money raised â" in McCain's case, a whopping $70,100, or 30 times the previous limit â" goes to the state and national arms of the candidate's party, which can then spend the unprecedented haul on behalf of the candidate. "This allows CEOs to walk in the door and drop $70,100," says Holman. "It basically allows campaigns to exceed the spending limits."

McCain has raised more than $63 million via these joint committees, thanks to more than 1,000 "megadonors" who have each given at least $25,000 to his campaign effort. Obama, by contrast, has some 471 megadonors â" and a close examination of their backgrounds underscores some of the differences in corporate America's attitudes toward the two candidates.

One of McCain's chief sources of corporate money is the private-equity firm Kohlberg Kravis Roberts, memorialized for its takeover of RJR Nabisco in the movie Barbarians at the Gate. Through the pretext of joint committees, 10 KKR executives have given McCain $285,000, and it's not hard to figure out why. Two of McCain's key campaign proposals â" lowering the corporate tax rate to 25 percent and making purchases of industrial equipment fully deductible â" would save a single KKR subsidiary, Energy Future Holdings, $49 million.

"Just in his tax policies alone, McCain is saving corporate America $175 billion a year," says James Kvaal, who analyzed McCain's tax policy for the nonprofit Center for American Progress.

McCain has also raked in big contributions from two other giants of the buyout world: the Carlyle Group (famous for its close ties to the Bush administration) and the Blackstone Group (whose co-founder, Pete Peterson, wrote a $28,500 check to McCain after he took home almost $1.8 billion from a public offering last year). McCain has also received monstrous sums from hedge-fund managers, attracted by his pledge to keep the tax rate on their earnings at only 15 percent.. Executives and family members in a single hedge fund, Knott Partners, have contributed some $225,700 to McCain's campaign.

Then there's the predictable influx of cash from would-be military contractors. John Lehman, a former secretary of the Navy whose firm builds the Superferry transport vessel, not only donated $28,500 of his own money, but bundled at least $250,000 for McCain from other donors. Donald Bollinger, who is a contractor on the controversial Littoral Combat Ship, gave $27,300 and bundled a whopping $500,000. Anyone want to bet on a decrease in Naval appropriations in a McCain presidency?

McCain has also received big money from telecommunications magnates. The senator has always been a friend to the industry: Back in 2003, just four days after AT&T sent him a check for $10,500, he sponsored a bill to ban state and local taxes on Internet service. Since 2007, McCain has taken in some $1.3 million from the communications industry. Just four members of the McCaw family, which owns the telecommunications firm Eagle River, have kicked in $123,200. McCain's campaign manager, Rick Davis, was a former lobbyist for BellSouth, Verizon and SBC Communications. His deputy campaign manager, Christian Ferry, was a partner to Davis at Verizon. One of his chief advisers, Charlie Black, is the head of the lobbying firm BKSH and Associates, which represents AT&T. His Senate chief of staff, Mark Buse, worked for AT&T Wireless. All told, of 66 current and former lobbyists working for McCain, some 23 come from the telecommunications industry.

Given McCain's telecom backing, it's not surprising that the senator has had one of his characteristic changes of heart. As recently as last November, McCain was staunchly opposed to retroactive immunity for telecommunication companies that took part in Bush's illegal spying on American consumers, saying their actions "undermine our respect for the law." Now, jammed to the gills with telecom cash, McCain calls himself an "unqualified" supporter of immunity, praising the telecom industry's warrantless wiretapping as "constitutional and appropriate."

All the same, plenty of other evidence suggests that much of Wall Street is betting on an Obama win. In fact, some observers believe that KKR announced a multibillion-dollar public offering this summer because it expects McCain to lose. "They're doing the public offering now so that the compensation can be taxed at the lower rate while Bush is still in office," says a strategist for a major labor union. "They're betting Obama is going to win, and they're getting their money while they can."

Other companies are getting in on the ground floor with the new chief by stuffing money in his ears. Overall, Obama is flat-out kicking McCain's ass when it comes to Wall Street contributions, raking in nearly $9 million from securities and investment executives, compared to $6.2 million for McCain. Obama has received more contributions from Goldman Sachs than from any other employ

RDB (1)

MichaelSmith (789609) | more than 4 years ago | (#28565265)

I thought DEC RDB was a pretty good query language. I never got into SQL as a result. I am glad people are thinking about alternatives.

Next Up... (1, Funny)

grepya (67436) | more than 4 years ago | (#28565269)

...say no to the tyranny of... er.. English. Let's stick with the combination of grunts, squeals, crying and gesturing that has proven so effective for toddlers all over the globe for thousands of years. And if we surrendered the traditional languages that we are so irrationally attached to, who knows what revolutionary new communication scheme the next-generation kids will come up with.


The problem is performance not SQL (3, Interesting)

presidenteloco (659168) | more than 4 years ago | (#28565289)

The problem is the performance of transactions and persistence and distribution of data techniques, not
whether we are using a logic-like STRUCTURED QUERY LANGUAGE to ask for data matching certain conditions.

The latter is still, and will continue to be, very useful.

It's just that now that we can assume local clusters and WANs worth of co-operating data stores, there
are probably better, more performant ways of implementing persistence, replication, distribution of data
than traditional RDBMS implementations.

The two concerns: The logical model of how we QUERY for data (or combine it in bulk), which is the core of SQL,
and how we persist it and retrieve it quickly, now have more options for being separated.

Re:The problem is performance not SQL (1)

Crias (1388217) | more than 4 years ago | (#28565763)

The problem, I think, runs a little deeper than all that though.

SQL is unfortunately tied fairly tightly to an RDBMS implementation. All those "join" statements, various ways of expressing "constraints" such as "foreign keys" - all are considered "integral" parts of SQL.

No, you don't have to provide them. A Post-Relational like Amazon SimpleDB could, theoretically, use SQL for querying and just trim back the feature-set.

But perhaps it'd be wiser to look at a query language more specific to the Post-Relational model?

Perhaps SQL stopped being "SQL" and started being "Structured Relational Query Language". *shrugs*

Re:The problem is performance not SQL (4, Insightful)

oGMo (379) | more than 4 years ago | (#28565955)

It's just that now that we can assume local clusters and WANs worth of co-operating data stores, there are probably better, more performant ways of implementing persistence, replication, distribution of data than traditional RDBMS implementations.

You can also assume magical fairy dust and free energy, but that doesn't make it so. You can ask if there are better ways, but you can't assume it, and in the end you will find there is no magic.

Clusters and replication are NOT NEW. Not even remotely new. There is, in fact, nothing new architecturally at all that would indicate some new capability that hasn't already been repeatedly analyzed and tried. That doesn't mean you can't tweak something for a situation, or that you need a giant Oracle database for everything, but "the web" and "cheap hardware" change the equation by precisely nothing.

What has changed the equation is cheap, unimportant data, which covers the majority of the web. "Real" applications, where data integrity is important (like say, your bank account), and immediate accuracy guaranteed, require the main thing you use a database for: data integrity. Your facebook page, your google search, that blog entry, or some video on youtube: these don't matter. If it's a little slow, or doesn't update immediately, or you get an error, no one is losing money. No one cares.

In essence, if a reliable database isn't important for your app, your app isn't really handling important data. This may be fine; in the mainstream, there's a lot of noncritical stuff. But this doesn't make databases unimportant.

Not mutually exclusive (3, Insightful)

JobyOne (1578377) | more than 4 years ago | (#28565357)

It's pretty easy to say "yes" to alternatives without saying "no" to SQL.

Just because a crowbar can pull out a stubborn nail better doesn't mean they should replace all the hammers. Then what would we put nails in with? Different tools for different jobs.

Nailguns (1)

tehdaemon (753808) | more than 4 years ago | (#28565505)

Most nails are put in with nailguns. Hammers these days are mostly used for demolition of various sorts, including pulling nails. T

RDBs are good, but SQL is horrible (0)

Cyberax (705495) | more than 4 years ago | (#28565369)

The idea of RDB is cool, relational algebra is quite neat. But SQL itself is horrible.

I'd like to have a language which will allow me to access intermediate tuples cleanly and return hierarchic structures. For example, if I want to fetch all customers and all their bids in one query I have to use inner join. And that results in LARGE number of rows (Cartesian product of customers and their bids).

Also, I'd like to see stuff which is not easily expressed in relational algebra, like running sums or grouping on a computed field.

Re:RDBs are good, but SQL is horrible (1)

larry bagina (561269) | more than 4 years ago | (#28565447)

If you want hierarchical data, you could use a hierarchic database.

Re:RDBs are good, but SQL is horrible (1)

Cyberax (705495) | more than 4 years ago | (#28565519)

I do not want hierarchical data storage. I want to create trees from relational data.

I don't see anything that prevents me from doing this in theory. In fact, ANSI SQL already has support for hierarchic queries (which makes it Turing-complete, BTW).

Re:RDBs are good, but SQL is horrible (1)

Marcos Eliziario (969923) | more than 4 years ago | (#28565827)

The way to store tree like structures on a relational database is using nested sets, not pointer-like ids.
The main current backslash against databases is that most developers don't have a clue about set theory, relational algebra, let aside the inner workings of a concurrent database system.
Many of the problems solved by RDBMS are going to have to be solved again by those new tools that are promised to replace RDBMS.

Those who ignore history, are condemned to repeat it.

Re:RDBs are good, but SQL is horrible (0)

Anonymous Coward | more than 4 years ago | (#28565497)

The idea of RDB is cool, relational algebra is quite neat. But SQL itself is horrible.

I'd like to have a language which will allow me to access intermediate tuples cleanly and return hierarchic structures. For example, if I want to fetch all customers and all their bids in one query I have to use inner join. And that results in LARGE number of rows (Cartesian product of customers and their bids).

Also, I'd like to see stuff which is not easily expressed in relational algebra, like running sums or grouping on a computed field.

If you're getting a Cartesian product from a query like that, either the DB architect was a moron or (most likely) you need to learn about the WHERE clause.

Re:RDBs are good, but SQL is horrible (1, Informative)

Anonymous Coward | more than 4 years ago | (#28565539)

select * from customers c, bids b where c.customer_id=b.fk_customer_id order by c.customer_id, b.bid_date

Seems pretty simple. What's wrong with an inner join? Your getting exactly the number of rows that you need to answer your question, no more no less.

A cartesian product would be more like: select * from customers c, bids b. But that's not what you want.

As for hierarchical structures, Oracle db has ways to do this, although I admit the syntax isn't that straight forward: http://download.oracle.com/docs/cd/B19306_01/server.102/b14200/queries003.htm

Re:RDBs are good, but SQL is horrible (1)

godrik (1287354) | more than 4 years ago | (#28565593)

For example, if I want to fetch all customers and all their bids in one query I have to use inner join. And that results in LARGE number of rows (Cartesian product of customers and their bids).

I am not sure I get your point. If you do an inner join it means you want all the tuple < player,bid > that makes sense. If there is a lot of them, well, there is a lot of them, there is nothing to do about it. If you complain about each player being repeted on several bid (since they bid more than once). It should not be a problem, as long as you stay in the RBMS, this should not incur any overhead. When you read them, you can just compress them on the fly.

If you really want to remove those "extra" player values, why do you want to have a single query ? You can just make a query for each player.

Also, I'd like to see stuff which is not easily expressed in relational algebra, like running sums or grouping on a computed field.

technically, they cannot be expressed in relationnal algebra, you have to add non algebraic operator to do that. SUM and GROUP BY in SQL are not part of relationnal algrebra. The problem with those operators is that it is difficult to do any optimization on them. Howver, the user may still want to have them. I would also be interested in a language that can express such things efficiently

Re:RDBs are good, but SQL is horrible (1)

jyx (454866) | more than 4 years ago | (#28565641)

For example, if I want to fetch all customers and all their bids in one query I have to use inner join. And that results in LARGE number of rows (Cartesian product of customers and their bids)

You might want to explain yourself a bit more there, if I want all customers and all their bids I would expect a LARGE number of rows. What magic algorithm is out there that will give you all your data, but at the same time make it less than what it is?

Or do you want just one row of customer data and then all there orders under that? Good luck getting your admin staff creating reports off that spreadsheet.

I think what you are interested in reporting tools - they do the things you ask for often in a nice drag and droppy way - but you still need to get the data to those reports and I haven't found anything better for that job than SQL (yet).

Data is hard work, eventually any solution to querying databases is going to be as complicated as SQL because there is a infinite number of ways people will eventually want to look at it.

Re:RDBs are good, but SQL is horrible (1)

Cyberax (705495) | more than 4 years ago | (#28565761)

Let's suppose that we have 1000 customers and each customer has 100 bids, and each bid has 5 sub-items.

If we retrieve all of them using inner joins - we'll have to transmit and read 100*1000*5 rows. Quite a large number.

If we first fetch customers and then fetch their bids (using a second query) and then sub-items we'll have to read 1000+1000*100+1000*100*5 rows. However, each time we fetch only relevant data which can result in huge savings of bandwidth (some database protocols are naive enough to transmit full rows).

Re:RDBs are good, but SQL is horrible (1)

caerwyn (38056) | more than 4 years ago | (#28565897)

I'm not sure what your point here is. People do that sort of sequential querying all the time- each query simply asks for the subset of data of interest. What, exactly, are you unable to do in SQL that you want to be able to do?

Re:RDBs are good, but SQL is horrible (1)

Cyberax (705495) | more than 4 years ago | (#28565993)

Why do I need to do several queries? It would be nice to be able to do this in a single query.

Re:RDBs are good, but SQL is horrible (1)

caerwyn (38056) | more than 4 years ago | (#28565919)

Also, I'd like to see stuff which is not easily expressed in relational algebra, like running sums or grouping on a computed field.

Grouping on a computed field is quite easy, so if you're waiting for SQL to support it... you've been waiting too long, it already does.

As for running sums, that's the sort of thing that Oracle already has and just went into the Postgres 8.4 release that was on slashdot the other day.

Your SQL complaints are a bit out of date. :)

Cartesion Product? (2, Insightful)

gbutler69 (910166) | more than 4 years ago | (#28565941)

Epic Fail. You're wrong. It in now way results in a "Cartesion Product". That would be a "Cross Join", not an "inner join". From my experience, people who complain about SQL and relational database, are, for the most part, ignorant. They really don't even understand what they are saying or what they are talking about. I've seen so much abuse and misunderstanding of relational data and SQL in my career, that I just have to laugh at this sort of thing.

Yeah, so why are they better? (5, Insightful)

Anonymous Coward | more than 4 years ago | (#28565371)

If I was to read the article, I bet somewhere someone would be wittering on about Key Value Datastores.

The brainchild of a generation brought up on high level collections, they learn one (in this case Map) and apply it to everything.

Sadly SQL, and RDBMS, works for most people. It maps object data well (oh whaaaa, i have to do foreign keys - GROW SOME FUCKING BALLS YOU LAZY GRADUATE!) and it is well understood. And with abstractions like LINQ to query them, even the lazy dumb Windows .NET programmer doesn't have to strain their brain to learn SQL.

And when you have terabytes of specific unique data, you clearly should go away to work out how best to store it. Even a RDBMS/SQL solution is too generic for all problems.

Re:Yeah, so why are they better? (1)

Hurricane78 (562437) | more than 4 years ago | (#28565577)

Well, file systems, databases, object inheritance trees, etc, they all are based on the incomplete concept of hierarchical trees and maps. While in reality, everything can be generalized trough graphs. Generic graphs. Of course everyone got its own poor fix for this. File systems have links, databases have foreign keys, and OO languages have interfaces or multiple inheritance. It's a mess, because it is an afterthought.

I stopped using all those approximations of data structures, and use my own high-performance ontologic graph library for everything that I would use a treelike structure for. I also can stick it on top of a file system or RDBMS, and even have a UI element to browse it. I do not look back. :)

Re:Yeah, so why are they better? (1)

schnablebg (678930) | more than 4 years ago | (#28565929)

You use your own hand rolled libraries for standard data structures? I really hope I don't end up on a project with you or inheriting one you've worked on.

Re:Yeah, so why are they better? (1)

spiffmastercow (1001386) | more than 4 years ago | (#28565703)

I'd say LINQ is significantly harder to use than SQL most of the time. The only real exception is when you need to convert the value of a subquery into a comma-delimited list

Re:Yeah, so why are they better? (3, Insightful)

fabs64 (657132) | more than 4 years ago | (#28565869)

Saying RDMS's map object data well is a bit of a stretch, they map relational data well and that's it.

http://www.codinghorror.com/blog/archives/000621.html [codinghorror.com] for some good background on the problems.

For me using an RDMS as the persistence layer for an object-oriented application has ALWAYS felt like a bit of a kludge. Like we're using it just because it's what we have, rather than the best tool for the job.

What's the benefit exactly? (3, Insightful)

SendBot (29932) | more than 4 years ago | (#28565389)

I'm not seeing anything that offers a real advantage over using advanced features like one finds in postgres combined with memcached. Some of my program likes to think of its data as a structured object while other parts like seeing that data as rows in a table (they even link up to other tables through foreign keys!).

Re:What's the benefit exactly? (1)

phantomfive (622387) | more than 4 years ago | (#28565611)

The main problem with relational databases is that they use a completely different storage scheme than your program does. Databases are organized into tables, rows and columns, but programs are organized into random access variables, structs, and classes. Thus, to use data from a relational database in your program, you need to have a conversion layer that converts from tables to random access, and back. These guys are saying it would be nice if we had a way to store this stuff that didn't require a conversion layer.

And I agree. However I also think it would be nice if I could keep all my data in RAM all the time, for easy access. It's just not practical. If all your queries are straightforward selects on ID, then there really is no great reason to use a full database. But once you start doing more complex joins and searches, a database is, while not always convenient, still more convenient than the alternatives.

Re:What's the benefit exactly? (1)

hibiki_r (649814) | more than 4 years ago | (#28565707)

A conversion layer is wasteful when there's only one way to look at your data. In that case, key value pairs can perform better, no question.

The problem lies in situations where you need to look at the data in 5 different ways. or 50. Then, a single object model for your data is a whole lot less practical than having a conversion layer, and have the data in a very flexible format, like a relational model.

Re:What's the benefit exactly? (1)

SendBot (29932) | more than 4 years ago | (#28566039)

Well, I have a conversion layer to create the object my program uses, but I can't think of a need to convert it back. All the things that make it what it is are a result of all the little things that interact with the db. Using triggers, it knows when to update parts of itself. The parts that interact with the db often don't care about the object, even if it's being used as in input to those parts.

When I DO need to care about the object, replicate it, or maintain persistence, then I use...


memcached. (I rtfa'd and even amazon's thing said it was a basically a key -> value system)

If I did this exclusively with my object instead of sql, I don't know right away how I would do all my searches and processing because everything is so hugely related... and I think the whole point of this nosql thing is that it's a non-relational alternative for when things are pretty basic, but comprise enormous data size.

Except in the in end (1)

xednieht (1117791) | more than 4 years ago | (#28565475)

The Patriots themselves levied their own heavy taxes emulating those against which they had originally rebelled

In the end it's all just 1's and 0's.

What else is there to use besides SQL (1)

Orion Blastar (457579) | more than 4 years ago | (#28565485)

go back to flat files aka DAT files.

Use the old DBase III standard DBF files?

Use the old Lotus 123 WK1 files?

Use MS-Office MS-Access MS-Excel etc files?

Use comma separated values files?

SQL set a standard for relational databases, a structured query language that almost any database can use and then build extensions to it.

Will the Post-SQL age begin, and will it be object oriented and a fifth generation language?

Re:What else is there to use besides SQL (1)

godrik (1287354) | more than 4 years ago | (#28565635)

go back to flat files aka DAT files.

Technically, that is what they do. Basically, they just say that they do not need classical RDBMS to do their job. I agree with them that RDBMS makes poor implementation of big dictionnaries. :)

How about saying yes to the alternative (4, Insightful)

syousef (465911) | more than 4 years ago | (#28565535)

Saying no to SQL and relational databases is just fine if you've got something better to replace it with. However I know of no such thing. The reason they're popular is that they are so powerful for data storage. If something better came along you wouldn't even need to say no to SQL. You'd just say yes to the newer better rival.

XML / XPATH / XQUERY / XSLT / Xhausted (0)

Anonymous Coward | more than 4 years ago | (#28565555)

SQL can suck. The alternatives the PHB might choose aren't necessarily better. Be careful what you wish for.

Whatever (0)

Anonymous Coward | more than 4 years ago | (#28565557)

Like unix being dead - someone else thinks SQL is dead and worthless.

I disagree, there is no ONE solution. SQL works great for many types of data access. But an object based db can be great for other types.

SQL is dead. Long live SQL. :-)

Misses the point (1)

kc8jhs (746030) | more than 4 years ago | (#28565583)

There are plenty of ways to store data inexpensively in a RDBMS. There are plenty of GPL and low cost RDBMS available.

The real issue is that the more and more we move into complex data structures and we push the limits of what an ORM can do with those simple, inexpensive RDBMS, the more problems we run into trying to map our objects into rows in tables.

Here [appspot.com] is one of the more interesting solutions that I've seen to the problem, but it only work over relatively simplistic data where managing indexes by hand is ok, and it's okay for the indexes to be incomplete at any given moment. Ironically, that gives them more availability than trying to force MySQL to do indexes. But it really depends on the data and needs.

SQL is not a database (5, Insightful)

j. andrew rogers (774820) | more than 4 years ago | (#28565627)

SQL is not a database, it is a standard interface to a feature set commonly associated with relational models. Before everyone standardized on SQL, there were other relational query languages. The "No" part of "NoSQL" refers to the fact that some basic elements of relational implementations cannot be usefully expressed using a much simpler distributed hash table model.

All the "NoSQL" does is eliminate all the parts of traditional relational databases that do no scale -- discarding the bottleneck rather than fixing it. These are things like joins and external indexing. Unfortunately, discarding those things means you discard a lot of very important functionality as a practical matter, notably the ability to do fast, complex analytics. Adopting the NoSQL architecture runs contrary to the trend toward more real-time, contextual analytical processing. There are a great many analytical applications that are not amenable to batch-mode pattern-matching, and the NoSQL model is a lot less applicable than I think some people want to acknowledge. In its domain, it is a great tool but it has many, many prohibitive limits. We are essentially trading power for scale.

That said, do not take this as an endorsement of traditional SQL relational databases either, as they have a number of serious limitations themselves. As just mentioned, a number of the core analytical operations those models support are based on algorithms that scale poorly. The SQL language itself has mediocre support for many abstract data types (e.g. spatial) and data models (e.g. graph), which in part reflects the inadequacies of the assumed underlying database algorithms (e.g. B-trees) that are implicit in SQL. The inability to efficiently do event-driven/real-time applications is also more a reflection of the access methods used in databases than any intrinsic weakness in SQL; SQL may be clunky for that purpose, but that is not the real limiter.

A truly revolutionary deviation from SQL would usefully implement a superset of the features SQL supports, not take them away. Of course, we would need access methods more capable than hash tables and B-trees to useful implement those features, which is a lot more work than discarding features that scale poorly. NoSQL is a stopgap technical measure for that small subset of applications where the serious tradeoffs are acceptable.

Pros & Cons of non-relational solutions (5, Interesting)

kpharmer (452893) | more than 4 years ago | (#28565701)

Note that most of these solutions come from the interwebs, social networks, etc. And it isn't so much anti-sql as it is anti-relational database (sql != rdb).

The basic premise is that we need different solutions that: can scale very high for very narrowly scoped reads & writes, don't need to perform ranged queries / reporting /etc, and don't need ACID compliance. And that may be the case. Sites like slashdot, facebook, reddit, digg, etc don't need the data quality that ebay needs.

On the other hand, ebay achieves scalability AND data quality with relational databases. And when I've worked with architectures that scale massively and avoid the relational trap for better solutions - they inevitably later regret the lack of data quality and complete inability to actually get trends and analysis of their data. It *always* goes like this:
    Me: So, is this thing (msg type, etc) increasing?
    Developer: No idea.
    Me: Ok, so lets find out.
    Developer: How?
    Me: I don't know - typical approach - lets query the database.
    Developer: It'll take four+ hours to write & test that query and then days to run. And when it's done we might find that we wrote the query wrong.
    Me: What?!?
    Developer: We had to do it this way, you can't report on 10TB databases anyhow
    Me: What?!? Are you on crack? there are dozens of *100TB* relational databases out there that people are reporting on
    Developer: well, we probably don't need to know what that trend is anyhow
    Me: I'm outta here

Data out-lives applications (5, Insightful)

4to6Offshore (594235) | more than 4 years ago | (#28565865)

First: my mantra: Data belongs to the organization, not the application... if the app fails and data is accessible then we all go on - if the data fails or is locked away - what was the point of the app again?

In a SQL database then data is understood by the organisation, DBAs and data architects. If left to app developers taking an app-centric approach to data... I get nervous quickly.

So long as the data is just as definable and accessible as current SQL databases then all good - give me an app with some odd-ball storage then it is bye-bye.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account