Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Keeping Google's In-house Database Ticking

Zonk posted more than 6 years ago | from the need-synonyms-for-huge dept.


An anonymous reader writes "ZDNet has a short but interesting piece on the what Google did with its 12GB database when it became a challenge for the finance department. The database was split into three, says Chris Schulze, technical program manager for Google — one for the current financial planning projections, one for the actual current data from existing HR and general ledger systems, and one storing historic information. The article says Google has been using a variety of products from Hyperion (recently bought by Oracle) to manage its internal financial systems since 2001."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered


12gb? (0)

Anonymous Coward | more than 6 years ago | (#18899179)


Re:12gb? (0)

beckerist (985855) | more than 6 years ago | (#18900279)

I would assume that 12gb is either:
A) The size of ONE of their DB's (ie: Google Earth, Google Talk, etc...) but considering that GMail alone allows for 2gb of use per person, and I'm using almost 40% myself...12gb is probably an underestimate. Exa? Zetta? Yotta? ...Udeka? [slipperyfish.net]

Re:12gb? (2, Informative)

yahooadam (1068736) | more than 6 years ago | (#18901529)

you can practically gather from the summary, without even RTFA that this is talking about google's financial database, which is most likely used to find out how Google is doing, nothing to do with the services Google offer the public

WTF WTF? (1)

SharpFang (651121) | more than 6 years ago | (#18899189)

"Right now, we're on a not very powerful Windows box," Couglin said. "We definitely are wanting to go to Unix when we go to System 9."

Re:WTF WTF? (5, Insightful)

pasamio (737659) | more than 6 years ago | (#18899245)

Its an advertisement! Read the bottom: "Angus Kidman travelled to Orlando as a guest of Hyperion". The thing mentions Hyperion a dozen times, its the old trick of substituting news with press releases written by companies.

Google Magic (0)

Anonymous Coward | more than 6 years ago | (#18899275)

It has Google in the name, that magically transforms it from a lame press release to "Stuff that Matters"

Re:WTF WTF? (5, Insightful)

eln (21727) | more than 6 years ago | (#18899385)

It's not only a press release, it's a very unimpressive one. Hyperion can handle data larger than 12 GB?! Stop the presses! You could manage a company of 50, maybe even 60 employees with that!

Plus, the "story" says that in order to manage such a large (*cough*) amount of data, the solution was to partition the database into 3 different parts. Now, I can see partitioning it for ease of management along functional areas, but certainly not because it grew to 12 whole gigabytes. If you can't handle chunks of data larger than 4 GB without partitioning it, you're in big trouble.

I'm guessing the "anonymous reader" who submitted this works for Hyperion.

OLAP is a different beast (5, Informative)

alxtoth (914920) | more than 6 years ago | (#18899875)

12 Gb of _relational_ database falls under "nothing to see, move along". But Essbase http://en.wikipedia.org/wiki/Essbase [wikipedia.org] is doing OLAP http://en.wikipedia.org/wiki/OLAP [wikipedia.org] , which means that data is pre-aggregated across multiple _hierarchies_ . Those 150 users are likely the top management looking at the revenue, or reviewing the budget.
In Open Source land there are similar projects: http://freshmeat.net/search/?q=olap&section=projec ts [freshmeat.net]

Re:OLAP is a different beast (1)

jhfry (829244) | more than 6 years ago | (#18915203)

Mod the parent up... a quick read of the the linked Wikipedia article on OLAP and I can see how 12GB of data can be so problematic.

The problem had nothing to do with the amount of data... but the amount of RAM and Processing power required to support even the small amount of data in a OLAP cube.

Read the Wikipedia article and learn somthing before you jump to conclusions!

Re:WTF WTF? (1)

paulerdos (205999) | more than 6 years ago | (#18901185)

Hyperion can handle data larger than 12 GB?! Stop the presses! You could manage a company of 50, maybe even 60 employees with that!
Hasn't Google been hiring aggressively recently? I'm pretty sure they have more than 60 employees...

Re:WTF WTF? (1)

Hoi Polloi (522990) | more than 6 years ago | (#18900085)

Oracle has to pay for all that dough it spent on buying Hyperion somehow. I'm surprised it didn't toss in more Oracle references too.

Re:WTF WTF? (1)

beset (745752) | more than 6 years ago | (#18899745)

Now i'm a Unix fanboy but that's crazy. We run Microsoft Dynamics Ax (mssql based) on a 7 year old server with a 20gb database. The server hasn't been rebooted in nearly 2 years, has 2gb of ram and quad p3 733s and absolutely flys.

Some marketing firm is going to get a big bonus for such a decent slashvertisment.

Re:WTF WTF? (0)

Anonymous Coward | more than 6 years ago | (#18913747)

I have several E4500 servers at home that have 12Gb of memory and one of 14Gb of memory.

How can this thing be that slow? 14GB of memory should be nothing for google!

Only 12 GB? (4, Funny)

operagost (62405) | more than 6 years ago | (#18899297)

12 GB? You call that big? I haven't seen an Exchange mail store that small!

Re:Only 12 GB? (1)

markg11cdn (1087925) | more than 6 years ago | (#18899949)

Lamest article (spam) of the day? Is this a very late April Fools joke? I've got tables that are much larger than 12GB in my Oracle DB that perform fine without partitioning. Indexes and good coding.

Only 12GB? (4, Insightful)

WapoStyle (639758) | more than 6 years ago | (#18899317)

I don't get it, that doesn't seem like much to me.

We have many databases that are larger here from MSSQL to Oracle, some around the 600GB mark.

What's so special about Google's database?

Re:Only 12GB? (0)

Anonymous Coward | more than 6 years ago | (#18899471)

Agreed..12GB is chump change, but because it is about Google is it news worthy. If this same article had been about Microsoft, the slashdottians would have been yelling that Microsoft is sooo dumb they can't deal with 12GB of data properly.

It's Google (1)

PIPBoy3000 (619296) | more than 6 years ago | (#18899599)

As far as I can tell, the only reason this is news is that it's Google. I manage several very large database, some in the hundreds of GB. Probably the most interesting of the big ones involves auditing people who are accessing a medical records system. The tricky part isn't managing every command passed by tens of thousands of users, but rather trying to find ways to pull out the needle of bad behavior from the endless normal activities. Was doctor A supposed to look at patient B's record? Is user A somehow related to patient B?

The only thing of technical note in the article is the ordinary problem with database jobs taking a long time. On a related note, I've kept waffling on whether or not to break off the above audit database to its own server. The processing time for some of the import jobs is over an hour. Strangely enough, advances in hardware have been such that it still resides on our main database/web server without any problem. Maybe Google just needed to throw hardware at the problem.

Re:Only 12GB? (4, Informative)

alxtoth (914920) | more than 6 years ago | (#18900159)

TFA is about a _cube_ of 12 Gb . Not _relational_ database. Read my other post http://developers.slashdot.org/comments.pl?sid=232 481&threshold=1&commentsort=0&mode=thread&pid=1889 9385#18899875 [slashdot.org]

Re:Only 12GB? (1)

alxtoth (914920) | more than 6 years ago | (#18909505)

It is one thing to insert/retrieve a row to/from a 600 GB database, another do a SUM .. WHERE..join..join..join.. GROUP BY TIME,PRODUCT over "only" 12GB . And since it is called online analytical processing, you would expect results .. today. Essbase does several equivalent queries per second.

Re:Only 12GB? (5, Informative)

hemp (36945) | more than 6 years ago | (#18900183)

Google's Hyperion database is an OLAP ( on-line analytical processing ) database rather than an OLTP ( on-line transaction processing ) database. OLAP databases are optimized more for processing human queries rather than standard transactions (like most MSSQL and Oracles are). Hyperion incorporates multi-demensional data hierarchies and other data formats that are difficult if not impossible to model in straight SQL(think of a Rubik's cube in 7 demensions).

The downside of this approach is that it can cause lengthy time periods when the cubes needs to be re-calculated. In Google's case, evidently, this took 48 hours.

Re:Only 12GB? (1)

qray (805206) | more than 6 years ago | (#18902307)

Shoot 15 years ago I was working with MS Access databases around 750megs of data. (Not that was a good idea at the time). Took quite a while to run Access's repair utility on them.

I hope they never have to deal with AVI or other similar large 21 gig files. I guess you could chop them up as well and watch them individually.

Seriously the only reason I could see for splitting them up is load balancing. A high volume transaction rate might force one to do something like that.

Re:Only 12GB? (1)

recharged95 (782975) | more than 6 years ago | (#18909579)

One word. Sarbanes-Oxley. I'm sure the query requirements are a b*tch.

Then again 12 GB is a walk in the park for Oracle Financals. Again this is another tech company that serves great product, but uses wacky internal setups. Not a good sign of eating your own dog food.

Err..yeah...it's really a Hyperion ad (3, Funny)

kiwimate (458274) | more than 6 years ago | (#18899403)

This is the bit that gets me in the summary:

ZDNet has a short but interesting piece

Interesting to whom, precisely? Hyperion's marketing department? Scant technical details and really only notable for the link to the photos of Google's new Sydney office which are kind of interesting, I suppose, in an "ooh wow shiny...okay what's next?" kind of way.

Re:Err..yeah...it's really a Hyperion ad (0)

Anonymous Coward | more than 6 years ago | (#18899493)

Submitted by Anonymous, no less. And then +'d up through the Hosepipe. Gee, I wonder how this ever made it to the front page of Slashdot.

A thinly veiled press release about a financial database that is a whole 12GB! Well, Slashdot just hit a whole new low. Oh and look, it was posted by Zonk as well. Doesn't that just round it off for you?

Re:Err..yeah...it's really a Hyperion ad (1)

garcia (6573) | more than 6 years ago | (#18900071)

Interesting to whom, precisely? Hyperion's marketing department?

Or I suppose to users of Hyperion and the staff that uses it daily -- like me. While I don't particularly care for how we are directed to use Hyperion (no ad-hoc reporting but instead pre-created queries that we can only modify the reporting of the end result) in theory it could be an extremely useful tool for many companies.

It's much easier to learn than what is offered in Access or other reporting tools I have used. The only way I could use it for my ad-hoc reporting would to be to import a CSV of a table I want to report from and then use their tools from that. Not so good for what I like to do (automation) but I have used it to make pretty charts when the boss asked.

Oooh... Google DB (0)

Anonymous Coward | more than 6 years ago | (#18899405)

Guess this is more of the Google/MySQL database bonanza...

Please guys, if you're looking to pump before you IPO and dump, do it on Wall Street, not Slashdot.

Serious WTF (0)

Anonymous Coward | more than 6 years ago | (#18899417)

Google have terabytes of data. I imagine Google engineers are now horribly embarrassed by what the stupid finance department got up to.

Google's search database is over 12 gigs (0)

Anonymous Coward | more than 6 years ago | (#18899439)

The databases google uses to run searches are quite a bit over 12 gigs...as are the database backends behind gmail and google maps. What the heck is this article even talking about?

It is as if the reader who submitted it thinks a 12 gig DB is big for google.

Press release (2, Insightful)

gtoomey (528943) | more than 6 years ago | (#18899449)

1. Move on, nothing to see
2. Sack Zonk (sorry man you post some good stories, this ones a stinker)

12 TB? (0)

Anonymous Coward | more than 6 years ago | (#18899503)

12 Terabytes maybe? What the hell are they talking about? This is suppose to advertise some company? Ohhhh! Look at you!! Your system manages 12 GB of data!!!

12 GB ain't nothing. Hell, my NNTP db is way bigger than that and that only contains header information.

HR? (1)

EveryNickIsTaken (1054794) | more than 6 years ago | (#18899537)

one for the actual current data from existing HR and general ledger systems
Since when does HR have anything to do with accounting or finances?

Re:HR? (0)

Anonymous Coward | more than 6 years ago | (#18899807)

one for the actual current data from existing HR and general ledger systems
Since when does HR have anything to do with accounting or finances?

Quite a lot, actually. (and yes, I am an accountant)

People tend to like to be paid. Compensation rates, pay schedules and the like tend to HR items. Further, people like benefits. If Google offers choices of benefits, the deductions from pay may vary. Benefit choices tend to be HR items.

Finaly, there are these things called taxes. Since TFA is apparently about Google Australia, (and no I didn't RTFA) I can't say what the taxes are there, but in the USA, income tax withholding rates, state and local income tax and unemployment taxes may vary from employee to employee. Again, all of that data about employees is in HR's hands.

It makes sense that these systems are two parts of a single whole since employees drive a lot of accounting activity.

Re:HR? (1)

EveryNickIsTaken (1054794) | more than 6 years ago | (#18899925)

Payroll != HR in any medium to large-size company.

Re:HR? (0)

Anonymous Coward | more than 6 years ago | (#18900197)

Granted. Hyperion is an ERP (or is it BPM?) app, and HR's files can be useful for projections and planning. Why store it separately if you're going to need to use it frequently?

(CAPTCHA= profound. This post is not. Oh well.)

Re:HR? (0)

Anonymous Coward | more than 6 years ago | (#18899813)

Payroll?? Perhaps their employee benefits and salary projections are linked in. Don't know for sure, just speculating. -AZ

Question (-1, Troll)

Anonymous Coward | more than 6 years ago | (#18899565)

What kind of bear is best.

FALSE! Black bear.

Black bears eat beets.

Bears. Beets. Battlestar Galactica.

Advertisement (1)

bernywork (57298) | more than 6 years ago | (#18899597)

Also, I think they are talking about AU only. I highly doubt the US only has a 12 GB database.

Re:Advertisement (1)

vidarh (309115) | more than 6 years ago | (#18900029)

It's a financial system. 12GB of financial data is quite a bit - it could very well be worldwide.

Re:Advertisement (1)

bernywork (57298) | more than 6 years ago | (#18902749)

I wonder then what they count as "Financial data" and "Sales data" versus other people. I know companies with 1000 users who have a hell of a lot more data than this.

Re:Advertisement (0)

Anonymous Coward | more than 6 years ago | (#18901877)

No, that's just without the conversion rate.

12 GB is not 12 gigs. (5, Funny)

nathan s (719490) | more than 6 years ago | (#18899703)

Obviously that's 12 GOOGLE-Bytes*. Which are far huger than ordinary bytes, or even gigabytes, and therefore much more interesting.

* Note that GoogleBytes are still in beta and therefore the exact amount of storage in a single GB is yet to be determined.

Re:12 GB is not 12 gigs. (1)

Fbelch (9658) | more than 6 years ago | (#18900691)

Don't forget GoogleBytes continually increment.... as time goes on! Since it is beta!!

It's a hit piece (0)

Anonymous Coward | more than 6 years ago | (#18899705)

The point of the piece is not to laud Google's technical prowess (cough) in partitioning the data but to embarrass the (potentially non-responsive) vendor into improving their product.

Certainly Hyperion can't be happy about having an extremely high profile customer publicize needing to partition a modest 12GB database even if it is running on a Windows platform. And Google is not likely to have gone public with this unless they've already tried to unsuccessfully resolve the problem with Hyperion.

You think GB Stands for Gigabytes!? (2, Funny)

VE3OGG (1034632) | more than 6 years ago | (#18900113)

No no no! It stands for Googlebytes. Each Googlebyte is approximately 1024x10^10,241,024 bytes. So as you can see, a 12 Google Byte database is quite substantial...

Re:You think GB Stands for Gigabytes!? (1)

Fbelch (9658) | more than 6 years ago | (#18900735)

Don't forget GoogleBytes continually increment.... as time goes on

Since it is beta!!
You can't control it


Re:You think GB Stands for Gigabytes!? (1)

NevarMore (248971) | more than 6 years ago | (#18903069)

How many Libraries of Congress and Olympic Size Swimming Pools is that??

No Clue (0)

Anonymous Coward | more than 6 years ago | (#18900121)

Is it me, or does it seem like they have no clue regarding ERP architectures? And, as everyone has said, 12GB is an impossibly small size. The 'BASE' install of most financials packages is more than twice as large as this. And they can scale very easily, we have some databases approaching 2 TB.

Re:No Clue (1)

JoeCommodore (567479) | more than 6 years ago | (#18900267)

Right. And the partitioning of data is something anyone who has worked with DBs have either considered or implemented, not really news in the structure change either.

Glitch in the Matrix (2, Insightful)

suv4x4 (956391) | more than 6 years ago | (#18900239)


The database grew in size to more than 12 gigabytes, and the period restructuring required to ensure accuracy could see the system, which is now used by more than 150 staff, taken offline for two hours at a stretch.

"Right now, we're on a not very powerful Windows box," Couglin said.

Uhmm, maybe it's some other Google, right...?

I can't be reading a press release from Google, the one that has more or less a copy of the whole Internet on its servers, whining about the difficulties of managing a small database on a slow Windows machine.

Re:Glitch in the Matrix (1)

sweetlipsbutterhoney (1018188) | more than 6 years ago | (#18902383)

I would have thought this weird, too, until I started working as an IT Auditor and saw all manner of crazy old legacy systems supporting the accounting and financial reporting systems of major companies. I've seen major tax expenses totaling millions of dollars tracked through some of the most wicked Excel spreadsheets you could imagine. There was one fairly major software company I worked on ($1 Billion in revenue last year) that ran their whole online company store (where 90% of it's sales went through) on a couple of NT boxes.
The point is that accountants and accounting departments do not like big changes, so my experience has been that systems supporting financial reporting have a long life.

Re:Glitch in the Matrix (0)

Anonymous Coward | more than 6 years ago | (#18903743)

It's a press release from Hyperion... take it with a grain of salt. I wonder how badly those quotes are taken out of context.

Oh, my! (3, Informative)

Jerky McNaughty (1391) | more than 6 years ago | (#18900311)

So Google used horizontal partitioning [wikipedia.org] to split load across servers? Wow, that's rocket science. None of us in the database community have thought of doing this before. :-) But, if you want to find some news here, you can. One nice thing that Google did recently was to donate their horizontal partitioning code for Hibernate to the open source community. Hibernate Shards [hibernate.org] definitely needs a lot of work to get it to the point where it does a lot of stuff that people would want, but, hey, release early and often!

C'mon throw us a bone Slashdot! (0)

Anonymous Coward | more than 6 years ago | (#18906199)

Couldn't they have at least partitioned it among three iPod Nanos running Linux?

Drink from the firehose (0)

Anonymous Coward | more than 6 years ago | (#18900385)

The next Google article's in the pipes:

In this article we'll follow through a case study on the difficulties of watching a HD DVD on a slow Windows XP machine, and the hurdles the Google team had to meet, to watch the entire rental movie and return it in time.

"We had to Google for the keys, decode it, split it in pieces and setup a small cluster of servers to play the audio and picture separately, then feed the low-def stream back to our slow Windows box in real-time" said Chris Schulze, technical program manager for Google, said during the presentation.

He added: "We could've rented the DVD instead, but we like to solve tough challenges the Google way".

Bunch of pansies... (1)

sonofagunn (659927) | more than 6 years ago | (#18900585)

Their Hyperion Essbase cube was 12 GB? And they had to partition it into 3? That's nothing. We have MS Analysis Services cubes of almost 400 GB (partitioned into 3 seperate ones, like Google). If this is supposed to be an advertisement for Hyperion, it's not very impressive. Of course, we are using 3 seperate 8 processor Itanium boxes with 64 GB RAM. That helps some.

This is an OLAP system (1)

kiwipom (920352) | more than 6 years ago | (#18900933)

Everyone here seems to be forgetting that Hyperion is an OLAP Cube holding highly aggregated data, consequently it doesn't have to store enormous amounts of data, it probably only hold last years actuals and this years actuals and budget data which even for a v.large company is pretty small. Consequently 12GB is actually a lot of data for the product. Think about the purpose of the product before picking holes in it. I don't work for Hyperion, but have done a few projects with it's Essbase product, which is actually shit hot.

Ironical (0)

Anonymous Coward | more than 6 years ago | (#18901065)

Read the freakin article:

"Google is used to sifting through huge amounts of information to generate its search results, but a 12 gigabyte database proved something more of a challenge for its own financial management and planning systems."

The whole point is that Google is used to dealing with terabytes and a simple little 12GB database actually posed a challenge to them. I really hope the irony isn't lost on this crowd...

Huh??? (0)

Anonymous Coward | more than 6 years ago | (#18901629)

12 GB is very small beer nowadays. I work on a data warehouse of >20Tb, once considered
very large indeed, now just another database. Sheeeeit, I wouldn't hesitate to run a 12Gb
database on my laptop....

Why the hell do they think this is news???

3fu3k! (-1, Flamebait)

Anonymous Coward | more than 6 years ago | (#18902505)

That he documents head Spinning you. The tireless

yu0 Fail it (-1, Troll)

Anonymous Coward | more than 6 years ago | (#18903861)

knows for sure what tO get soMe eye you got there. Or

Sergey, Larry give me a call... (1)

corecaptain (135407) | more than 6 years ago | (#18904069)

I have a spare PC running centos and mysql that can handle those troublesome 12GBs like a chainsaw
cutting through butter.

Call me and I can drive over to the plex today and get it running over the weekend for a very reasonable fee...

This must be a joke? right? Google has problems with 12GB of data?

someone please tell me it's at least 12 TB w/thousands of concurrent users...

Financial Data (1)

c0d3r (156687) | more than 6 years ago | (#18933217)

What everyone needs to realize is that this is Financial Data. I worked with a database of over 4GB of nothing but sales orders for Cisco, and that was only for one technology group. This translates to a lot of money, and keeping the integrity, security and performance of these kinds of databases are very, very important and very stressful due to the responsiblility. Also, for financial data, correctness is more important than mondo fast algorithms that add complexity. Divide the 12GB by average value of each record to see how big the database is in business value.
Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account