Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Inside Amazon's Data Centers

CmdrTaco posted more than 3 years ago | from the lookit-all-dem-wires dept.

Cloud 42

1sockchuck writes "Amazon Web Services usually doesn't say much about the data centers powering its cloud computing platform. But last week the company held a technology open house to discuss the company's infrastructure, sharing cost data and a glimpse of a modular data center design. The key point: AWS is growing like crazy. 'Every day Amazon Web Services adds enough new capacity to support all of's global infrastructure through the company's first 5 years, when it was a $2.76 billion annual revenue enterprise,' said AWS Engineer James Hamilton, whose presentation (PDF) is available online."

cancel ×


Sorry! There are no comments related to the filter you selected.

Don't read this if you are a (non-tech) manager. (0)

SMoynihan (1647997) | more than 3 years ago | (#36425184)

Don't read this if you are a (non-tech) manager. It might contradict the self-evident truth that the cloud really is up in the sky, managed by security conscious fairies and run on pixie dust.

Then again, I'm likely posting on the wrong site here...

Re:Don't read this if you are a (non-tech) manager (2)

fuzzyfuzzyfungus (1223518) | more than 3 years ago | (#36425268)

I think that it was manager-safe enough. There were a few diagrams that you should be careful to skip over(a few had percentage numbers that weren't finance related!); but the rest had numbers with dollar signs in front of them safely ensconced next to a variety of recognizable corporate logos with a generic-but-high-tech photo of some server racks in the background.

A few good buzzphrases as well. I'm definitely going to ask my department about why they aren't being more proactive and action-oriented about getting our network "on the Moore's law path". We'll never be ready for exascale cloud-centric computing if they don't...

Re:Don't read this if you are a (non-tech) manager (0)

Anonymous Coward | more than 3 years ago | (#36425554)

I thought this was the common consensus on how the "cloud" looks like: []

PDF is heavily slashdotted (1)

Anonymous Coward | more than 3 years ago | (#36425210)

Please use Coral Cache [] to reduce the load on the original.

Re:PDF is heavily slashdotted (2)

jewelises (739285) | more than 3 years ago | (#36425226)

Someone should host the PDF on S3.

Re:PDF is heavily slashdotted (0)

Anonymous Coward | more than 3 years ago | (#36428056)

>Someone should host the PDF on S3.


Re:PDF is heavily slashdotted (1)

Anonymous Coward | more than 3 years ago | (#36425228)

They should have hosted it in the cloud.

Re:PDF is heavily slashdotted (4, Informative)

jewelises (739285) | more than 3 years ago | (#36425262)

Coral Cache isn't working for me, but google has cached [] it.

Is that where they keep the data? (1)

For a Free Internet (1594621) | more than 3 years ago | (#36425212)

Now I'm told that the data exists on cumputers, and that these computers are connected to a network. But the question is, what is the network connected to? And what about the books?

Once you start to think aboyut all this it quickly becomes an overwhelming problem. My solition is to throw corn kernels at people while they arent looking and then run away behind a tree, so that it.

What good is that (3, Insightful)

Anonymous Coward | more than 3 years ago | (#36425264)

"Every day Amazon Web Services adds enough new capacity to support all of's global infrastructure through the company's first 5 years, when it was a $2.76 billion annual revenue enterprise,"

What good is that when their network design is so flawed that a single human error can bring the whole service down for a week?
Amazon Web Services? No, thanks, I prefer reliability.

Re:What good is that (0)

Anonymous Coward | more than 3 years ago | (#36425296)

Do you have a link to that incident?

Re:What good is that (0)

Anonymous Coward | more than 3 years ago | (#36425342)

Not all failures are public.

Re:What good is that (1)

0123456 (636235) | more than 3 years ago | (#36425352)

Do you have a link to that incident?

He does, but unfortunatley the cloud server is down.

Re:What good is that (5, Informative)

nereid666 (533498) | more than 3 years ago | (#36425406)

Nope (1)

aclarke (307017) | more than 3 years ago | (#36430314)

The anonymous coward claimed that "the whole service [was] down for a week". The link you posted isn't even remotely close to that.

Re:What good is that (0)

Anonymous Coward | more than 3 years ago | (#36425408)

What good is that when their network design is so flawed that a single human error can bring the whole service down for a week?

That never happened btw.

any reason they don't buy larger servers? (1)

alen (225700) | more than 3 years ago | (#36425356)

presentation said average is $1450 per server which is an entry level 1U. why not just buy fewer larger servers to share the workload? it would probably save on power costs?

Re:any reason they don't buy larger servers? (2)

PickyH3D (680158) | more than 3 years ago | (#36425386)

I'd bet that buying the cheaper, yet still powerful computers is probably more cost effective considering the rate that these machines burn out under their kinds of load. Why spend so much money on a machine that will be broken, when you can just buy multiple cheap ones instead? Then, when one burns out, you're still running at a higher capacity even with the negative of higher power draw.

Re:any reason they don't buy larger servers? (2)

tibit (1762298) | more than 3 years ago | (#36425404)

They didn't say if it's per a real or virtual server methinks, and I don't know if the assumption that they pay whatever anyone else would be paying holds water either. I'm sure they can get a lot more "server" for $1.5k than me or you would.

Re:any reason they don't buy larger servers? (5, Informative)

micron (164661) | more than 3 years ago | (#36425466)

Looking at a 3-5 years TCO, and power costs where these data centers are located, power costs are noise in the equation.
Taking advantage of commodity pricing in the lower tiers is where the savings is at. Example, single socket systems are a lot cheaper on the procs and mainboards than dual sockets. Quad socket processors are significantly more expensive per proc..
At $0.10 per KwH, a 400W server is $350/year to power. Quad socket processors (Intel I7) can be as high as $4500 each!

Re:any reason they don't buy larger servers? (1)

SuperQ (431) | more than 3 years ago | (#36426020)

Don't forget to factor in PUE into the power costs. An average datacenter has a 1.5x multiplier for power. Over 3 years a 400W machine will cost about $1500 to power.

your calculation is flawed (1)

dutchwhizzman (817898) | more than 3 years ago | (#36426244)

You need to factor in the cost of cooling those KwHs too. Not just that, but there are large area's of the planet where you can't buy a KwH for 0.10 US dollar. Try 3 times as much for an average European country. You'd be looking at two to six times your $350/year easily for running your 400W server just in power costs. Also, those $4500 CPUs won't cost that much for large companies like Amazon and Google. Rest assured they pay probably substantially less than $1000 for those CPUs. These companies are big enough to have their own servers made to their own specs. They don't pay list price to AMD, Intel, Dell, HP or IBM, but they do have to pay for actual power bills, until they have their own power plants.

Re:your calculation is flawed (1)

micron (164661) | more than 3 years ago | (#36428226)

Your points are exactly why these large data centers are locating in areas with access to cheap power. I used $0.10 as an example, however, that is extremely HIGH when factoring in the deals that large data centers strike with regional power providers that are giving cheap access to hydro power. This is the exact reason folks are not putting large data centers in Europe and the Bay Area. Power has to be cheap for the economics to work out.
Also, read the papers published by Google and Facebook. These guys are pushing PUE below 1.2, whereas a typical data center is 2.0 or higher!

Re:your calculation is flawed (1)

micron (164661) | more than 3 years ago | (#36428262)

They pay the per 1,000's price (or close to it), which is not list price. These numbers are published. This is the quantity in which they purchase the processors, so it makes sense. The number you can't see is the negotiated power deals.
The price for a quad processor capable system is still 4x to 10x what a single socket processor costs.

Real world at scale. (1)

Anonymous Coward | more than 3 years ago | (#36425474)

Larger servers are prohibitively more expensive than el cheapo low-end servers, but fail at effectively the same rate.

As for power costs, no, not necessarily - unless your large consolidation boxes are underpowered. Most of the 4U boxes I've dealt with have massive power requirements.

Re:Real world at scale. (2)

mlts (1038732) | more than 3 years ago | (#36426946)

I would disagree at the same rate figure.

What you pay for with the bigger servers is redundancy. The higher end servers that Oracle and IBM offer cost more, but they engineer for reliability, not absolute cheapness of price as in the commodity x86 market. Yes, you can improve uptime by adding redundancy on upper layers up to and including the backend app.

On one end, you have FB's solution where reliability isn't as much as issue as deploying fast. The top layer backend app handles the redundancy. On the other end, you have mainframes and IBM Parallel Sysplex. Most businesses end up somewhere in between.

Almost always, you get what you pay for when it comes to servers.

open networking (1)

callmebill (1917294) | more than 3 years ago | (#36426172)

Probably the most interesting slide is "Sea Change in Networking" where they compare side-by-side the openness and accessibility of software and hardware for commodity servers vs. the complete lack of it for networking equipment. I.e., on the server side you have access to APIs and hardware information to create and customize your own good stuff. Whereas on the networking equipment side, whatever comes out of the box you'll have to take it or leave it. With Amazon being such a large consumer of this equipment, I wonder (A) what's the opportunity for new networking equipment companies with a different attitude toward openness and how Amazon might like to start buying their gear, or (B) whether the incumbent suppliers will respond by tweaking their offerings so that Amazon would be less disgruntled.

FPGA (1)

Lennie (16154) | more than 3 years ago | (#36427030)

The problem with networking is, you need dedicated hardware (ASICs) to able to push the packets at those speeds.

Those ASIC's are not the same for every vendor, I wouldn't be surprised if their are big differences in implementations and very strongly tied to the 'control plan' (the software running on the management-CPU).

Ofcourse the control plane could be a lot more open.

The closest things I can think of are the open source software which can be used to program FCGA's: the NetFPGA and liberouter project do something like that. They both can handle packets at 10GB/s and use FPGA's.

And supposedly their is one vendor who created open source code for their switches. But isn't a big vendor and I don't think anyone has actually seen the code yet.

Also I don't remember the name right now. :-)

Am I the only one? (2)

kannibul (534777) | more than 3 years ago | (#36426194)

Am I the only one that thinks that cloud computing is really NOT going to take off? Amazon is spending a boatload of money, in the hopes that other people are going to be willing to offload a boatload of the hopes that they can process data faster/better/etc off-site. Just the security issues alone concern me, but the additional bandwidth is what really gets things going... Just saying - from a business perspective (and my perspective as a network admin), it seems that Cloud Computing is a 90's .com bubble that will pop in the near future...can Amazon (or any other company) REALLY offer a reduction in costs, or an increase in performance that justifies the (obvious increase in) cost, especially when you look at the whole package? I think not. Same reason not every company has made the switch on the backend to free server-level operating systems (FreeBSD or Linux, namely) - it doesn't matter if the platform may perform better in every regard or has inheritantly better security and even can do the job just as well or gets rejected simply because the costs will increase - in the case of linux/unix, because the skillset to operate and maintain it is more rare and rare = higher price...and unlike a product you purchase (optionally) once like is annual. Cloud computing...would be annual, vs a piece of hardware you own, with software that you own the rights to use

Re:Am I the only one? (2)

donnyspi (701349) | more than 3 years ago | (#36426356)

Yeah, the thing is it already has taken off. They can hardly add additional capacity fast enough. AWS is a very profitable business for Amazon. Our company switched from a traditional data center setup to running fully on AWS and it did reduce our costs and increase our agility and scalability. That said, like anything it's not a fit for everybody.

Re:Am I the only one? (1)

kannibul (534777) | more than 3 years ago | (#36426944)

I agree that it would have to be something very adaptable to usage with cloud-based approached...still, even where I work (Oklahoma) we're on a tornado-prone area, and I work for a small "State and Local" government agency. We considered moving our datacenter to a location that is rated for EF5, flood proof, redundant this and that, and would safehouse our data entirely from any imaginable threat except nuclear. The costs were reasonable as well for what we'd co-locate...but what killed it was the need for bandwidth. Given our funding situation (current backlash against government spending, and my employer is a common target it seems for spending cutbacks) - having that additional cost per-month to have a live-failover location wasn't going to work for us. In the end, we did nothing about it...just we make sure our backups are good and send those off to an EF5 rated we have for decades. Basically, what I'm saying is the same thing - that if your business model is such that having something like this in place, where it can be done, great, but, I imagine that there is a large customer-base that it's just not applicable. I fear that Amazon (and others) are taking a bit more of a "build it and they will come" approach, vs watching what the market will do and sustain. Right now, it's a buzzword...just like IPO and Web Site and Internet were a buzzwords in the 90's.

Re:Am I the only one? (2)

silky1 (1609493) | more than 3 years ago | (#36426370)

I would think staffing a team to maintain hardware in an IT Department is much more expensive than paying Amazon to run the backend. Thinking of many corporations, all have data centers, IT staff, hardware, software etc, and here is Amazon with a single data center capable of providing all this infrastructure at what has to be a competitive price for them to survive. This has to be a better option for many companies, especially small companies. My behemoth company is constantly looking for off shoring opportunities to further reduce costs so smaller companies may not have this as an option. This seems like a simple decision in using Amazon's services. Comparing to the 90's and bubble, at least this is a tangible business model and not a dream like many of those companies back then were.

Re:Am I the only one? (2)

noahm (4459) | more than 3 years ago | (#36426514)

Well, they did say that "Every day Amazon Web Services adds enough new capacity to support all of's global infrastructure through the company's first 5 years". They wouldn't be doing that if it wasn't taking off.

Keep in mind that "cloud computing" is a really vague term that is used to describe all sorts of different technologies and platforms. It's still really immature, in terms of best practices and that sort of thing. There are still a lot of people throwing around the "cloud will solve all our problems" mantra.

At a previous job (in a computer science research lab), the director got all excited about Amazon's cloud stuff. He wanted my team to move everything to AWS, from our email service to our backups of user data. We researched pricing, thought about how it'd change our service model, and wrote up our findings. When we presented the results to him, he simply refused to believe that the AWS solution was actually significantly more expensive than what we had been doing. It was inconceivable to him that there were workloads that didn't see a major cost reduction when moved to the cloud. Eventually saner heads prevailed, and we were able to convince him that we shouldn't move all our stuff to AWS, but it took a lot of effort to overcome the buzzword enthusiasm.

That said, however, I really think there are situations where the AWS model makes a lot of sense. Your application must be able to scale to additional servers easily, transparently, and at least semi-automatically. It must be easy to deploy in multiple datacenters, so you can insulate yourself from outages like the big one that hit Amazon's East Coast region a few months ago. And it must be easy to scale your application down to fewer servers when the workload is lighter. If your usage patterns are spiky enough that you will occasionally need lots of capacity for a short time, but less capacity most of the time, then AWS is a really interesting option.

I haven't thought as much about Google AppEngine or Microsoft Azure, but they'd likely be similar. The big difference between them and AWS is that they essentially abstract away all the scaling stuff. It's a potentially good model, but you need to write your application specifically to target their environment. You need to do that for AWS as well, but the model is somewhat more similar to the traditional server-oriented model.

Re:Am I the only one? (1)

mlts (1038732) | more than 3 years ago | (#36427008)

It hasn't just taken off, it has already made a sonic boom that has rattled eardrums of anyone in virtually every IT shop worldwide.

Every company out there hears that the mystical Cloud can solve all computing problems, has no security issues, etc. Reality hits when the realization comes into play that choosing a cloud provider means a permanent relationship -- it is virtually impossible to change providers due to each having different APIs. The fact that one will have to pay for a data center somewhere also rings true. Almost invariably, a cloud solution will be more costly than having the capacity in house in the long run.

Re:Am I the only one? (1)

alen (225700) | more than 3 years ago | (#36427830)

it's not a reduction in costs as in a reduction in up front cash required. in finance class in college we learned that a fast growing company can easily go bankrupt due to the fact that cash comes in months after the revenue is recognized but expenses like salary and bills have to be paid out on time.

in this case small companies can pay a little at at a time to amazon and grow rather than spend a fortune on infrastructure

Re:Am I the only one? (2)

dohnut (189348) | more than 3 years ago | (#36428664)

As people have said, it already has taken off. Personally, I use AWS for much humbler reasons than most. I use it for hosting my personal domain(s) -- web, DNS, SVN, etc. I used to just have a linux server in my basement do it all, but then you have a $500 machine to maintain, a static IP to provision, possible TOS violations with your ISP, poor upload speeds, etc. For a few dollars a month I can host everything I used to at home on a virtual linux server with redundant storage attached and excellent bandwidth. It was a no-brainer for me.

I also store important documents and other irreplaceable items (photos, video recordings, etc) with their S3 service. I still have things stored locally, but it gives me additional piece of mind to have it somewhere redundant and external. You don't need to use Amazon for that, but it is more convenient for me than backing stuff up to media and then physically locating it somewhere off-site.

As far as the recent newsworthy outage at AWS, I trust Amazon will learn from their mistakes and fix them more than I trust my local ISPs to do so. Also, that network outage did not affect any of my sites or data.

Re:Am I the only one? (1)

NateTech (50881) | more than 3 years ago | (#36432304)

Mainframes never worked. ;-)

It's just another swing back to what was originally promised out of the 90s "client/server computing" buzzword.

"Thin clients", and keep the data and processing power on the centralized servers.

The "new" part about this "cloud" stuff is that instead of needing to ask for your own "timeslices" on a big shared computer, they created a way to stuff your own virtualized OS on the thing, and made the "big server" out of a bunch of little ones.

Nothing really all that new here. The main driver is that mobile and hardline bandwidth is cheap now, and people who have more than one network-connected device, want their "stuff" always accessible to them from anywhere they can get Net access.

What is amazon doing different? (1)

Synerg1y (2169962) | more than 3 years ago | (#36426300)

I've seen a few different data centers, they all look about the same, why should we care? I'm pretty sure it was SUN Microsystems who came up with the modular shipping container design, which was back in the day.

It's over 9000! (1)

synapse7 (1075571) | more than 3 years ago | (#36426606)

Is this the same web service systems that couldn't support the demand(how much could it be) for lady GaGa's album?

Somebody missed ... (2)

Anomalyst (742352) | more than 3 years ago | (#36427006)

the opportunity to crow about how effective it might be in generating bitcoins.

The only problem I have with Amazon (1)

MJMullinII (1232636) | more than 3 years ago | (#36432704)

Is that they are, IMHO, extremely hypocritical in how they conduct themselves.

On the one hand, they put themselves forward as this oh so "wholesome" family environment, while doing absolutely NOTHING to stop their "Pro" sellers from listing the most vulgar items you would care to search for.

These pages are also listed with the helpful "Sell yours here!" but if you actually set up an account and attempt to sell said items yourself, you run the definite risk of having your account terminated by what might as well be an anonymous email with no way to explain EVEN IF you intent was simply to alert them to their flawed system (meaning if you weren't just trying the old "they do it too" excuse)

Now, before someone mistakes me for a conservative (little "c", not in relation to anything political) prude, let me say I know this because I set up such an account :)

I saw that the adult items were selling and since I didn't have a "Pro" account -- that is too say, I couldn't sell anything that wasn't already listed on Amazon's site, you can't create pages yourself unless you have a "Pro" account -- I simply took it on faith that if Amazon was allowing the pages to remain, they must have been kosher with it. That is until about my first week in when my account was permanently suspended.

Now I want to make clear that I know full well what Amazon says -- I know they forbid "pornography" (though their description of "pornography" is, in my opinion, open enough to include almost anything they might simply decide they don't like) and I'm not claiming I should get some special exemption from the rules -- I'M SAYING THAT THEIR BEHAVIOR -- leaving pages with items they claim to forbid up and encouraging others to "join up" -- WOULD -- if done by a police officer on the street -- BE CONSIDERED ENTRAPMENT IN A COURT OF LAW.

I've since switched to eBay, where they have a section for "adult" items with age verification, etc. and that's cool. I guess my complaint, though, is that eBay never claimed they don't allow such products, Amazon does and, for lack of a better term, that makes Amazon a liar. Reminds me of the scene from the pilot episode of "Boardwalk Empire", when the kid shoved the envelope into Steve Buscemi's hand after robbing the bootleggers in the woods, he said "I didn't ask for this". The kid turned to him and said "That's your cut, regardless -- you can't be HALF a gangster".

His point was that if you're going to be a crook, THEN BE A CROOK and stop pretending you're anything else. Likewise, if Amazon intends to make money selling "unwholesome" merchandise THEN DO WHAT EVERYONE ELSE DOES -- speaking of setting up an area with age verification, etc. -- AND STOP PRETENDING YOUR SOMEHOW "BETTER" THAN YOU REALLY ARE.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>