Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

eBay Deploys 100TB of SSDs, Cuts Rackspace By Half

Soulskill posted more than 3 years ago | from the go-big-or-go-home dept.

Data Storage 197

Lucas123 writes "eBay's QA division was facing mounting performance issues related to its exponential growth of virtual servers, so instead of purchasing more 15k rpm Fibre Channel drives, the company began migrating over to a pure SSD environment. eBay said half of its 4,000 VMs are now attached to SSDs. The changeout has improved the time it takes the online site to deploy a VM from 45 minutes to 5 minutes and had a tremendous impact on its rack space requirements. 'One rack [of SSD storage] is equal to eight or nine racks of something else,' said Michael Craft, eBay's manager of QA Systems Administration."

Sorry! There are no comments related to the filter you selected.

depends if you are IO bound or need storage (2)

smash (1351) | more than 3 years ago | (#36969364)

For sites like ebay i have no doubt this makes sense. For the average small business I suspect they are far less IO bound and need storage...

Re:depends if you are IO bound or need storage (1)

Anonymous Coward | more than 3 years ago | (#36969516)

Any business dealing on the web (aka hosting) would benefit from ssd more due to the nature of hosting (lots of random reads).

That said, why not a hybrid enviroment with ssd on the forefront and hdd on the back. The right tool for the right job.

SSD for random light reads.
HHD for large sequential block reads

Re:depends if you are IO bound or need storage (3, Insightful)

myurr (468709) | more than 3 years ago | (#36969634)

Actually the vast majority of websites, even business websites, are really low traffic and they benefit far more from storage space (especially when shared with other sites) than speed. Operating system RAM caching will usually make up any performance deficit those kinds of low traffic sites may experience, where the majority of the traffic that does go to their sites tends to be read only and directed at only a few pages on any given day. Premature optimisation adds either (or both) complexity and expense, and is unnecessary for 90+% of the web.

Scalability is a nice problem to have, and the majority of websites would do an awful lot better if they worried about driving traffic more than they worried about scalability.

Re:depends if you are IO bound or need storage (1)

Anonymous Coward | more than 3 years ago | (#36969704)

Yes but how many companies on the low end actually host their own servers? I'd imagine the majority go to a consultant or hosting company. In these cases (not that they have much say in the matter), a SSD would be better since it's shared (on the startup / low end) among many business owners.

If a company is hosting it's own server, it's because A) they require it or B) there it staff made a bad choice
In A) SSD is generally better
in B) Why are they even attempting it with all the extra issues of maintenance like power backup, software backup, hardware failures, etc

So the way I see it, the companies that DO have a choice, tends to be companies that would *generally* benefit from it.

Re:depends if you are IO bound or need storage (1)

Anonymous Coward | more than 3 years ago | (#36969648)

Pity SSD's thrash the pants of 15krpm drives in sequential reads as well as random.
a 15k fibre channel drive only puts out between 50 and 100mb/s depending on the place on disk
high performance ssd's will do read and write at 500mb/s through sata or sas

Re:depends if you are IO bound or need storage (2)

arglebargle_xiv (2212710) | more than 3 years ago | (#36970458)

Any business dealing on the web (aka hosting) would benefit from ssd

Because SSD is web scale.

SSDs are faster in all respects. (1)

unity100 (970058) | more than 3 years ago | (#36970714)

not only random reads.

you have to use ssds for ALL reads, but, hdds for all writes. since ssd life is still limited with number of write operations conducted.

78% power savings - that's pretty awesome too (1)

spineboy (22918) | more than 3 years ago | (#36969778)

Might even pay for itself by the years end
Barring any major catastrophes - expect to see may companies with server farms to go this route soon

Re:78% power savings - that's pretty awesome too (1)

arglebargle_xiv (2212710) | more than 3 years ago | (#36970510)

Might even pay for itself by the years end

Given that an OCZ Colossus 2 (just under 1TB) retails for $2,500 (and even when converted to Canadian dollars that's still a lot), you'd have to be using some awfully expensive power generation for it to pay for itself by years end. Or were you referring to the SATA cables that you needed in order to plug them in? You could probably pay those off in power savings.

Re:78% power savings - that's pretty awesome too (1)

EraserMouseMan (847479) | more than 3 years ago | (#36970638)

You guys are all missing the point. eBay missed the boat. They blew their whole wad on SSD failing to realize that GE just announced a new age of optical Nirvana.

Re:78% power savings - that's pretty awesome too (0)

Anonymous Coward | more than 3 years ago | (#36970718)

No, it doesn't pay for much. An SSD consumes 1 watt or less. For simplicity, lets just say....zero watts. My older had drives consumed about 10 watts, but more recent ones (last 3-4 years) seem to have been 6-8 watts. For simplicity, lets say 10 watts. So a generous estimate is that an SSD will save you 10 watts per drive.

For a device in use 24/7, a good rule of thumb is that 1 watt costs $1 per year (based on a cost of 11.5 cents per kwh...your cost may be more or less). Right now the cheapest SSDs I've seen is $65 for a 64GB Kingston at costco. So even with our generous 10 watts saved, and assuming you can get away with this budget drives, You are still looking at over 6 years to break even.

Re:depends if you are IO bound or need storage (1)

TheRaven64 (641858) | more than 3 years ago | (#36969864)

How many small businesses have more than 1TB of data, in total? Unless you're in a business that does a lot of image or video work, you're probably generating less than 1GB per month. One drive - hard disk or solid state - will probably have a large enough storage capacity for a typical small business. The bigger problem is redundancy and backups, not total storage capacity.

Re:depends if you are IO bound or need storage (2, Insightful)

hairyfeet (841228) | more than 3 years ago | (#36969930)

I agree but there should be an up side to this, in that we'll get some solid data to see if Atwood at coding horror is correct that SSDs need to be judged on a hot/crazy scale [codinghorror.com] due to the insanely high SSD failure rates.

The reason I personally won't recommend SSDs to customers or carry them myself is after having my two "must have teh benchmarkz!" gamer customer buy top o' the line SSDs both of them had the SSD fail without warning which for me is unacceptable. Sure they got them replaced under warranty, but so what? That didn't cover their downtime, the cost of the 1Tb drives they had to pick up to hold them over while they waited on the RMA, or the data they lost between their last backup and the SSD failure. in the end they went dual Raptors in RAID 0 with the 1TB as backup and game storage.

The nice thing about HDDs is in my experience one gets plenty of warning before they go tits up. The drive gets noisy, you get Windows delayed write failures, the drive starts running hot, you get SMART warnings, something. with both of the SSDs it was just...poof. Dead drive. on one I was able to retrieve a little bit of the data, on the other it wouldn't even show up in BIOS.

That is why unless I have a customer where IOPS is all like in TFA or someone that lives on a heavily mobile laptop AND has the time for daily backups or stores all their important data in the cloud to just stay away until they get the kinks worked out. it is still too new and IMHO they haven't really got the tech down yet, hence all the failures. I tell my customers to have a fat HDD with plenty of cache along with giving Windows 7 plenty of RAM (4Gb minimum, 8Gb is better) to really use Superfetch and preload their apps into memory and they'll be happy. Yeah it doesn't boot with SSD speed, but who boots anymore?

Re:depends if you are IO bound or need storage (1)

hawkinspeter (831501) | more than 3 years ago | (#36970426)

I've seen plenty of traditional HDDs fail without warning. I'd be wary of looking for signs of impending failure on any kind of drive.

Remember, RAID is used to protect against drive failure. Dual Raptors in RAID0 is just asking for trouble - you're doubling the chance of failure. They'd be much better off with dual SSDs in RAID1 - much better performance and power consumption and better protection against sudden failure.

I'm worried about your customers that "don't have time" to perform backups. What do they do when their laptop is stolen? Maybe their work and thus their data doesn't have much value.

You might not think that SSDs are ready yet, but there are plenty of people/companies that know differently.

Re:depends if you are IO bound or need storage (1)

vlm (69642) | more than 3 years ago | (#36970590)

I'm worried about your customers that "don't have time" to perform backups. What do they do when their laptop is stolen? Maybe their work and thus their data doesn't have much value.

Or they do everything, including storage, online. I'll publicly admit I don't backup my work laptop... The only reason I have it is to SSH, and I'm not backing up gigs of stuff when all I basically need is "putty", which I already have on a flash drive, and is widely available on the internet.

Customs wants to search my laptop, OK, its basically a vanilla install with putty. And a lot of empty space.

Re:depends if you are IO bound or need storage (1)

hawkinspeter (831501) | more than 3 years ago | (#36970818)

Actually, I tend to make sure that any work I do is either replicated elsewhere (checked into SVN, stored on DropBox, cloud storage etc) or is easy to re-create. HairyFeet's customer however, lost a load of work when their drive broke without warning.

I tend to think of hard drives as temporary storage - if the data is important, have it in multiple physical places.

Re:depends if you are IO bound or need storage (1)

DAldredge (2353) | more than 3 years ago | (#36970618)

OCZ SSD drives?

Is this a Slashvertisement? (0)

MikeTheGreat (34142) | more than 3 years ago | (#36969372)

I'll be honest - I didn't really RTFA that closely, in part because it just fawns over the SSDs.

Can someone tell me why this is significant? (Because it's EBay, because it's the first large-scale deployment of SSDs like this, etc, etc)?

Thanks in advance (and sorry about the clueless SSD noob posting :) )

Re:Is this a Slashvertisement? (3, Informative)

smash (1351) | more than 3 years ago | (#36969390)

essentially because SSD has far better IOPs, you need less units to get the same speed. So you can cut the size of the storage array in half. Ebay are clearly io bound rather than needing huge storage space. So for them, its a win.

For others, maybe not so much.

Re:Is this a Slashvertisement? (0)

Anonymous Coward | more than 3 years ago | (#36969450)

Seems like it's no different than the story about AOL from last year [slashdot.org] , just twice the storage. But I'm sure AOL has upgraded some.

Re:Is this a Slashvertisement? (2, Funny)

erroneus (253617) | more than 3 years ago | (#36969884)

I don't know, but I would be checking ebay for a butt-load of cheap 15k fiber channel drives for sale there.

Re:Is this a Slashvertisement? (0)

Anonymous Coward | more than 3 years ago | (#36970832)

I don't know, but I would be checking ebay for a butt-load of cheap 15k fiber channel drives for sale there.

Awesome. I'll have to contact the vendor to see if they'll agree to accept payment on the side via google checkout.

Re:Is this a Slashvertisement? (1)

tbannist (230135) | more than 3 years ago | (#36971006)

It's not significant at all, they're moving from FibreChannel drives which are notoriously small and expensive to SSDs. My last employer had a FibreChannel disk array. If I remember the prices, we were faced with a choice like 240 GB FibreChannel for $1000 each or 2 TB IDE drives for $100 each. It was obvious that moving to anything that wasn't FibreChannel was a good idea, because for the same price we could get either 480 GB of FibreChannel or 40 TB of IDE.

Eve Online (0)

Anonymous Coward | more than 3 years ago | (#36969396)

Eve Online went this route too for their blades.

Wonder why not 2.5" SAS drives.. (1)

RightSaidFred99 (874576) | more than 3 years ago | (#36969422)

They're expensive, but nowhere near as expensive as SSD. I guess if performance is that important, it makes sense but how many Ebay/Google/Amazon situations are there out there?

Re:Wonder why not 2.5" SAS drives.. (2)

fuzzytv (2108482) | more than 3 years ago | (#36969466)

Because you won't get the IOPs or speeds you get with SSDs? SAS driver are still the traditional drives, so the random access is a pain.

Re:Wonder why not 2.5" SAS drives.. (0)

Anonymous Coward | more than 3 years ago | (#36969492)

SAS and SSD are not mutually exclusive.

Re:Wonder why not 2.5" SAS drives.. (0)

Anonymous Coward | more than 3 years ago | (#36969494)

Don't know for sure, but enough to keep enterprise SSD providers like Nimbus and Violin and TMS in business. And still, you don't need to be big to need speed. We're very small potatoes in the scheme of things, but we have enough ram on our servers to entirely cache our important indexes. The speed is worth the money.

Re:Wonder why not 2.5" SAS drives.. (1)

beelsebob (529313) | more than 3 years ago | (#36969498)

Because you're talking about random access rates of a few hundred mess per second vs 10 mess per second if you're lucky seeking all over on a platter. The number of I/O ops an SSD can get through compared to even a fast HDD is insane.

Re:Wonder why not 2.5" SAS drives.. (1)

bertok (226922) | more than 3 years ago | (#36969522)

Because for small random I/O operations, SSDs are about 100x as fast as mechanical drives, but nowhere near 100x as expensive. This is not too dissimilar to moving from spools of tape to hard drives. Tape still has better capacity per dollar than disk, but you don't see anybody booting their PC off tape, do you?

You don't have to be a mega-corp for the speed advantage to make sense, which is why usage is increasing across the board.

For example, I recently installed a relatively small terminal server farm, and one of the performance tests I did revealed that for typical business application usage it's just not possible to utilise much more than 30% of the CPU and memory capacity of a server blade using mechanical drives!

Server OEMs like HP are already shipping blades with SSDs as a standard option, and I've seen SSD storage arrays in production usage. The time is coming very soon now when "storage" in the enterprise will mean solid state only, and spinning media will be used for the same role as tape is now: archival and backup.

Re:Wonder why not 2.5" SAS drives.. (1)

FreakyGreenLeaky (1536953) | more than 3 years ago | (#36970054)

Exactly. We use SSDs wherever performance is needed and once you (and your customers) have experienced the gains, there is simply NO going back. Even for a small accounting database server, using SSDs provides compelling gains - eg, for batch runs, instead of waiting half an hour, the wait is now something like a minute or two. The traditional HD will end up as an archive medium eventually... Seagate better get their shit together or risk becoming another SGI, SUN, whatever.

The gains far outweigh the costs.

Re:Wonder why not 2.5" SAS drives.. (1)

EraserMouseMan (847479) | more than 3 years ago | (#36970802)

Seagate, Maxtor, Western Digital, etc will fade slowly into extinction much like the 3.5" floppy disk. Heavy competition from SSDs will force these companies to consolidate together just to stay alive. Eventually nobody will want to buy a mechanical drive anymore. Mechanical HD companies will not begin manufacturing SSDs because they are not set up at all to fab silicon. Only the RAM & CPU shops are. And, surprise, look at all the SSD manufacturers they are yesterday's RAM manufacturers.

Re:Wonder why not 2.5" SAS drives.. (1)

Charcharodon (611187) | more than 3 years ago | (#36970338)

Because for small random I/O operations, SSDs are about 100x as fast as mechanical drives, but nowhere near 100x as expensive. This is not too dissimilar to moving from spools of tape to hard drives. Tape still has better capacity per dollar than disk, but you don't see anybody booting their PC off tape, do you?

Just had a flashback of my TRS-80 that I had as a kid. Pressing play on the tape drive to listen to the data so I could line up on a program to load. Cassett tapes were expensive so you put lots of programs on them and had to write down the counter number in order to fast forward to them.

Seek time could be measured in minutes if my little sister or parenst were bugging me.

Re:Wonder why not 2.5" SAS drives.. (1)

TheRaven64 (641858) | more than 3 years ago | (#36969888)

Small SAS drives have really small platters, which means that their seek time is somewhere around quarter to half the seek time of a cheap drive. This means they get 2-4 times the number of seeks per second, which means 2-4 times the number of I/O operations per second (IOOPS). For a site like eBay, where people browse fairly randomly over their entire site, the IOOPS number is the most important. A cheap SSD gets 100 times as many as a cheap hard disk. More expensive ones get 1000 times as many. Unless the cheap SSD is 25 times as expensive as the SAS drive, then it's a better choice. Even then, one SSD requires a lot less power and cooling than 25 SAS drives.

I'm slightly surprised that they didn't go with something like ZFS's L2ARC, with maybe 20TB of flash and 100TB of slower disks. I'd have thought that the majority of eBay's disk accesses would be contained within a subset of their data (most people look at new listings and ones that will expire soon), so a big flash cache would give them almost the same performance at a lower price.

Uhh.. cost? (2)

Dahamma (304068) | more than 3 years ago | (#36969434)

Of course everyone would love to replace all of their storage with SSD if price was no object.

The closest they come to mentioning cost is:

Though SSD is typically a magnitude of order more expensive than hard disk drive storage, Craft said the Nimbus arrays were "on par" with the cost of his previous storage, which a Nimbus spokesman said was from NetApp and HP 3PAR. (Craft declined to identify the vendors).

So, cost of new SSDs was similar to whatever HDDs they bought years ago? Yeah, that's kind of how it goes...

Nimbus prices its product on a per-terabyte basis - it charges $10,000 per usable terabyte

$10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

Re:Uhh.. cost? (1)

peterbye (708092) | more than 3 years ago | (#36969580)

What's 'a magnitude of order'?

Re:Uhh.. cost? (1)

sheddd (592499) | more than 3 years ago | (#36969688)

10x

Re:Uhh.. cost? (0)

Anonymous Coward | more than 3 years ago | (#36970570)

no, 10x would be an order of magnitude, not a magnitude of order!

Re:Uhh.. cost? (0)

Anonymous Coward | more than 3 years ago | (#36969744)

The opposite of an order of magnitude?

Re:Uhh.. cost? (1)

qxcv (2422318) | more than 3 years ago | (#36969866)

Singular of "several magnitudes of orders".

Re:Uhh.. cost? (5, Informative)

Anonymous Coward | more than 3 years ago | (#36969598)

Depends on your workload. (Disclosure: I work in storage for a living.)

Sometimes, what you need is raw, bulk storage. There are two serious contenders in this space: tape, and disk. You use tape if you have a lot of data you need to store, but not much that you need to access regularly: less power, and it scales to near infinite levels of storage (at the cost of very slow access for a given piece of data.) Or you use disk if you need to access most of it reasonably regularly. SSDs are not, and never will be, a contender in this space - you're paying through the nose on a per GB basis.

On the other hand, sometimes what you need is IOs per second. Database administrators are very familiar with this - you need a bit of data from over here, and a bit of data from over there, and maybe a little bit more from somewhere in the middle, and you need it five minutes ago. Traditionally, you got this performance by building a large array across many spindles, and giving up half, three quarters, or even more of your disk space, in return for that nice, fast section of disk on the outside of the platter. Lots of running hard drives, drawing lots of power, generating lots of heat, and costing a lot of money for storage space that isn't even half used - because if you throw something else on that empty space, you completely ruin the throughput of your major production database.

In that latter space, SSD is king. Sure, it's more expensive on a dollars per GB basis, but hey, guess what - GB isn't the important metric here. You figure out which bit of data is being hammered, and you move it across to the SSD. Rather like profiling an application: pick the function that takes 90% of the time in the software, optimise the wazoo out of it, and you get a significant improvement (rather than picking something at random and optimising it to billy-oh, and getting not much return for your investment.)

So yeah - SSDs aren't going to compete in raw capacity any time soon. But in random I/O performance, they make a hell of a lot of sense. In some respects, yes, they most definitely are cheaper than traditional platters of spinning rust - see the aforementioned massive RAID set across dozens of spindles.

VMs are in the IO category for sure (4, Informative)

Sycraft-fu (314770) | more than 3 years ago | (#36970246)

You discover modern hardware does virtualization real well. You get a good host software, like vSphere or something on new hardware and you have extremely near native speeds. The CPUs handle almost everything just like it was running as the host OS, and sharing the CPU resources works great. Memory is likewise real good, in fact VMs only use what they need at the time so they can have a higher memory limit collectively than the system RAM and share, so long as they don't all try to use it all at once.

You really do have a situation where you can divide down a system pretty evenly and lose nothing. Let's say you had an app that used 2 cores to the max all the time and 3GB of RAM. You'd find that it would run more or less just as well on VM server with 4 cores and 8GB of RAM, half assigned to each of two VMs, as it would on two 2 core 4GB RAM boxes. ...Right up until you talk storage, then everything falls down. You have two VMs heavily access one regular magnetic drive at the same time and performance doesn't drop in half, it goes way below that. The drive is not good at random access and that is what it gets with two VMs hitting it at the same time, even if their individual operations are rather sequential.

It is a bottleneck that can really keep things from scaling like the other hardware can handle.

At work I use VMs to maintain images for instructional labs (since they all have different, custom requirements). When I'm doing install work on multiple labs, I always do it sequentially. I have plenty of CPU, a hyper-threaded 4 core i7, plenty of RAM, 8GB, there's no reason I can't load up multiples. However since they all live on the same magnetic disk, it is slower to do the installs in parallel than sequential.

If I had an SSD, I'd fire up probably 3-4 at once and have them all do installs at the same time, as it would be faster.

Re:VMs are in the IO category for sure (1)

complete loony (663508) | more than 3 years ago | (#36970800)

For running on HD's I think there should be one OS with one picture of the filesystem. So that caching, prioritisation and request reordering can all occur in one place, with as much information about the immediate future as possible. Each VM should then import a section of that filesystem. But if you're talking about databases, they tend to replace most of the filesystem level caching anyway, so you'd still be better off with SSD's and / or dedicated storage.

Re:Uhh.. cost? (0)

Anonymous Coward | more than 3 years ago | (#36970346)

SSDs are not, and never will be, a contender in this space

Bullpoop, of course they will be - eventually.

Re:Uhh.. cost? (1)

bareman (60518) | more than 3 years ago | (#36970474)

If I had a mod point, I'd mod this response upwards.

The parent to it was considering only capacity. And from a capacity standpoint, no SSD doesn't make any sense and is about 10x the price. But for performance SSD is king and yes, cheaper than the spinning rust.

Re:Uhh.. cost? (2)

LordLimecat (1103839) | more than 3 years ago | (#36970988)

You figure out which bit of data is being hammered, and you move it across to the SSD. Rather like profiling an application: pick the function that takes 90% of the time in the software, optimise the wazoo out of it, and you get a significant improvement (rather than picking something at random and optimising it to billy-oh, and getting not much return for your investment.)

Or you do what eBay is apparently doing and say, screw it, we're doing 5 blades, and throw all of your storage on SSD,

Re:Uhh.. cost? (1)

SecurityTheatre (2427858) | more than 3 years ago | (#36969612)

Keep in mind, they're likely running parallel RAID configs to maximize throughput, meaning something like RAID 1 or RAID 5-1 or some other proprietary method that may provide only 1/3 or 1/5 the usable capacity in order to get the throughput required.

In fact, since these drives can theoretically be about 50x faster in random-seek reading of data, you could actually say that per the throughput AND size, they're actually cheaper than a comparably capable array of spinning discs.

Re:Uhh.. cost? (1)

smash (1351) | more than 3 years ago | (#36969620)

Again, depends if you need bulk storage, or fast i/o. If, to get the IO throughput required, you need to purchase far more hard drive spindles than you otherwise would need for the capacity required, then the total amount of storage you get being less with SSD may not be an issue (so long as it is "enough"). e.g. (fake numbers), if you need say 100k iops and 1tb of space, this could perhaps be done with 10 magnetic disks, or 2 SSDs. the additional space provided by the magnetic disk is of no use if your only reason for purchasing multiple disks is to gain IOPs.

Many big corps like ebay are likely far more concerned with gaining higher throughput than more storage space.

Re:Uhh.. cost? (1)

TubeSteak (669689) | more than 3 years ago | (#36969674)

$10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

There will always be applications where the high dollar [latest and greatest] solution will provide *vastly* better performance than the current 'standard'.

If you're replacing 8 or 9 TB of 15K drives with 1 TB of SSDs, then the accounting becomes a bit more manageable.
And raw price:performance doesn't always tell the whole story, not when paying more can save in other ways.

Re:Uhh.. cost? (0)

Anonymous Coward | more than 3 years ago | (#36969702)

If it is 10x more expensive, but 100 times faster, as long as you are IO bound, not space bound; then it make sense.

Re:Uhh.. cost? (0)

Anonymous Coward | more than 3 years ago | (#36969846)

$10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

Are you smoking crack? You can get 120gb SSD drives for $160-$260 these days, that comes out to around $1333-$2300 per terabyte. You also have to take into account that one SSD is faster than most RAID setups. It is also much cheaper to run SSDs in database servers than it is to use regular hard drives in a RAID array. Not to mention the savings in power use. Tom's Hardware has a good writeup on this.

http://www.tomshardware.com/reviews/ssd-reliability-failure-rate,2923.html

Re:Uhh.. cost? (1)

Charcharodon (611187) | more than 3 years ago | (#36970350)

1 TB can be had for about $1,675 these days. Take a look at OCZ SSDs on new egg. The price is falling nicely, though I doubt it'll catch spinners in the next 10 years for cheapness.

Re:Uhh.. cost? (1)

LordLimecat (1103839) | more than 3 years ago | (#36971024)

What are the failure rates? That seems rather relevant when youre dealing with massive RAID arrays.

Compact (2)

drmofe (523606) | more than 3 years ago | (#36969454)

So the entire eBay VM operation could fit into 6 racks? 200 physical servers @ 1RU each = 5 racks 10x 10TB 2U SSDs = half a rack 5x 2U switches = quarter rack

Zero details (3, Informative)

billcopc (196330) | more than 3 years ago | (#36969458)

TFA reads like a thinly-veiled promo for Nimbus Data Systems, which I can only guess are pushing a Linux-based SAN appliance full of SSDs. Big whoop.

What I would love to know is: Why does eBay need 4000 VMs ?

Re:Zero details (2)

Konster (252488) | more than 3 years ago | (#36969480)

They need 4000 VM's so they can buy 100TB of SSDs. :)

Re:Zero details (0)

Anonymous Coward | more than 3 years ago | (#36969504)

they need 8000 VMs to make their overdynamic site fast enough. They have only 4000 because they are skimpy

Re:Zero details (3, Funny)

Monoman (8745) | more than 3 years ago | (#36969892)

"We're not a storage team. We're Windows administrators who got into virtualization ... "

Re:Zero details (0)

Anonymous Coward | more than 3 years ago | (#36970008)

Because it's the only way to handle that many simultaneous connections under windows, duh.

Re:Zero details (0)

Anonymous Coward | more than 3 years ago | (#36970154)

What I would love to know is: Why does eBay need 4000 VMs ?

I used to work for a fairly small ISP. We had 400 VMs on VMWare, because we had to provide all the services an ISP provides: email in, email out, access to mailboxes, DNS caches, DNS resolvers, web servers, monitoring servers, RADIUS servers, LDAP servers, stuff for internal use, shell servers of various flavours, VoIP related servers, software load balancers...the list is a long one. Oh and double everything for redundancy and make some of clusters of up to 10 VMs for capacity.

When you're someone like eBay, you basically are your own ISP, and you still need to provide all those services to yourself. So I can easily imagine that when you're the size of eBay, having ten times that many seems fairly sensible.

Which kind of SSDs they've used? (1)

fuzzytv (2108482) | more than 3 years ago | (#36969482)

The article does not mention which kind of SSDs they've used, or have I missed something? That might be very interesting, especially when it comes to reliability. It's often claimed the SSDs are more reliable than traditional drives, but accrding to this http://www.tomshardware.com/reviews/ssd-reliability-failure-rate,2923.html [tomshardware.com] that's not really true.

Re:Which kind of SSDs they've used? (0)

Anonymous Coward | more than 3 years ago | (#36969526)

Reliable has several different meanings.
Reliable does need to mean that it doesn't fail. It could just mean that it fails in a more predictable manner.

Re:Which kind of SSDs they've used? (1)

justforgetme (1814588) | more than 3 years ago | (#36969530)

As far as I understand they used storage arrays from nimbus data [nimbusdata.com]

Re:Which kind of SSDs they've used? (1)

fuzzytv (2108482) | more than 3 years ago | (#36969588)

Yes, but Nimbus is not a SSD manufacturer. I'd expect them to buy a lot of SSDs and then build the storage arrays from them.

Re:Which kind of SSDs they've used? (1)

_Shad0w_ (127912) | more than 3 years ago | (#36969732)

I wouldn't be at all surprised if they just buy in bulk from a manufacturer and re-brand them, like Dell do with their drives. The last lot of Dell 2.5" SAS disks I looked at were just Seagate drives with a Dell sticker on them instead of the Seagate one (they didn't even change the model number).

Re:Which kind of SSDs they've used? (0)

Anonymous Coward | more than 3 years ago | (#36970146)

Did you even read that article?

"Fewer devices installed means fewer devices to fail, too. Since you use one solid-state drive to replace multiple hard drives, consolidation ends up benefiting the business adopting flash-based storage. If the swap were a 1:1 ratio, that argument wouldn't work. But at 1:4 or more, you're really cutting into the number of disks that would eventually fail, and we can't let that point be under-emphasized."

Article Heading :-? (1)

sonamchauhan (587356) | more than 3 years ago | (#36969486)

I got the impression ebay just terminated a hosting arrangement with Rackspace (the company) -- bringing it inhouse, and cutting Rackspace's revenues in half. :)

Re:Article Heading :-? (1)

Wizarth (785742) | more than 3 years ago | (#36969636)

That is also what I read!

deployment time nine times better? (1)

rbrausse (1319883) | more than 3 years ago | (#36969490)

has improved the time it takes to deploy a VM from 45 minutes to 5 minutes

uh, any logical explanation for this? SSDs are snappier and the peak I/O can be faster compared to spindle drives - but not by factor 9, or?

Re:deployment time nine times better? (1)

rtb61 (674572) | more than 3 years ago | (#36969540)

At a guess comparing unformatted hard disk drives to SSD, sneaky.

Re:deployment time nine times better? (1)

vadim_t (324782) | more than 3 years ago | (#36969626)

Yes, in the real world, data isn't laid out on the disk in the exact order it's going to be read. Especially when filesystem structures are involved.

Seeks are very expensive. I measured a desktop drive barely doing 300 KB/s when doing random reads. That's of course not very realistic either, but the real world performance is going to be somewhere in between that and the ideal contiguous read speed. Having a disk capable of 100MB/s managing only 5MB/s is quite possible.

I bet Rackspace did not like it.. (0)

Anonymous Coward | more than 3 years ago | (#36969520)

The cut Rackspace by half - Did they like... auction off half of that company?

Reliability? (2)

Vectormatic (1759674) | more than 3 years ago | (#36969660)

I read a blog-post a while back stating that SSDs fail a lot more then you would expect. Somewhere around a year of heavy use seems to take most of the life out of a consumer grade ssd. Now i wonder how putting SSDs into Raid 5 (or 6, or whatever) will behave. If a certain model of SSD croaks around X write ops, then i think the nature of Raid will mean that your entire array of SSDs will go bad pretty closely together. It must suck to have two more drives go belly up while rebuilding your array after the first drive failure.

Perhaps it would make sense to stagger SSDs in different phases of their lifetime to keep simultanious failures at bay, use some burned in drives and some fresh ones.

Re:Reliability? (1)

Terrasque (796014) | more than 3 years ago | (#36969724)

That's okay. I found a blog written by someone from the future, so I sent him a request to fetch some SSD durability data from 15 years into the future.

I expect a reply any minute now.

Re:Reliability? (1)

angryphase (766302) | more than 3 years ago | (#36969800)

I read on the The Register [theregister.co.uk] that in fact these are far from consumer grade SSD implementations. Instead they are using specific set of flash module configurations:

The system uses 24 flash modules, not standard off-the-shelf solid state drives (SSDs), with different flash densities per module to provide the 2.5, 5 and 10TB capacity points.

Re:Reliability? (1)

MemoryDragon (544441) | more than 3 years ago | (#36969958)

It really depends on the write cycles, the problem there is that you have about 3000 write cycles on a modern consumer grade ssd, combine that with the fact that most people buy really small ones to save some bucks and they start to use it heavily by swapping on them etc... and you have a situation where you very likely can reach those 3000s within a year or two.
If you use them in a sane manner and have a decent size so that wear levelling can do its magic you should not hit that limit within the lifespan of the computer itself.
But that is just consumer grade ssds, enterprise grade ssds are an entirely different game.

Re:Reliability? (1)

Vectormatic (1759674) | more than 3 years ago | (#36970044)

i fully realize that consumer grade SSD != enterprise SSD, but a tighter grouping of failures is still very likely due to the way SSDs work. I for one am curious to see what effect this will have on RAID strategies in the future. As for usage, people using their SSD for swap are either stupid, or like the speedup that much that they are willing to eat the cost of an SSD over a years time, but in enterprise land those drives can see very heavy use as well, especially when logging/databases are involved.

Also, shortly after posting the root post, i realized that theoretically rebuilding a raid array should only write heavily to the new disk and only read from the remaining parts of the old array, so triggering a second failure by rebuilding isnt that likely.

Re:Reliability? (1)

bug_hunter (32923) | more than 3 years ago | (#36970174)

Well they are just replacing their VM servers, the databases are possibly elsewhere on the network so the writes to the SSD in that scenario should only occur when they update a VM. (Just guessing though).
Still, I take your point of a series of SSDs used for the same purpose are more likely to fail around the same time than ye olde HDDs.

Re:Reliability? (1)

vlm (69642) | more than 3 years ago | (#36970686)

Perhaps it would make sense to stagger SSDs in different phases of their lifetime to keep simultanious failures at bay, use some burned in drives and some fresh ones.

Trust me, as I guy who's run raid arrays of spinning rust for well over a decade, you REALLY need to do that with old fashioned drives too.

Worst experience in the world is having a RAID-1 with two consecutive serial number drives and both bearings let go at the same time.

Performance or Price? (1)

mjwx (966435) | more than 3 years ago | (#36969668)

'One rack [of SSD storage] is equal to eight or nine racks of something else,' said Michael Craft, eBay's manager of QA Systems Administration."

Is he talking about performance or price. I can imagine that a single rack of enterprise SSD's could easily cost the same as 9 racks of anything else.

Re:Performance or Price? (1)

FreakyGreenLeaky (1536953) | more than 3 years ago | (#36970066)

Performance.

huh (0)

Anonymous Coward | more than 3 years ago | (#36969684)

so a drive with 1tb space is equal to 9 tb of anything else

BULLSH%T

Re:huh (2)

smash (1351) | more than 3 years ago | (#36970030)

More like, the I/Os per second of 1 rack of SSD has 9x the THROUGHPUT of 1 rack of magnetic media. remember, we're talking about massive arrays here to get speed, not necessarily for disk capacity. If you only need the capacity provided by 1 rack of SSD, then being able to cut your rack space in half or less by getting the required IO in less drives can save heaps. Potentially, it can save you needing to build a new datacenter....which ain't cheap.

Re:huh (1)

Neil Boekend (1854906) | more than 3 years ago | (#36970134)

It's about speed, not capacity. They used to have 15K disks, which have a latency of around 2 ms. They switched to SSD's, which have a latency of around 0.1 ms.
Why does latency matter for them? They have large amounts of relatively small files (small images, item descriptions and so forth). These files are spread across the disks. Each file requested means the latency is counted. If a webpage needs 100 files the latency of the disk is multiplied by 100. Each user (and there are many at any given moment) needs 100 files. As soon as possible.
The used to solve this by having far to many disks. The amount of storage space may have been twice what they expected to need, because more disks means you can acces more files at once, effectively cutting your acces time. Now file 2 doesn't have to wait until file 1 is done, because it's on another disk (or actually RAID). This increases the speed, but at a cost.
Add to that the fact that SSD's have insane transfer speeds. An 15K rpm disk may transfer 300 MB/s, an simple SSD can transfer 550. Theoretically they can go much faster, but SATA can't keep up. These systems don't use sata, so their transfer speeds may be much higher.
IANAEOTS, so correct me if I am wrong.

Return On Investment (4, Insightful)

ritcereal (1399801) | more than 3 years ago | (#36969722)

While most people instantly gravitate towards the upfront cost and performance of going solid state, I would make one important point. Reducing your data center space by 9 racks is significant in terms of power, cooling and that is all on top of the purchase price and support contracts. Regardless if ebay owns their own data center or colocates, the cost per square foot in a data center and the continued operation of such a large storage system is more then likely to provide a higher return on investment. eBay isn't in the business of looking cool and hip, they're in the business of selling stuff as cheaply as possible and I'm certain their CIO cares only about the bottom line.

Re:Return On Investment (1)

darkmeridian (119044) | more than 3 years ago | (#36969988)

And don't forget that the SSD solution uses less electricity. That means you're not only saving on the electricity, but you're also spending less money removing the heat associated with using electricity. Being able to rebuild a server in five minutes as opposed to forty five minutes probably means they can keep more backup servers slack and turn them on only in case of emergency. Since SSD use less electricity on standby, they probably can keep the SSD idling. And if it's so much smaller, you can pack more of them in the same space, which makes your facilities more future-compatible; you don't have to get new space to expand.

Yes! (1)

ThePhilips (752041) | more than 3 years ago | (#36970092)

Finally we are getting a chance of seeing real reliability stats of the SSD!

That if eBay would be kind enough to publish the data couple of years later.

Re:Yes! (1)

crashumbc (1221174) | more than 3 years ago | (#36970488)

This is first thing that came to my mind...

If the SSD's only last half as long you may not be getting the ROI your looking for.

Re:Yes! (1)

ThePhilips (752041) | more than 3 years ago | (#36970840)

I do not care about ROI of eBay - I want to see how SSD work in real-life compared to HDD.

As one never gets any meaningful information from vendors, the only hope is the feedback of the users. The user in the case is the eBay.

reliability? (1)

ushere (1015833) | more than 3 years ago | (#36970220)

i was under the (misguided?) impression that ssd's weren't, as yet, enterprise ready in terms of reliability?

Re:reliability? (1)

vlm (69642) | more than 3 years ago | (#36970646)

i was under the (misguided?) impression that ssd's weren't, as yet, enterprise ready in terms of reliability?

People rationalize what they want. Can you believe there are people that claim Windows is enterprise ready, merely because they want to / have to use it?

An indication of mistuning? (0)

Anonymous Coward | more than 3 years ago | (#36970266)

It might also might indicate that ebay has done mistuning of their environment.
Some idiots still think that for a web server you need to take the amount of total memory and divide it by the amount of ram one apache process takes. And then setting this value as the amount of apache tasks. Thus leaving too little 'empty' for filesystem caching. Sadly enough that mistuning advise was for years on the apache webserver faq.
Having enough memory available for filesystem caching is the number 1 priority to reduce IO operations. Only with large database servers which have completely random access, this might not help. And with web applications such as ebay it's rarely ever completely random.

But hey, it's ebay, and that they have a crappy infrastructure has been visible for years... running slowaris for a decade for SSL websites and such... absurd!
Any decent infrastructure should serve most of it's pages without any IO operation taking place at all. Just like Facebook and Google do. Disk is merely intended for persistant storage, but most gets served out of filesystem cache. But hey, the fact that they rely on 'virtual machines' already says enough about their level of intelligence...

SSD is however great for IOPs, as others indicate; especially oracle tends to do a lot of unpredicatble random IO which really gets a boots with SSDs... practical for small environments...
But I wonder what the ebay investment cost was, and what the performance increase would have been if they had invested that into RAM instead (without re-mistuning).

Greetings,
Jasper

They've got plenty of money, no wonder (1)

Kartu (1490911) | more than 3 years ago | (#36970440)

Took me a while to figure how much do they actually cut from the end price, it's not something they're pleased to tell you.
Especially with recent "adjustment of prices" at ebay, no wonder they have extra cache to waste on whatever idiotic idea comes to their IT management.
They get 8%+ of most below 500$/Euro items sold. Outrageous.

This is just great news! (1)

Max_W (812974) | more than 3 years ago | (#36970462)

I hope my hosting company will change to such SSDs, and I will not have to wake up those rotating disks early in the morning.

They should clean up their HTML first.... (0)

Anonymous Coward | more than 3 years ago | (#36970496)

Have you seen the mess that is Ebay's HTML and CSS? They could reduce the size of the average page by 50% probably, if they hired somebody who actually knew how to code properly.

Forget it (1)

alphatel (1450715) | more than 3 years ago | (#36970576)

I call total BS on this post for a few reasons
  1. 100 TB is about 250 SSD drives. Ebay runs a few hundred peta of storage over several datacenters throughout the US [dbms2.com] , not including all other countries.
  2. We run several racks at Telx. Our cost per month in NYC including electric is only about 3k per cabinet. Does Ebay really care about 3 cabinets?
  3. Over half the systems now access the shared storage which contains the drives? Yes, if I map a drive to a particular SAN I guess I am now accessing the data. Does that mean I am actually leveraging it?

All in all this is barely a dent in anything Ebay does. It sounds more like an experiment and hype of the drives they used.

only for reads. (1)

unity100 (970058) | more than 3 years ago | (#36970696)

SSD life is limited with number of write operations. you cant use them like normal disks in the business ebay is using.

but read operations are unlimited. so, if you are going to just read files from a hard disk, ssd makes the perfect candidate. in random reads, they are approx 40 times faster than best hdd at the minimum.

so, you just put 250 ssd disks, put your VM images on it, and, as the article says, it boosts your vm deployment time to other systems from 45 minutes to 5 minutes - there is nothing wrong with the article.

SSDs are still unreliable (1)

unity100 (970058) | more than 3 years ago | (#36970672)

even the most touted and expensive 'enterprise ssd' can die out on you unexpectedly.

Re:SSDs are still unreliable (1)

KillaGouge (973562) | more than 3 years ago | (#36970784)

As can the most touted and expensive 'enterprise hdd'

Recycle! (1)

trum4n (982031) | more than 3 years ago | (#36970892)

I want the old hard drives! Please?
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?