Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Mega-Uploads: The Cloud's Unspoken Hurdle

samzenpus posted more than 2 years ago | from the mountain-of-disks dept.

Cloud 134

First time accepted submitter n7ytd writes "The Register has a piece today about overcoming one of the biggest challenges to migrating to cloud-based storage: how to get all that data onto the service provider's disks. With all of the enterprisey interweb solutions available, the oldest answer is still the right one: ship them your disks. Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"

cancel ×

134 comments

Sorry! There are no comments related to the filter you selected.

Pro photography is a huge problem (5, Interesting)

Anonymous Coward | more than 2 years ago | (#40069091)

Returning from a site with a tethered computer full of 80 MP 16-bit raw files from a day's worth of shooting would break most bandwidth bills if you tried uploading all these images.

Re:Pro photography is a huge problem (1, Informative)

Anonymous Coward | more than 2 years ago | (#40069579)

Not really.
As a consumer I get 1Gbps/100Mbps for roughly 130 USD.
If I settle for 100/100, we're talking 50 USD.
250/100 is around 70 USD.

Per month.

Of course, this is consumer gear, so I'm only *guaranteed* 60% of the upstream.

Then again, if you're doing pro photography at 80MP, odds are you're doing it as a business, and should have little to no problem forking out the ~$800 a month a gigabit pipe would run you.

Oh wait, you live in the land of the brave and free, home of the 512Kbps broadband?
Sucks to be you.

Re:Pro photography is a huge problem (4, Interesting)

HapSlappy_2222 (1089149) | more than 2 years ago | (#40070103)

Just because you take pro pictures at 80MP doesn't mean your business has an extra $10,000 per year laying around for a business grade gigabit pipe; as the sole employee of my company, that'd mean I'm paying myself $20,000 a year instead of $30,000 (TBH, I'm lucky to be in the black, period, with only two years in). I store my images at my studio, back them up daily to a removable disk, and bring in a 3rd removable to copy them over once per week. All told? $500 for the 2 drives and a striped array on my studio PC. The self-storage backup technique works well for me.

Realistically, though, if I want to I can just upload them all to home or a cloud storage in batches overnight, the same way I download 10 gigabyte files at home. It's just plain easier to cart em around, though.

Re:Pro photography is a huge problem (0)

Anonymous Coward | more than 2 years ago | (#40073493)

has an extra $10,000 per year laying around

Lying around.

Hens lay eggs, money just lies around. Unfortunately.

Re:Pro photography is a huge problem (0)

Anonymous Coward | more than 2 years ago | (#40070183)

Please list 'where' in the US you can get such a gigabit internet line for $800?

Re:Pro photography is a huge problem (4, Informative)

rthille (8526) | more than 2 years ago | (#40070367)

The tiny town of Sebastopol CA, population ~7800 has gigabit fiber to the (some) doorstep for $69/month.

Re:Pro photography is a huge problem (1)

Gothmolly (148874) | more than 2 years ago | (#40071947)

And all the socialized deficit spending you can eat.

Re:Pro photography is a huge problem (1)

WOOFYGOOFY (1334993) | more than 2 years ago | (#40072001)

These two things are only related if said tiny town in Cali is subsidizing said connection. Did I miss that this is the case?

Re:Pro photography is a huge problem (2)

rthille (8526) | more than 2 years ago | (#40072445)

Um, no. Sonic.net is the business doing the rollout. Basically, they pay for it by getting 100% adoption (by eating the other services' lunches).

Re:Pro photography is a huge problem (1)

jedidiah (1196) | more than 2 years ago | (#40072115)

That covers about 40 people.

Now how about the rest of us?

Re:Pro photography is a huge problem (1)

rthille (8526) | more than 2 years ago | (#40072451)

Yeah, and I'm across town. Fuckers!
I'm not even in the "planned roll out area" :-(

Re:Pro photography is a huge problem (0)

Anonymous Coward | more than 2 years ago | (#40069957)

Shame on you for giving a real example.

The clowd clowns crowd us with their crappy (0)

Anonymous Coward | more than 2 years ago | (#40071463)

__ and expect us to like it and pay for the bandwidth.

For what amazon would charge us for 20 terabytes of clowd storage feeding 12 websites running on their clowd we built our own low energy datacenter with regional redundancy. We saved about 60 percent in annual costs so this pays for itself it 3 years.

Just Torrent (-1)

Jeremiah Cornelius (137) | more than 2 years ago | (#40069097)

The lot! Petabit transfers on tap.

Trickle up...

Re:Just Torrent (-1)

Anonymous Coward | more than 2 years ago | (#40069317)

That made no sense.

Re:Just Torrent (1)

Jeng (926980) | more than 2 years ago | (#40069401)

You might get one hell of a download rate from torrents, but you don't get a better upload rate.

Re:Just Torrent (1)

Jeremiah Cornelius (137) | more than 2 years ago | (#40069505)

I'll send you a growl notification when it finishes...

Re:Just Torrent (1, Funny)

Jeremiah Cornelius (137) | more than 2 years ago | (#40069823)

> You might get one hell of a download rate from
> torrents, but you don't get a better upload rate.

You, sir, are unfamiliar with how things work here, in Soviet Russia!

Re:Just Torrent (1)

Jeng (926980) | more than 2 years ago | (#40070007)

I do know at least how things work on /.

Natalie_Portman_and_hot_grits.torrent

The real hurdle (0)

Anonymous Coward | more than 2 years ago | (#40069131)

Getting around all the buzzwords

Re:The real hurdle (4, Insightful)

icebike (68054) | more than 2 years ago | (#40069213)

Getting around all the buzzwords

Well that's one hurdle.
The next is RECOVERY when ICE or FBI or some other 3letter agency walks in an takes your data because one tiny customer use the service for some allegedly nefarious purpose.

The key here is to use a service so big that even god himself would not dare take it down, although the Ayatollah might try. Small cloud services, even if multi-homed are a risky proposition. Even if you do manage to get all your data into them, they are not large enough to push back against any subpoena or search warrant that any misguided judge in some backwater jurisdiction may issue.

Re:The real hurdle (1)

Penguinisto (415985) | more than 2 years ago | (#40069899)

This, right here.

Unless that cloud service also comes with a guarantee* that the physical disks they park your data on are separate and distinct from anyone else's (and that no two customers share the same disks), you're just as borked as the perp when someone shows up at the colo with a warrant.

* Good luck with that. It would either cost you a mint, or you'd at best get your own LUN, which means approximately bupkis.

Re:The real hurdle (0)

Anonymous Coward | more than 2 years ago | (#40071531)

This, right there.

No, that.

Re:The real hurdle (5, Informative)

CAIMLAS (41445) | more than 2 years ago | (#40071149)

That is just one of many of the hurdles.

Really, these problems are problems because most 'cloud' shit is done wrong.

It's a bit of a worn out record here on Slashdot, but anyone or any company which is fully dependent upon The Cloud for business continuity is a fool.

* First off, there is no such thing as 'utility computing', and probably never will be due to the volatile nature of storage and its ongoing cost of maintenance.
* Second, if you do not maintain primary physical control of something, to the best of your ability, you do not control it.
* For primary IT infrastructure, it will cost more to do "Cloud" than local. If you can afford 2-3 servers a year, but not much more, and a nominal IT operations budget, chances are you should have an in-house "cloud" with off-site replication.
* Bandwidth costs both ways will kill you, as will latency in many cases, will kill Cloud functionality.

At this point, I still strongly recommend against public Clouding your systems unless they are:

a) (very!) low volume with use-based billing. This only makes sense for a low-volume public-facing site where you don't already have IT infrastructure (on a cost basis)
b) off-site 'hot' replication. You've got your inside 'private Cloud' which replicates to off-site systems. (Cloud is basically just colocated virtualization, after all.)
c) Other geographic/distribution requirements (eg. multisite organization with none serving as a good central hub). In this case, colocation of your own equipment makes more sense in many regards.

why dodge this question? (2)

Eponymous Hero (2090636) | more than 2 years ago | (#40069139)

Pressed if disks are accepted, the company responded that “All common database products provide a capability to extract to a common file format like .csv.”

what a professional answer. and by that i mean it didn't answer the question at all.

Re:why dodge this question? (3, Insightful)

Bogtha (906264) | more than 2 years ago | (#40069803)

You don't know the exact dialogue between the journalist and the rep. I've been quoted in print in similarly stupid ways when what I said made absolute sense in context to what was asked. "Pressed if disks are accepted" could have been something like the rep telling them about a new CSV import tool they had built, the journalist saying "So if I mailed you a 5TB database on a disk, could you import that?", and the rep replying "Sure, but you'd need to export the data first...".

Re:why dodge this question? (1)

filthpickle (1199927) | more than 2 years ago | (#40069815)

Didn't he? I recognize (because I have to do this too) a bit of the old 'no, you can't do that..or we have a workaround for it..and I will obliquely say that, but not come right out and do so....because you haven't signed a contract yet' (I admit to not having RTFA).

Station Wagon Full of Tapes (5, Funny)

viking099 (70446) | more than 2 years ago | (#40069177)

Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"

Yeah, the bandwidth is great, but the latency SUCKS.

Re:Station Wagon Full of Tapes (3, Funny)

MrP- (45616) | more than 2 years ago | (#40069217)

A simple solution would be to create station wagons with FTL engines.

Re:Station Wagon Full of Tapes (1)

Gunnut1124 (961311) | more than 2 years ago | (#40069689)

Or add more station wagons.

Re:Station Wagon Full of Tapes (1)

Imrik (148191) | more than 2 years ago | (#40070679)

That improves the already good bandwidth but doesn't do anything for the latency.

Re:Station Wagon Full of Tapes (1)

Gunnut1124 (961311) | more than 2 years ago | (#40070819)

It reduces the wait times for requests. More station wagons means that the average transmit/receive latency more closely represents the actual link latency.

I propose that we develop a circuit of tape drive depots along an interstate to offer proof-of-concept.

Re:Station Wagon Full of Tapes (1)

Eponymous Hero (2090636) | more than 2 years ago | (#40070777)

and only one-way, multiple lane roads. you know, for parallel processing.

Re:Station Wagon Full of Tapes (2)

Archangel Michael (180766) | more than 2 years ago | (#40069879)

Or at least a JATO unit

Better solution. (1)

www.sorehands.com (142825) | more than 2 years ago | (#40071787)

Add a flux capacitor to the station wagons.

What is your transmission speed when the transmission is completed in -1 hour?

Re:Better solution. (1)

DarwinSurvivor (1752106) | more than 2 years ago | (#40072145)

Well, mathematically you would actually have a negative bandwidth.

Re:Station Wagon Full of Tapes (0)

Anonymous Coward | more than 2 years ago | (#40069315)

But at least you get to start jamming "Headin to the highway BA-DA-DA-DA-DA-DA-DA Lookin for adventure BA-DA-DA-DA-DA-DA-DA"

Re:Station Wagon Full of Tapes (2)

ffejie (779512) | more than 2 years ago | (#40069613)

I love this thinking. There was a thread about this some time back that I found most enjoyable, despite my shoddy math.

A dumptruck full of harddrives. [slashdot.org]

Re:Station Wagon Full of Tapes (0)

Anonymous Coward | more than 2 years ago | (#40070677)

Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"

Yeah, the bandwidth is great, but the latency SUCKS.

Not necessarily.

If you have to transfer 10 TB of data, the latency of uploading it via even a 100 Mb/s connection will take a while. It may be quicker to send it to a NAS box with a box of machine RAIDed together (1-10 Gb/ps over the local LAN), and then ship the entire NAS overnight to the place that needs it.

Re:Station Wagon Full of Tapes (1)

nine-times (778537) | more than 2 years ago | (#40071043)

True, but that's not even the real problem. The idea of creating an "initial seed" is a good one, but it's only dealing with one sign of a larger problem: our internet connections are often not good enough to deal with the volume of data we're pushing, and so cloud storage solutions can only serve certain cases.

Storing files on a remote system with limited bandwidth is only good when you're dealing with generally small files. A limited number of large files can be fine so long as you're primarily syncing one-way and aren't changing the large files very frequently. Sending the files in a disk overcomes one single problem, which is the initial upload, but that's only the main problem when you're doing a one-way sync of a lot of data that isn't changing very often. If all the data is changing all the time, then the initial upload isn't going to be much smaller than the subsequent syncs. If you're planning on accessing the data, then you either need a syncing solution, or you'll be dealing with a lot of downloads. If you sync data, then there are a lot of additional problems, including conflict resolution. If you're changing moderately large files that take a while to transfer, then conflict resolution is going to get much messier.

Also, the reality of these services is it's not too unusual for them to fall out of sync, or for one of the copies to become corrupted. To make matters worse, they're usually not very good about rebuilding, which means you're back to sending disks back and forth to get things sorted out.

The unfortunate truth is, for a lot of people, cloud storage just isn't there yet. The biggest hurdle is that too many people are stuck on crappy Internet connections with slow upload rates. IMHO, I'll wait until transfer rates are around where internal networks were a decade ago, which means 100 Mbps symmetrical connections. Of course, that's not happening anytime soon, especially since the ISPs and media companies that they're partnered with have no interest in giving people decent upload rates.

Even when people have decent upload rates, there still aren't sufficiently good/open syncing protocols. Ideally you want something similar to rsync, but with robust 2-way syncing, including bullet-proof conflict resolution, and automatically and seamlessly encrypts the remote copy. Good luck with that. Given that tons of people are still using unencrypted FTP, CAs for HTTPs, and IPv4, I don't have a lot of faith in the adoption of new standards/protocols.

What I wouldn't give for some good programmers and industry clout.

Re:Station Wagon Full of Tapes (1)

jedidiah (1196) | more than 2 years ago | (#40072135)

No. The cloud is really irrelevant.

Either I can't push it out there and thus can't use it.

Or, I can push it out there but it is irrelevant because I could just host my own stuff to begin with.

I either can't use it or don't need it.

Re:Station Wagon Full of Tapes (0)

Anonymous Coward | more than 2 years ago | (#40071347)

Yeah, but you only have to send one packet via truck.

Re:Station Wagon Full of Tapes (0)

Anonymous Coward | more than 2 years ago | (#40071495)

On hearing that all I could think of was this scene from Revenge of the Nerds--

http://www.youtube.com/watch?feature=player_detailpage&v=l5iA0GapN7Q#t=143s

"I've got the ol' cruise control set at 35" with a cut away to them on what is clearly an interstate with cars flying by them...

Second biggest challenge (4, Insightful)

WOOFYGOOFY (1334993) | more than 2 years ago | (#40069181)

Second biggest challenge: trusting any of these places have the motivation to keep your data more secure than credit card companies do.

Re:Second biggest challenge (5, Insightful)

gestalt_n_pepper (991155) | more than 2 years ago | (#40069297)

No, that's third. The second biggest challenge is believing that those fine hosting companies with servers hosted in lower Slobbovia won't have a few entrepreneurial employees who will *actively* be searching your data for all that is monetizable.

Re:Second biggest challenge (1)

WOOFYGOOFY (1334993) | more than 2 years ago | (#40069449)

A special case of my number two perhaps.

I like your sig.

Re:Second biggest challenge (2)

betterunixthanunix (980855) | more than 2 years ago | (#40069977)

I thought the second biggest challenge is ensuring that the Empire does not raid the hosting company and render all your files inaccessible...

Re:Second biggest challenge (0)

Anonymous Coward | more than 2 years ago | (#40070353)

Safer there than hosted in the US though

Re:Second biggest challenge (1)

PPH (736903) | more than 2 years ago | (#40072041)

It has already been searched by Customs on its way out of the country (or the NSA if you pushed it out over the 'Net). And if it is of any interest to one of your competitors who happen to be buddies of the DHS, they'll be the ones contracted to do a 'threat analysis' on it.

Exit strategy (3, Insightful)

scsirob (246572) | more than 2 years ago | (#40069507)

No, the second biggest challenge is to come up with a viable exit strategy. Once you have several TB at this service provider, how will you move it out of there when the next provider has a better deal? That was one of the major big points for having a cloud in the first place, to have the freedom to move your compute requirements to a better, cheaper, faster (pick two) provider.

Even if you moved it in with a station wagon full of tapes or disks and your provider let you import it, I'm sure your provider will not be so helpful when you need to move it back out.

Blatant plug: Perhaps Actifio (www.actifio.com) can fix this for you, by replicating your data in, and also back out of production systems in deduped and compressed format.

Backups (5, Informative)

SJHillman (1966756) | more than 2 years ago | (#40069227)

My last employer offered offsite backups to clients. For the initial seed, we always tried to get them to put it on an external HDD and ship it to us (or at least DVDs). The only major exceptions were clients that were also on FiOS - that was the only case where over-the-net transfer was faster than the backup-and-ship-it method for the initial seed.

tapes have to be written and read (1)

rubycodez (864176) | more than 2 years ago | (#40069337)

station wagon has low bandwidth, the tapes have to be written and read.

Re:tapes have to be written and read (1)

h4rr4r (612664) | more than 2 years ago | (#40069465)

140MB/s is low speed?

That is the speed of LTO-5, at best.

Re:tapes have to be written and read (1)

rubycodez (864176) | more than 2 years ago | (#40071921)

LTO-5 doesn't do that over sustained time periods.

Re:tapes have to be written and read (1)

jedidiah (1196) | more than 2 years ago | (#40072153)

Then put it on a RAID array. I knew people that were doing this more than 10 years ago. They needed to move large amounts of data around. So they would FedEx a RAID array around.

Re:tapes have to be written and read (1)

fuzzyfuzzyfungus (1223518) | more than 2 years ago | (#40069469)

Random Access on tapes makes using your HDD as RAM seem like fun; but contemporary tape drives are actually pretty damn fast. It may be necessary to flog the tape minions on occasion, in order to spur them to greater effort; but that isn't tape's fault...

Re:tapes have to be written and read (1)

Jeng (926980) | more than 2 years ago | (#40069727)

That only depends on how many tape drives you utilize at once. If you are shipping 200lbs of tape, and using dozens of tape drives then you should have much better bandwidth than if you tried to send it over the internet.

Re:tapes have to be written and read (1)

ksandom (718283) | more than 2 years ago | (#40069921)

station wagon has low bandwidth, the tapes have to be written and read.

No. Bandwidth is how much you can send in one go (tapes/hdds in a car are extremely high). Latency is fairly much how long it takes you to do it. Throughput brings these together.

Re:tapes have to be written and read (1)

rubycodez (864176) | more than 2 years ago | (#40071899)

sorry pal, in digital technology bandwith is measured in quantity of information per unit time, it is rate of transfer

Re:tapes have to be written and read (1)

Animats (122034) | more than 2 years ago | (#40070719)

True. I was at one point involved in converting the Stanford AI Lab archives from 6250BPI tape to a file server. People were loading tapes for weeks. As soon as a tape was loaded, the data went over the Internet to a file server at IBM Almaden for format conversion. The transmission of a tape only took a few seconds. Of course, both IBM Almaden and Stanford have major backbone connections.

I'm not so sure... (2)

fuzzyfuzzyfungus (1223518) | more than 2 years ago | (#40069355)

I don't think that TFS's answer is necessarily the correct one. I'd really prefer to hold Ma Bell's feet to the fire concerning the fact that bandwidth(even in 'optimal' build-out areas, spare me the excuses about the boonies) has enjoyed a deeply underwhelming track record in terms of improvements in cost and quantity compared to most other aspects of contemporary computing.

Bandwidth (1)

Impy the Impiuos Imp (442658) | more than 2 years ago | (#40069365)

Intercontinental company I used to work for, once or twice a year they'd send an intern over the Atlantic in the SST with a case of tapes.

When it just positively had to be there asap...

Bandwidth of a Station Wagon (5, Funny)

Surazal (729) | more than 2 years ago | (#40069385)

Yes, never underestimate the bandwidth of a station wagon full of disks hurtling down the highway. The latency, on the other hand, leaves much to be desired, and I've heard the packet loss can be downright fatal.

Re:Bandwidth of a Station Wagon (1)

interkin3tic (1469267) | more than 2 years ago | (#40070769)

IP over avian carriers [wikipedia.org] is less fatal, though lower bandwidth and packet loss due to hawks are a problem then.

Bandwidth of a station wagon (0)

Anonymous Coward | more than 2 years ago | (#40069453)

I have never liked the station wagon analogy, because it misunderstands the thing we are trying to measure. In the example, we measure the bandwidth of the station wagon. But that's like measuring the bandwidth of a packet -- a nonsense concept. We measure the bandwidth of the channel, not the chunks of data which fly through it. To really get the right analogy, we should talk about the bandwidth of a freeway, not the station wagon which drives upon the freeway.

Bandwidth in the colloquial sense means "the amount of data which passes a given point, per second." So, imagine that you can load 25 TB in the form of tapes into a station wagon. For safety, these station wagons must drive a distance of 75 meters apart and a speed of 100 kilometers per hour. That means that one station wagon passes a given point every 2.7 seconds. That's 9.2 TB per second. Adding a second lane to the highway would double the bandwidth.

The stupid calculation which is often performed, on the other hand goes like this. You have 25 TB in the wagon, and you drive it to a location 10 hours away... Already you've gone off the tracks, because you are mentioning the TIME it takes to get to the destination, i.e. the LATENCY. And as anybody knows, the latency (or equivalently the distance between the points) has NOTHING to do with bandwidth.

Re:Bandwidth of a station wagon (3, Informative)

BradleyUffner (103496) | more than 2 years ago | (#40069605)

I have never liked the station wagon analogy, because it misunderstands the thing we are trying to measure. In the example, we measure the bandwidth of the station wagon. But that's like measuring the bandwidth of a packet -- a nonsense concept. We measure the bandwidth of the channel, not the chunks of data which fly through it. To really get the right analogy, we should talk about the bandwidth of a freeway, not the station wagon which drives upon the freeway.

Bandwidth in the colloquial sense means "the amount of data which passes a given point, per second." So, imagine that you can load 25 TB in the form of tapes into a station wagon. For safety, these station wagons must drive a distance of 75 meters apart and a speed of 100 kilometers per hour. That means that one station wagon passes a given point every 2.7 seconds. That's 9.2 TB per second. Adding a second lane to the highway would double the bandwidth.

The stupid calculation which is often performed, on the other hand goes like this. You have 25 TB in the wagon, and you drive it to a location 10 hours away... Already you've gone off the tracks, because you are mentioning the TIME it takes to get to the destination, i.e. the LATENCY. And as anybody knows, the latency (or equivalently the distance between the points) has NOTHING to do with bandwidth.

How can you say Time has nothing to do with bandwidth when, in your own example, you measured it in TB per SECOND?

Following your example again of 9.2TB/sec, that can be changed to 9.2TB * 60 /min, or 9.2TB * 60 * 60 /hour, or 9.2TB * 60 * 60 * 10 / 10 hours, which is the exact measurement that you seem to have a problem with earlier in your post (data in a 10 hour period).

Re:Bandwidth of a station wagon (0)

Anonymous Coward | more than 2 years ago | (#40070149)

Served.

Re:Bandwidth of a station wagon (4, Interesting)

Firethorn (177587) | more than 2 years ago | (#40070365)

Indeed. He also ignored the core reason for having said bandwidth - you have X amount of data to move in Y time (at under Z cost); what's the best way to do so?

As such, a 'packet' on the freeway system is rather expensive, so you don't want to be putting multiple station wagons on the system if you don't have to. Figure the driver costs $20/hour, the vehicle itself $.50/mile(gas, maintenance, insurance, tolls, etc...), and you're looking at 300 miles in 10 hours. For a single packet you're looking at $350 for that single 'packet'. If a single station wagon doesn't do it, perhaps a cargo van would, which doubles the capacity of the packet while only raising the cost $50, to $400. Still not good enough? Upgrade to a 'package van' like UPS/Fedex trucks. Next step would be a Semi.

In any case, I'd say that you could fit 25TB into a motorcycle today - 3 TB drives are fairly common now, and I can fit 10 into my saddlebags easily. Heck, I can get 1.5TB native tapes [wikipedia.org] , about the same size as a HD. Padding it's dimensions up, it's 11 x 11 x 3 cm = 363 cm^3, or 2,755 per cubic meter.

A 2008-11 Dodge Grand Caravan Cargo van [allpar.com] - 143.8 cubic feet = 4.07 cubic meters, giving me room for 11k 1.5TB tapes. 16.5k TB, in 10 hours, if I have a single cargo van. Ouch. Disregarding media cost, that's ~$400.

Do this daily, we're looking at 1.5 terrabits per second. Don't know of any connections that fast.
Monthly, we're down to ~50 gigabit (rounding down). I can guarantee that a 50 gigabit connection will cost more than $400.
Annually, it's 'only' 4 gigabit, and I pay more than $100/month for my megabit class connection, which ISN'T utilized 100%, unlike my calc.

You don't normally need to figure out the bandwidth of the freeway because:
1. Generally 1 vehicle 'packet' is sufficient, and due to the high marginal cost per said vehicle, you normally only want to send one.
2. The roads are used for more than data shipment, which would be like trying to figure out how much bandwidth you have available for VOIP by looking at total circuit bandwidth.

Don't need to ship that much? You should be able to ship about 30 of them for $60, second day air. That's 45TB, or about 140 Mbit of 100% saturated traffic for a month. BTW, during my calcs for paying fedex to ship them, I think that weight might actually be enough of an issue to increase gasoline consumption - but I think I've established that even $800 would be cheap if you need to ship that ridiculous of an amount of data.

Re:Bandwidth of a station wagon (1)

bennomatic (691188) | more than 2 years ago | (#40069647)

Well, ihe station wagon is sort of a joke, but they could reword it as "throughput" and have it be accurate. The point is to compare one delivery mechanism with another. If you have 25 TB to send and your upstream connection only allows you to average 1Mbps then delivery time will be measured in years; same amount of data could likely be walked faster between almost any two walkable points on Earth.

Re:Bandwidth of a station wagon (0)

Anonymous Coward | more than 2 years ago | (#40069653)

I just hope you're trying to troll...

Snail mail FTW! (1)

jtownatpunk.net (245670) | more than 2 years ago | (#40069501)

I got my first linux distribution (I don't remember if they were called distributions back then) shipped on tape to the campus computer lab where a group of us brought our computers to copy the files.

Re:Snail mail FTW! (1)

couchslug (175151) | more than 2 years ago | (#40069589)

I got mine in 1999 from Cheapbytes:

http://www.cheapbytes.com/ [cheapbytes.com]

Corel Linux FTW!

Re:Snail mail FTW! (1)

Dogtanian (588974) | more than 2 years ago | (#40069797)

I got mine in 1999 from Cheapbytes: http://www.cheapbytes.com/ [cheapbytes.com]

Clicked on the link out of curiosity, got a splash screen that looked like it was designed in the 90s. (*) Anyway, clicked on the "Click here to enter the CheapBytes store" and I got...

Great Success !
Apache is working on your cPanel® and WHM Server

If you can see this page, then the people who manage this server have installed cPanel and WebHost Manager (WHM)

So are they still trading or is this just a zombie remnant? Guessing that their business would have shrunk quite a lot since the days when everyone was on dialup and you'd have had to be on crack to consider downloading even a CD's worth, let alone a DVD. (Used to order Linux discs quite a lot myself, haven't done it since I got broadband.)

(*) Come to think of it, old-style splash-screens-for-the-sake-of-it (and people thinking they were actually a good idea even when they were obviously just irritating corporate-ego "look at me" wankery that got in the way) are pretty 90s in themselves.

Satellite cap (1)

tepples (727027) | more than 2 years ago | (#40070293)

Guessing that [a CD distributor's] business would have shrunk quite a lot since the days when everyone was on dialup and you'd have had to be on crack to consider downloading even a CD's worth, let alone a DVD.

Shrunk? Yes, I'll grant. Still useful in places that can't get FTTH, DOCSIS, or DSL? Yes. Satellite and cellular are still capped to about one DVD a month, with single or dual layer depending on which plan you choose.

Re:Satellite cap (1)

jedidiah (1196) | more than 2 years ago | (#40072213)

Cheapbytes didn't even really come around until broadband was commonplace. The fact that they are "cheap" is a reflection of that. Before then, you had more expensive CD sets.

It's cheap for a reason.

Re:Snail mail FTW! (1)

jedidiah (1196) | more than 2 years ago | (#40072203)

I got mine in 1994 from Walnut Creek. 6 CDs. 4 Distributions.

Re:Snail mail FTW! (1)

jaymemaurice (2024752) | more than 2 years ago | (#40073047)

I am much younger, I got my linux from a dead tree.

Re:Snail mail FTW! (1)

jaymemaurice (2024752) | more than 2 years ago | (#40073071)

That is relevant because it was easier/faster to get an up-to date-ish linux cd with a book that had to go through various editors, a printing press and the book stores distribution system then download the cd over a 33.6 modem... and then it was easier to read the book then load /search a document that large in a word processor and stare at a low res CRT screen.

A problem bigger than getting your data on ... (4, Insightful)

petes_PoV (912422) | more than 2 years ago | (#40069655)

... is getting it all back OFF again when you want to switch service providers.

The one thing you want never to happen is that you get locked in to a single cloud service. They might go bust, they might become uncompetitive. They may become politically "unfriendly" or tainted with customers you have no desire to be associated with - or any of a number of other reasons to say "adios".

Just like with disaster planning, all the processes and procedures, agreements and SLAs are worthless until you've actually PERFORMED the operation and done so without a major service interruption. How many cloud users have gone that far - and how many are locked in but don't know it?

Re:A problem bigger than getting your data on ... (2)

jtownatpunk.net (245670) | more than 2 years ago | (#40070529)

We've already seen the unsinkable cloud get sunk. Amazon's never-down cloud has rained out at least once in some regions. Another problem is the cloud provider making changes to their services that impact the way your company operates. My last company was in the process of Googleizing when I left a year ago and Google's already made changes that must have required training and documentation updates. And they can remove apps and services at any time so some unpopular product that happens to be very useful to your company could disappear because it's not profitable for the hosting company to continue to support it. At best, you might be able to keep the service until your contract renewal date.

Then what do you do? Take everything to another cloud? Good luck with that.

Re:A problem bigger than getting your data on ... (0)

Anonymous Coward | more than 2 years ago | (#40070533)

... is getting it all back OFF again when you want to switch service providers.

Oh that shouldn't be a problem, because you'll want to keep local backups, anyway. I mean, who in their right mind is going to transport the sole source of data to an off-site storage facility? So, technically, once you've copied the data for transport, shipped it for transfer, and if you've received them back unmolested, you now have triple redundant storage capacity (one online, two locally).

I actually find the whole premise quite amusing... ship my data discs to the online storage provider. Yeah, that's gonna happen LOL. If I am unable to convince my employer of what a bad idea "The Cloud" is, and s/he has *that much data* to move, the cloud provider can ship their discs to me and I'll do the copying, thanks.

Re:A problem bigger than getting your data on ... (0)

Anonymous Coward | more than 2 years ago | (#40070791)

I had the same problem moving our data to Google Apps. At least Amazon s3 provides (for a fee) the ability to ship them drives. Uploading 100's of Gigs of data to Google proved impossible via their web software. We discovered an app, Syncdocs http://syncdocs.com, that does reliable migration of huge data sets to Google Drive.

The cost of migrating data will provide a new vendor lock-in, just like Microsoft locked us in with Office file formats.Once they have all your data, you're stuck. You can check out, but you can never leave...

here in the US... (1)

slashmydots (2189826) | more than 2 years ago | (#40069901)

Here in the US we have a $270/month cable connection that's 10x2Mb so yeah, our daily backup uploads would take longer than 24 hours, or as they call it on the street, a day. We don't have the best compression or de-dupe but still. I can't even imagine quickbooks off the web let alone our actual databases. As far as I'm concerned, cloud stuff isn't a magical futuristic awesome technology to migrate to, it's a crappy, slow, unstable, budget solution that small business might consider. I just love the ads for "build your own local, onsite cloud solution!" or as I like to call it, "not cloud."
Also, the networking book I read estimated that the bandwidth vs cost of Fedex 1 day guaranteed overnight to anywhere US to US was a better cost compared to bandwidth solution than a station wagon on the freeway. Barracuda backup solutions agrees and uses that as the initial backup method.

Re:here in the US... (1)

HapSlappy_2222 (1089149) | more than 2 years ago | (#40070231)

Yeah; it's funny to me that they call "shipping disks" the "oldest answer, but still the right one". As if the other options have ever been even marginally realistic... "Bringing a shipping container in from Tokyo? While you *could* dog-paddle it home on your back, the oldest option - sending it by ship - is still the right one."

Storage nodes shipped to customers (1)

Anonymous Coward | more than 2 years ago | (#40070077)

I work at a cloud storage gateway company...in engineering...if that helps.

    For customers in the 1-40TB range:
    o 1Gbps links into the cloud vendor are available, even if only for a short initial duration.
    o A good cloud storage gateway can maintain that 1Gbps.
    o A good cloud storage gateway will compress, de-duplicate, and otherwise squeeze every bit of redundancy out of the data to reduce the transmitted and resting data size.
    o A good cloud storage gateway encrypts the data before it goes to the cloud.
    o At 1Gbps, 1TB is about 2.5 hours. So in a regular weekend, 21TB can be uploaded. If that was compressed/de-duplicated you might be talking about 30-100TB of client data sent up in a weekend.
    o Many fortune 1000 companies that have 1-5 Gbps into their cloud storage.

    For the larger customers that think in PB:
    o Some Cloud Storage vendors have shipped storage nodes to the customer.
    o Local storage nodes become the target for a cloud storage gateway.
    o Write all the data at 10Gbps or more with multiple gateways.
    o Ship the storage nodes back to the cloud vendor.
    o Note that not all vendors will do this...but it is worth asking if you speak in PBs.

    So, yeah, a sneaker net is still the answer for the large data sets, but 1Gbps isn't too shabby for most everyone else.

TCP/USPS (1)

nilbog (732352) | more than 2 years ago | (#40070361)

Ah yes, the TCP/USPS revolution has finally arrived!

Re:TCP/USPS (1)

marcosdumay (620877) | more than 2 years ago | (#40072023)

You must use a hell of a large window size with all that latency.

Aspera and Friends (3, Interesting)

PhillC (84728) | more than 2 years ago | (#40070375)

You you always use a UDP solution such as Aspera [asperasoft.com] . Fast transfer speeds, bandwidth management and they have a specific AWS implimentation. [asperasoft.com]

Other options to look at include Smartjog [smartjog.com] , whose new Bolt product looks quite interesting, Riverbed's Steelhead [riverbed.com] product, Filecatalyst [filecatalyst.com] and Signiant [signiant.com] .

There are many solutions around now to deal with large file transfers for both small and large business. Most of them use UDP instead of TCP/IP, with Checksums to ensure all data is reliable delivered. Even with just 1Mbps upload speeds, something like one of the above named products will be advantageous. I've worked in the media industry for a number of years, and this type of thing is being used in Film and Television all the time. Of course, there are still tapes being shipped around, but in emerging markets, such as Russia for instance, the file transfer really beats a tape being stuck in customs for weeks or months.

But then... (4, Insightful)

TemperedAlchemist (2045966) | more than 2 years ago | (#40070477)

How did you manage to fix armed FBI storming your servers located in another country problem?

Re:But then... (1)

aiht (1017790) | more than 2 years ago | (#40073203)

How did you manage to fix armed FBI storming your servers located in another country problem?

Unless you're not in the US, and "another country" is the US - what the hell are the FBI doing there?

Not just a "public" challenge (2)

dave562 (969951) | more than 2 years ago | (#40070707)

I am dealing with this as well, albeit on a different scale. About a year ago, the powers that be decided that they were going to develop a private cloud for the company. Nobody really considered how to migrate 500+TB of data from three separate sites into the new cloud. We are doing a mixture of over the wire replication (for sites with 100Mb+ of bandwidth), physical replication (using NAS devices and tape), and synchronization using DoubleTake for the SQL data and Vice Versa Pro for file system data. It is a massive undertaking, made even more difficult by the fact that we are working with production systems with locked in SLAs that need be maintained.

For the average person, and even most enterprises, I honestly believe the best way to get into "the cloud" is by following a well planned out, phased approach. The first phase should be using the cloud as a DR target. Only when both sides of the equation are balanced and able to operate independently of each other can you consider doing away with one and moving to the other.

Re:Not just a "public" challenge (2)

ediron2 (246908) | more than 2 years ago | (#40071951)

Actually, enterprise issues regarding data synchronization quickly make get problematic.

Have just watched a migration from private mail servers to cloud-based email. Months in, it quickly became apparent that a few short days or even a few weeks of pain associated with migrating users cold turkey (and then importing requested data from Notes once it had become static) would have been astronomically less cost and pain compared to wiring the connector and having two frameworks alive (and borking the sync in weird mediocre ways) simultaneously for months. Keep in mind, 'just migrating email' ends up meaning email plus antispam plus accounts plus calendars plus messages plus archives plus attachment storage plus service accounts (and changing hardcoded email addresses in code) plus security plus distro lists... etc, etc etc.

It can be straightforward (but not trivial) to export static data from SQL or another known framework and migrate it into similar frameworks. And it's not much harder to translate (like SQL into non-relational (NoSQL or similar) frameworks): Understand the data, design to the new constraints and advantages, plan migration, do a trial of the migration, then cycle thru again with more or all (depending on how well the first migration went).

Doing so on data that remains alive and breathing gets damned hard fast. Even if there are migration tools, are they ok with transactional data changes? If one side of the migration fails, do both frameworks get messages to cancel the transactions? What about record deletion? How is that synchronized? Given that the new cloud vendor may have few customers (insert tiny #) customers, does a well-tested connector exist between your old and new data stores? What about all in-house apps using the data? Are there mobile apps or custom code or ERP connections to your data server? What about data structure changes between the two: nothing maps perfectly.

Ugly. Just Ugly.

And that's a planned migration. Now think of going the other way. As TFA hints at, we all can write the headline and article now for going the other direction the day some cloud provider implodes: Company Widgetcorp declared bankruptcy today. Their tragic fall from stalwart Rusell-2000 midcap manufacturer to receivership happened unexpectedly: Their cloud-based XXX provider shut off servers without warning less than 60 days ago, and Widgetcorp was never able to recover critical processes and data.

The bandwidth of a fully laden alimentary canal. (1)

queazocotal (915608) | more than 2 years ago | (#40071055)

Is around a gigabyte per second.

(100 packs of 16*64GB microSDs, in appropriate packaging, swallowed at intervals over the course of a day)

Re:The bandwidth of a fully laden alimentary canal (1)

Crypto Gnome (651401) | more than 2 years ago | (#40071927)

Not that there isn't a right time and place for MicroSD but the above suggestion is pretty much semantically identical to Garbage In Garbage Out.

Re:The bandwidth of a fully laden alimentary canal (3, Funny)

FunkSoulBrother (140893) | more than 2 years ago | (#40072979)

African or European?

Perhaps the question is ... (1)

PPH (736903) | more than 2 years ago | (#40072013)

.... how did you end up with all that data on the servers in your mom's basement in the first place.

The 'Cloud' option needs to be a part of your system design in the first place. So you begin to accumulate all that data in The Cloud from the word go.

Canada (3, Interesting)

DarwinSurvivor (1752106) | more than 2 years ago | (#40072189)

In canada, unless you need low latency, the internet is about the most expensive method you could possibly use to transfer data. source [dslreports.com]

AWS Import/Export service... just ship the disks. (2)

isaac (2852) | more than 2 years ago | (#40072655)

http://aws.amazon.com/importexport/ [amazon.com]

http://awsimportexport.s3.amazonaws.com/aws-import-export-calculator.html [amazonaws.com]

It's not rocket science. Yes, shipping drives is the cheapest, fastest option for a lot of people.

YMMV, speaking for myself, not my employer, etc. etc.

-Isaac

Dude! Where's my cloud? (0)

Anonymous Coward | more than 2 years ago | (#40073503)

Wait until your "cloud" operator goes bust or their data center burns down or something else happens that makes it impossible to get to your data.

Speaking from a disaster recovery perspective, "The Cloud" (man I hate that term) can only be either the primary location with data backup elsewhere or the backup location with the primary elsewhere, the minute companies start using the cloud exclusively they are putting their business at risk.

When "the Cloud" becomes a secure system which is automatically replicated to multiple locations around the planet and it is "owned" by an entity which cannot go bust or hold my data to ransom then I will consider it as being a good place to put my exceptionally valuable data, until then it's a no no.

MegaUploads: The other unspoken hurdle (2)

Arancaytar (966377) | more than 2 years ago | (#40073571)

Namely, the increased risk that your data will become collateral damage in the War On Piracy.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>