Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Large File Problems in Modern Unices

CmdrTaco posted more than 11 years ago | from the stuff-to-deal-with dept.

Unix 290

david-currie writes "Freshmeat is running an article that talks about the problems with the support for large files under some operating systems, and possible ways of dealing with these problems. It's an interesting look into some of the kinds of less obvious problems that distro-compilers have to face."

cancel ×

290 comments

Sorry! There are no comments related to the filter you selected.

In Soviet Russia... (-1, Offtopic)

xintegerx (557455) | more than 11 years ago | (#5161547)

In Soviet Russia, X-Modern Problems are Filed in Boxes in 1991.

FP (-1, Troll)

Anonymous Coward | more than 11 years ago | (#5161548)

first post mothafucka

Why large files (-1, Troll)

amigaluvr (644269) | more than 11 years ago | (#5161555)

Can anyone give a good reason for needing files larger than 2gb? The seek times alone withinr these files must be huge, and it smacks a bit of inefficienecy

sure its just as bad to have an app use hundreds of say 4kb files or so, but two GIGABYTES???

Re:Why large files (3, Funny)

mr.henry (618818) | more than 11 years ago | (#5161558)

Who needs more than 512k of RAM??

Re:Why large files (2, Funny)

Big Mark (575945) | more than 11 years ago | (#5161590)

Come on. Even Bill Gates admitted that half a meg ain't enough.

640K, on the other hand, should be enough for anyone...

-Mark

Re:Why large files (0)

Anonymous Coward | more than 11 years ago | (#5161645)

ought to be

data warehouse, and any database for that matter (5, Insightful)

CrudPuppy (33870) | more than 11 years ago | (#5161591)

my data warehouse at work is 600GB and grows at a rate of 4GB per day.

the production database that drives the sites is like 100GB

welcome to last week. 2GB is tiny.

Re:data warehouse, and any database for that matte (2, Insightful)

hector13 (628823) | more than 11 years ago | (#5161608)

my data warehouse at work is 600GB and grows at a rate of 4GB per day. the production database that drives the sites is like 100GB welcome to last week. 2GB is tiny.
And you store this "production database" as one file? didn't think so (or atleast I hope you don't).

I am not agreeing (or disagreeing) with the original post, but having a database > 2 GB has nothing to do with having a single file over 2 GB. A db != a file system (except for MySQL perhaps).

Re:data warehouse, and any database for that matte (2, Informative)

CrudPuppy (33870) | more than 11 years ago | (#5161776)

the datafile size averages 8GB in the warehouse.

video, mp3's, even dvds are beyond 2gb (2, Informative)

xintegerx (557455) | more than 11 years ago | (#5161563)

Question answered, move along, nothing to see here :)

Re:Why large files (1)

tgeerts (556153) | more than 11 years ago | (#5161565)

Video + Audio >= 2GB

Re:Why large files (0)

amigaluvr (644269) | more than 11 years ago | (#5161576)

Good answer. A 2gb movie would have to go for nearly 4 hours and that includes audio. Explain?

a 20mb mp3 can go well into an hour. Explain?

If you really need a movie which hits tjhat many hours you would be breaking it up into cd sized chunks anyway

Re:Why large files (1)

AvitarX (172628) | more than 11 years ago | (#5161606)

Maybe high quality audio+vidio for say...
making a movie will be larger then that.

I guess a lot of the editing would probably be done scen by scene, and then you could on the fly merge and compress them so that at no point you use more then 2gb, but it seems that if you make a 2 hour dvd it would be nice to keep the 4gb image file on your hardrive if you planned to reburn it.

Not a scattering of scenes that it would recreate the image on the fly.

It is kind of a dumb question when we have computers being marketed as home dvd makers why would be need that big of a file.

Re:Why large files (5, Interesting)

CoolVibe (11466) | more than 11 years ago | (#5161636)

raw video can easily exceed 2 GB in size. Why raw video? Because (like others said) it's easier to edit. Then you encode to MPEG2, which will shrink the size somewhat (usually still bigger than 2 GB, ever dumped a DVD to disk?), so it'll be "small" enough to burn onto a DVD or somesuch. Oh, editing 3 hours of raw wave data also chews away at the disk size. Also, since you need to READ the data from the media to see if it looks nice, you need to have support for those big files as well. Right, now why don't we need files bigger than 2 GB again? Well?

Oh, you're still not convinced, well see it this way: when in the future will you ever need to burn a DVD?

Well? A typical one sided DVD-R holds around 4 GB of data (somewhat more), if you use both sides, you can get more than 8 GB of data on it. That's way bigger than 2 GB, no? Now, how big must your image be before you burn it on there? well?

Right...

Wrong. (1)

I Am The Owl (531076) | more than 11 years ago | (#5161726)

You obviously have never done any work with video before. Most DV will eat up 2GB easy with 15min of footage or less.

Re:Why large files (1)

amigaluvr (644269) | more than 11 years ago | (#5161602)

Oh I see now raw video is larger than I thought, oops

Re:Why large files (-1, Troll)

CoolVibe (11466) | more than 11 years ago | (#5161569)

Well, large multi-gigabyte broadcast quality MPEG2 video files seem to be a thing I store regularily on my disks. So yeah, what the hell do I need files bigger than 2 GB for, huh?

Re:Why large files (3, Informative)

voodoopriestess (569912) | more than 11 years ago | (#5161570)

Databases, Movie files, Backup files (think dumps to tapes). Animations, 3D modelling.... Lots of things need a > 2GB file size. Iain

Re:Why large files (5, Insightful)

Big Mark (575945) | more than 11 years ago | (#5161571)

Video. Raw, uncompressed, high-quality video with a sound channel is fucking HUGE. Look how big DivX files are, and they're compressed many, many times over.

And compressing video on-the-fly isn't feasible if you're going to be tweaking with it, so that's why people use raw video.

-Mark

Re:Why large files (1)

gbitten (306952) | more than 11 years ago | (#5161833)

Another example of large file utility are the database files. In my job, the DB machine (Solaris) hasn't sufficient disk space to generate the DB dump. The biggest dump have 11GB and I wasn't able to put it in Linux box (RH 6.2), so I used FreeBSD 4.2 with sucess.

Re:Why large files (2, Insightful)

Ogion (541941) | more than 11 years ago | (#5161572)

Ever heard of something like movie-editing? You can get huge files really fast.

Re:Why large files (5, Interesting)

Anonymous Coward | more than 11 years ago | (#5161574)

Real analytical work can easily produce files this large. Output for analyses of structures with more than half a million elements and several million degrees of freedom can EASILY produce output of over two gigs. Yes, these results can and should be split, but sometimes it makes sense to keep them together as a matter of convenience. Plus, there IS a small performance hit when dealing with multiple files on most of the major FEA packages.

Re:Why large files (2, Interesting)

bunratty (545641) | more than 11 years ago | (#5161821)

Over Christmas and New Years, I helped my wife run a simulation of 1000 different patients for an acedemic pharmacokinetics paper. The run took ten days and had an input file of about 1.5 GB. If her computer was faster, or she had access to more computers, she would have wanted to simulate more patients and would easily have needed support for files larger than 4 GB. As CPUs get faster and hard disks get larger, there will be much more demand for these large files as well as more than 4 GB per process.

640K is enough for you! (0)

Anonymous Coward | more than 11 years ago | (#5161584)

Title says it all... Who are *YOU* to decide that *we* do not need 2GB files?

Re:640K is enough for you! (1)

SoSueMe (263478) | more than 11 years ago | (#5161832)

Who are we to tell them what they have to accomodate?
Don't like the way a particular *NIX works? Don't use it.
Try something else.

Re:Why large files (4, Informative)

hbackert (45117) | more than 11 years ago | (#5161586)

vmware uses files as virtual disks. 2GB would be a really, really small disk. UML does the same, using the loop device feature of Linux. Again, a filesystem in a file. Again, 2GB is not much. Simulating 20GB would need 10 files.

Feels like 64kbyte segments somehow...and I really don't want to have those back.

64KB memory segments (1)

KDan (90353) | more than 11 years ago | (#5161648)

Oh come on, those were fun, when you had to load into memory and uncompress a file larger than that :-)

Oh the fond memories :-)

Daniel

Re:Why large files (1)

Timesprout (579035) | more than 11 years ago | (#5161597)

For when Jaron Lanier [sun.com] decides to update his website with 10,000,000 lines of script

Re:Why large files (0)

Anonymous Coward | more than 11 years ago | (#5161605)

Ever heard of something called data processing? Banks, credit card companies, public utilities, etc. all have huge databases that get processed all the time, and involve working as well as final output files in the range of 2 to 20 gigabytes in size.

Seek times are irrelevant in these situations, since the files are processed serially.

Get out into the real world and see what real industry does with real computers.

Re:Why large files (2, Insightful)

Idaho (12907) | more than 11 years ago | (#5161612)

Can anyone give a good reason for needing files larger than 2gb?

I can think of some:

  • A/V streaming/timeshifting
  • Backups of large filesystems (since there exist 320 GB harddisks now, I don't think I should create 160 .tgz files just to back it up, do I?)
  • Large databases. E.g. the slashdot posts table will be easily >2 GB, or so I'd guess. Should the DB cut it in two (or more) files, just...because the OS doesn't understand files >2 GB? I don't think so...

And that's just without thinking twice...there are probably many more reasons why people would want files >2 GB.

that's not a good question (0, Redundant)

xintegerx (557455) | more than 11 years ago | (#5161614)

Don't moderate up ignorance.

That's whining... But I see his point--the only reason right now is for video files. If you want to get your video from your camcorder, it's not going to go straight to CDRW or DVD, it's going to your HARD DRIVE storage. You are going to edit it, right?
Since you probably want to have the best quality, a single file will take a lot of space. (No I don't do this video thing, but I did my own research. Many people do have video, and for computer editing there is no reason to cap a file size.)

Ok fine, I guess he kind of has a point in that question....

Re:that's not a good question (0)

Anonymous Coward | more than 11 years ago | (#5161676)

Video is certainly not the only thing that will make gargantuan files. Massive databases and simulation data will eat 2gb like a twinkie.

Re:Why large files (1)

chrisbolt (11273) | more than 11 years ago | (#5161617)

Database servers?
Web server log files?
tarballs?

Take your pick.

Re:Why large grapes (1)

edox. (467593) | more than 11 years ago | (#5161639)

Dont be the good old fox .)

Re:Why large files (0)

Anonymous Coward | more than 11 years ago | (#5161659)

Can anyone give a good reason for needing files larger than 2gb?
A msg-id history file on a newsserver with long retention.

Q: Why large files? A: Disk images too (2, Interesting)

Anonymous Coward | more than 11 years ago | (#5161661)

While almost all the examples given are good, I don't think anyone has mentioned complete disk images. I have recently had to do this in order to recover from a hardware issue (drive cable failure resulted loss of MBR, nasty) and on a TiVo unit that had a bad drive.

I have most all of my older system images available to inspect. The loopback devices under Linux are tailor made for this type of thing.


I am puzzled as to why you mention the seek times. Surely you would agree that the seek time should be only inversely geometrically related to size, the particular factors depending on the filesystem. Any deviation from the theoretical ideal is the fault of a particular OS's implementation. My experience is that this is not significant.

(user dmanny on wife's machine, ergo posting as AC)

Re:Why large files (2, Interesting)

bourne (539955) | more than 11 years ago | (#5161679)

Can anyone give a good reason for needing files larger than 2gb?

Forensic analysis of disk images. And yes, from experience I can tell you that half the file tools on RedHat (like, say, Perl) aren't compiled to support >2GB files.

Re:Why large files (0)

Anonymous Coward | more than 11 years ago | (#5161696)

Well, the applications I support store and interpret Seismic data. One survey can routinely be in the >100GB range. The visualisation apps we make are often asked to load 2-20GB in memory alone (that's why we still use Sun and SGI systems to do it, though we are actively pursuing Linux too). So 64-bit filesystems and files are kinda important to us.

Umm, scientific computing (1, Insightful)

Anonymous Coward | more than 11 years ago | (#5161713)

Many large-scale computing projects easily generate hundreds of gigabytes and even terabytes of data. They are writing to RAID systems and even parallel file systems to improve their IO.

Think beyond the little toy that you use. These projects are using Unix (Solaris, Linux, BSD and even MacOSX) on clusters of hundreds or thousands of nodes.

Re:Why large files (2, Insightful)

benevold (589793) | more than 11 years ago | (#5161723)

We use a Unidata database here for an ERP system, each database is more than 2gb a piece (more like 20 gb) of relatively small files, when the directories are tarred for backup reasons they are usually over 2gb which means that gzip won't compress them. Unless I'm missing something I don't see an alternative for files large than 2gb in this case. Sure on the personal computing level the closest thing you probably get is ripping DVD's but there are other things out there, and I realize this is tiny in comparison to some places.

Re:Why large files (1)

Veteran (203989) | more than 11 years ago | (#5161740)

I have run into problems trying to compress a tar archive of my home directory which has been around since 1995 when I switched to Linux. The two gig limit runs into trouble here.

Re:Why large files (3, Insightful)

kasperd (592156) | more than 11 years ago | (#5161742)

The seek times alone withinr these files must be huge

Who moded that as Insightful? Sure, if you are using a filesystem designed for floppy disks, it might not work well with 2GB files. In the old days where the metadata could fit in 5KB a linked list of diskblocks could be acceptable. But any modern filesystem uses tree structures which makes a seek faster than it would be to open another file. Such a tree isn't complicated, even the minix filesystem has it.

If you are still using FAT... bad luck for you. AFAIK Microsoft was stupid enough to keep using linked lists in FAT32, which certainly did not improve the seek time.

Re:Why large files (1)

martinschrder (21036) | more than 11 years ago | (#5161750)

Bitmap files for image setters can easily become huge. Think of 500x100(cm)x1000x1000(pixels).

AutoZone's 1TB DB (0)

Anonymous Coward | more than 11 years ago | (#5161754)

I'm posting AC on this one.

I can tell you that, to my knowledge, the AutoZone corporation has databases which exceed a terabyte in size. Yes, that's 1 terabyte. When you consider the sheer number of AutoZone retail locations, combined with their giant inhouse catalog, sales records going back umpteen years, customer data, etc. it's not hard to imagine such a large database.

I'm not saying that one huge database is the way to go. But I am saying that AFAIK it's in practice. 2 gigs is nothing when it comes to file size.

Re:Why large files (1)

Perl-Pusher (555592) | more than 11 years ago | (#5161817)

Science Data usually consist of huge multidimensional arrays. I have seen satellite data in huge netcdf files that are very close if not slightly larger than that.

Re:Why large files (1)

markz (448024) | more than 11 years ago | (#5161820)

database dumps - one of our smaller database dumps is 2.3 GB compressed. The dumps are the easiest method of backup and distribution - locally and (very) remotely.

Re:Why large files (1)

joto (134244) | more than 11 years ago | (#5161835)

Can anyone give a good reason for needing files larger than 2gb?

Yes. Sometimes you need to store a lot of data. Even DVD's has 4.3 GB of data these days. But that's not even much compared to the amount of data we handle in seismic research. I would believe astronomists, particle physicists and a lots of other people also routinely handle ridiculous amounts of data.

By the way, in producing the DVD, you would naturally work with uncompressed data. How would you handle that?

The seek times alone withinr these files must be huge, and it smacks a bit of inefficienecy

And because it is inefficient, we should not support it? As a matter of fact, any file larger than one disk-block is inefficient. Maybe we should stop supporting that as well?

sure its just as bad to have an app use hundreds of say 4kb files or so, but two GIGABYTES???

As I've said, it's not really that much, depending on the application.

Something Large On Unix (-1, Offtopic)

Anonymous Coward | more than 11 years ago | (#5161559)

How do they store CowboyNeal's image?

Not really that groundbreaking... (4, Interesting)

CoolVibe (11466) | more than 11 years ago | (#5161560)

The problem is nonexistant in the BSD's, which use the large file (64 bit) versions anyway. And that you have to use a certain -D flag if your OS (like Linux) doesn't use the 64 bit versions. Whoopdiedoo. Not so hard. Recompile and be happy.

Unices? (0, Offtopic)

ToKsUri (608742) | more than 11 years ago | (#5161573)

Unices plural for unix?

Re:Unices? (2, Informative)

moonbender (547943) | more than 11 years ago | (#5161640)

Yes. Just like "matrices" is the plural of "matrix". Not that the words have a similar etymology - according to dictionary.com [reference.com] it's, in the authors' words, "A weak pun on Multics".

Re:Unices? (1)

Q Who (588741) | more than 11 years ago | (#5161838)

Just like "matrices" is the plural of "matrix".

"Matrices" is a plural form of "matrix." The other one is "matrixes."

Re:Unices? (0)

Anonymous Coward | more than 11 years ago | (#5161654)

Speech impediment, insensitive clod!

Re:Unices? (1)

Looke (260398) | more than 11 years ago | (#5161695)

Geeks seem to have a weird fascination for strange spellings. "-ces" is the traditional plural ending of Latin words ending in "x". Obviously, "Unix" does not originate from Latin, and "Unices" is thus nothing but a (bad) joke. (The same applies to "emacsen", and there are a few others around as well.)

Re:Unices? (0)

Anonymous Coward | more than 11 years ago | (#5161743)

Like the ever popular viri or virii

For some reason that one just makes me cringe in the same way wierd or definately does.

Unices just sounds kind of cool though. go figure :)

Re:Unices? (1)

N1KO (13435) | more than 11 years ago | (#5161813)

Virus comes from latin.

Re:Unices? (0)

Anonymous Coward | more than 11 years ago | (#5161825)

And its latin plural was never viri, or virii

Re:Unices? (1)

david-currie (104829) | more than 11 years ago | (#5161746)

I'd never heard emacsen, but VAXen is commonly used for multiple VAX machines, I believe.

Re:Unices? (0)

Anonymous Coward | more than 11 years ago | (#5161697)

No. An army women named Unice all with files stuffed up their butts.

Its funny how some lamers dont listen... (3, Insightful)

cheekyboy (598084) | more than 11 years ago | (#5161596)

I said this to some unix 'so called experts' in 95, and they said, oh why why do you need >2gig

I can just laugh at them now...

Re:Its funny how some lamers dont listen... (1)

FooBarWidget (556006) | more than 11 years ago | (#5161706)

No you can't, both Linux and FreeBSD support files > 2 GB. Apparenly you've laughed all for nothing.

Re:Its funny how some lamers dont listen... (0)

Anonymous Coward | more than 11 years ago | (#5161710)

'doze couldn't even give you a >2gb PARTITION in 95.
Now that's funny.

Wrong point of view. (0, Interesting)

Krapangor (533950) | more than 11 years ago | (#5161611)

There is not a problem with support of large files in Unix system, there is a problem with incompetent people using too large files in Unix systems.
It's an old and well known problem that programmers and users tend to keep very large files for laziness and logical errors.
However it's also an old and well known fact that large files are bad for performance per se due to several reasons:
  • fragmentation: large files increase to fracmentation of most file systems, at least of any system with uses single indexed trees/B-trees and nonlinear hashes
  • entropy pollution: large files increase to overall entropy on the harddisk leading to worse compression ratios for backup and maintenance
  • data pollution: the use of large files tempts users to store all kinds of redundant, reducible, linear and irrelevant data wasting storage space and I/O time
So I don't see why admins should provide a "work-around" for the filesize limits. These limits are there for very good reasons and in my opinion they are even much to big. You should always remember that the original K&R Unix had only 12 bits for file size storage and was much faster than modern systems, in fact it did run on 2,2 MHz processors and 32 kB of RAM which wouldn't be sufficient for even a Linux of Windows XP bootloader.
Think about it.

Re:Wrong point of view. (1, Interesting)

Anonymous Coward | more than 11 years ago | (#5161657)

As others have noted, there are plenty of good reasons to have files greater than two gigs including video editing and scientific research. The file size limits aren't there for a very good reason at all. Someone years ago had to weigh whether to make small files take up a huge amount of room by using 64 bit addresses that would allow multi-terabyte files to exist against using 32 bit addresses that would make small files smaller and create a 2 gb file limit. At the time, it made perfect sense because nobody was using files anywhere near 2 gb... But now they are.

Re:Wrong point of view. (4, Insightful)

KDan (90353) | more than 11 years ago | (#5161660)

Two words:

Video Editing

Daniel

Re:Wrong point of view. (1)

N1KO (13435) | more than 11 years ago | (#5161678)

In a couple of years, will todays large files be considered large? Ten years ago having hundreds of 4MB files on a pc would've been considered crazy. Now everyone with an mp3 player is used to it.

Re:Wrong point of view. (5, Funny)

heby (256691) | more than 11 years ago | (#5161692)

"oh yes, those were the days." - misty eyed smile - "when i was young and filesizes were small. you should have seen it. today's youth is so spoiled that they don't even learn assembly language any more. i tell you, you're all going to die because of your large files, yes, die!" - madly waves his cane in the air - "2gb, that's more than anybody will ever need and you are greedy for even more! the holy bit will punish you for this, it will!" - dies of a heart attack.

Re:Wrong point of view. (1, Insightful)

Anonymous Coward | more than 11 years ago | (#5161720)

the use of large files tempts users to store all kinds of redundant, reducible, linear and irrelevant data wasting storage space and I/O time

As opposed to a million 4k files that are each 1k of header?

Re:Wrong point of view. (5, Insightful)

cvande (150483) | more than 11 years ago | (#5161722)

In a world everything is small and manageable. Unfortunately, some databases need tables BIGGER than 2gb. Even splitting that table into multiple files still finds you with files larger than two gb. Try adding more tables? OK. Now they've grown to over 2gb and the more tables the more complicated everthing gets. I still need to back these suckers up and a backup vendor that I won't name can't help me because their software wasn't large file (for Linux) ready. So let's get into the game with this and make it the default so we don't need to worry about these problems in the future. Linux IS an enterprise solution.....(my $.02)

Re:Wrong point of view. (4, Insightful)

costas (38724) | more than 11 years ago | (#5161774)

Maybe in your problem domain that's true. I work with retailer data mines and we've hit the 2GB file limit, oh, 4-5 yrs ago? We've been forced to partition databases causing maintainance issues, scalability issues, and the like, just because of the size of a B-tree index.

True, it looks like the optimal solution is lower-level partitioning, rather than expanding the index to 64bits (tests showed that the latter is slower), but that still means that the practical limit of 1.5-1.7 GB per file (because you have to have some safety margin) is far too constraining. I know installations who could have 200GB files tomorrow if the tech was there (which it isn't, even with large file support).

I am also guessing that numerical simulations and bioinformatics apps can probably produce output files (which would then need to be crunched down to something more meaningful to mere humans) in the TB range.

Computing power will never be enough: there will always be problems that will be just feasible with today's tech that will only improve with better, faster technology.

Re:Wrong point of view. (1)

Q Who (588741) | more than 11 years ago | (#5161812)

Lmao...

Your other trolls are nice too, but this one is hilarious... "entropy pollution", hehe :)

"Linux of Windows XP bootloader", this one is amazing. I wonder whether it's a typo, or intentional...

Re:Wrong point of view. (4, Interesting)

Yokaze (70883) | more than 11 years ago | (#5161843)

I'm not a specialist on this matter, so maybe you can enlighten me, where I am wrong or misunderstood you.

> fragmentation: large files increase to fracmentation of most file systems
What kind of fragmentation?

Small files lead to more internal fragmentation.
Large files are more likely to consist of more fragments, but when splitting this data into small files, those files are fragments of the same data.

>entropy pollution
What kind of entropy? Are you speaking of compression algorithms?

Compression ratios are actually better with large files than small files, because similarities between files across file-boundaries can be found. Therefor, gzip(bzip2) compresses a single large tar-file. (Simple test, try zip on many files and then zip without compression and subsequent compression on the resulting file).

>data pollution
How should limiting file size improve that situation? Then, people tend to store data in lot of small files. What a success. People will waste space, whether there is a file size limit or not.

>These limits are there for very good reasons and in my opinion they are even much to big.

Actually, they are there for historical reasons.
And should a DB spread all its tables over thousands of files instead of having only one table in one file and mmapping this single file into memory? Should a raw video stream be fragmented into several files to circumvent a file limit?

>[...] original K&R Unix [...] was much faster than modern systems

Faster? In what respect?

This is only a problem for *BSD because... (-1, Flamebait)

Anonymous Coward | more than 11 years ago | (#5161613)

It is official; Netcraft confirms: *BSD is dying

One more crippling bombshell hit the already beleaguered *BSD community when IDC confirmed that *BSD market share has dropped yet again, now down to less than a fraction of 1 percent of all servers. Coming on the heels of a recent Netcraft survey which plainly states that *BSD has lost more market share, this news serves to reinforce what we've known all along. *BSD is collapsing in complete disarray, as fittingly exemplified by failing dead last [samag.com] in the recent Sys Admin comprehensive networking test.

You don't need to be a Kreskin [amazingkreskin.com] to predict *BSD's future. The hand writing is on the wall: *BSD faces a bleak future. In fact there won't be any future at all for *BSD because *BSD is dying. Things are looking very bad for *BSD. As many of us are already aware, *BSD continues to lose market share. Red ink flows like a river of blood.

FreeBSD is the most endangered of them all, having lost 93% of its core developers. The sudden and unpleasant departures of long time FreeBSD developers Jordan Hubbard and Mike Smith only serve to underscore the point more clearly. There can no longer be any doubt: FreeBSD is dying.

Let's keep to the facts and look at the numbers.

OpenBSD leader Theo states that there are 7000 users of OpenBSD. How many users of NetBSD are there? Let's see. The number of OpenBSD versus NetBSD posts on Usenet is roughly in ratio of 5 to 1. Therefore there are about 7000/5 = 1400 NetBSD users. BSD/OS posts on Usenet are about half of the volume of NetBSD posts. Therefore there are about 700 users of BSD/OS. A recent article put FreeBSD at about 80 percent of the *BSD market. Therefore there are (7000+1400+700)*4 = 36400 FreeBSD users. This is consistent with the number of FreeBSD Usenet posts.

Due to the troubles of Walnut Creek, abysmal sales and so on, FreeBSD went out of business and was taken over by BSDI who sell another troubled OS. Now BSDI is also dead, its corpse turned over to yet another charnel house.

All major surveys show that *BSD has steadily declined in market share. *BSD is very sick and its long term survival prospects are very dim. If *BSD is to survive at all it will be among OS dilettante dabblers. *BSD continues to decay. Nothing short of a miracle could save it at this point in time. For all practical purposes, *BSD is dead.

Fact: *BSD is dying

640 K ought to be enough for anybody (3, Funny)

cyber_rigger (527103) | more than 11 years ago | (#5161622)

--Bill Gates

Re: 640 K ought to be enough for anybody (1)

Looke (260398) | more than 11 years ago | (#5161709)

Yeah, we should all switch to OpenOffice. I had a 20,000 row, 13 MB Excel file, which I resaved in OpenOffice Calc format. It came out at a sweet 640 KB ;-)

It will happen with time_t, too (5, Informative)

wowbagger (69688) | more than 11 years ago | (#5161629)

We are seeing problems with off_t growing from 32 to 64 bits. We are also going to see this when we start going to a 64 bit time_t, as well (albeit not as badly - off_t is probably used more than time_t is.)

However, the pain is coming - remember we have only about 35 years before a 64 bit time_t is a MUST.

I'd like to see the major distro venders just "suck it up" and say "off_t and time_t are 64 bits. Get over it."

Sure, it will cause a great deal of disruption. So did the move from aout to elf, the move from libc to glibc, etc.

Let's just get it over with.

huh? (0, Redundant)

inviagrated_amnesiac (455650) | more than 11 years ago | (#5161631)

wtf kind of sentence construction is this:

"It's an interesting look into some of the kinds of less obvious problems that distro-compilers have to face."

Why not:

"It is an interesting problem that some distro-compilers have to face."

Re:huh? (1)

KDan (90353) | more than 11 years ago | (#5161681)

It's certainly something that George Orwell would have frowned upon [mtholyoke.edu] , but it's not incorrect sentence construction per se.

PS: Read that Orwell article if you haven't yet, it's really very good

Re:huh? (2, Informative)

JanneM (7445) | more than 11 years ago | (#5161685)

Because the sentences mean different things.

"It is an interesting problem that some distro-compilers have to face."

talks about the problem facing distro compilers, whereas

"It's an interesting look into some of the kinds of less obvious problems that distro-compilers have to face."

Talks about the article adressing these problems. /Janne

Re:huh? (1)

david-currie (104829) | more than 11 years ago | (#5161711)

Because it's not an interesting problem. It's a fucking boring problem if _you_ have to deal with it. But it's interesting to read about because it's the kind of thing you probably haven't thought about if you don't compile distributions. I meant what I wrote.

Re:huh? (0)

Anonymous Coward | more than 11 years ago | (#5161732)

More words is meaning smarter! Especially big words. 1up, yatta!

Re:huh? (2, Interesting)

RumpRoast (635348) | more than 11 years ago | (#5161792)

Actually you changed the meaning of that sentence. I think really we object to:
"It's an interesting look into some
of the kinds of less obvious problems that distro-compilers have to face."

"of the kinds" really adds nothing to the meaning here, nor does "have to"

Thus we have:

"It's an interesting look into some of the less obvious problems that distro-compilers face."

The same sentence, but much cleaner!

Thanks! I'll be here all week.

Re:huh? (1)

david-currie (104829) | more than 11 years ago | (#5161837)

Now this I can accept. I promise to think about what I write next time. ;)

A woman's perspective . . . (5, Funny)

pariahdecss (534450) | more than 11 years ago | (#5161633)

So my wife says to me, "Honey, do I look fat in this filesystem ?"
I replied, "Sweetie, I married you for your trust fund not your cluster size."

How large are we talking? (1)

httpamphibio.us (579491) | more than 11 years ago | (#5161649)

It doesn't give a specific filesize in the article...

Re:How large are we talking? (1)

voodoopriestess (569912) | more than 11 years ago | (#5161663)

2^32 Bytes (aka 2GB).

Iain

Funny...in AIX... (4, Informative)

cshuttle (613776) | more than 11 years ago | (#5161665)

We don't have this problem-- 4 petabyte maximum file size 1 terabyte tested at present http://www-1.ibm.com/servers/aix/os/51spec.html

Well........ (-1, Redundant)

Anonymous Coward | more than 11 years ago | (#5161668)

There is not a problem with support of large files in Unix system, there is a problem with incompetent people using too large files in Unix systems. It's an old and well known problem that programmers and users tend to keep very large files for laziness and logical errors. However it's also an old and well known fact that large files are bad for performance per se due to several reasons: * fragmentation: large files increase to fracmentation of most file systems, at least of any system with uses single indexed trees/B-trees and nonlinear hashes * entropy pollution: large files increase to overall entropy on the harddisk leading to worse compression ratios for backup and maintenance * data pollution: the use of large files tempts users to store all kinds of redundant, reducible, linear and irrelevant data wasting storage space and I/O time So I don't see why admins should provide a "work-around" for the filesize limits. These limits are there for very good reasons and in my opinion they are even much to big. You should always remember that the original K&R Unix had only 12 bits for file size storage and was much faster than modern systems, in fact it did run on 2,2 MHz processors and 32 kB of RAM which wouldn't be sufficient for even a Linux of Windows XP bootloader. Think about it.

Have you ever seen some people's email? (4, Insightful)

alen (225700) | more than 11 years ago | (#5161672)

On the Windows side many people like to save every message they send or receive to cover their ass just in case. This is very popular among US Government employees. Some people who get a lot of email can have their personal folders file grow to 2GB in a year or less. At this level MS recommends breaking it up since corruption can occur.

MOD UP (1)

xintegerx (557455) | more than 11 years ago | (#5161739)

I e-mailed somebody on the Board of Higher Ed of my State for some answers, and they simply replied

Please call me at #-###-###-###.

Thanks

He has a really good point if mail programs put archives in one big zip-equivalent file, because these CAN get huge.

Re:Have you ever seen some people's email? (5, Funny)

nentwined (626268) | more than 11 years ago | (#5161801)

I agree with MS on this one. government employees shouldn't be allowed to hold their positions for longer than a year. DOWN WITH GOVERNMENTAL CORRUPTION! ... :)

Re:Have you ever seen some people's email? (2, Informative)

sqrlbait5 (67782) | more than 11 years ago | (#5161810)

Yeah, but if you're using NTFS, where there doesn't appear to be a max file size, you still get the 2GB limit on Outlook files. Every damn version of Outlook has had this 2GB limit, but OutlookXP doesn't actually fix the problem, just warns the user at 1.87GB. We have people hitting their limit all the time at work, but that's because they like to send artwork and whatnot and not clear out their folders.

Switch to gnu/hurd (3, Funny)

Anonymous Coward | more than 11 years ago | (#5161690)

It has a nice small 1gb filesystem limit. I have partitioned my hard disk in to 64 little chunks and it runs very slowly, and unstabilly, but its completley open source and im happy.

Another example of why Freebsd is superior to (-1, Troll)

Anonymous Coward | more than 11 years ago | (#5161702)

Linux sucks

Yay Linux! (-1, Flamebait)

I Am The Owl (531076) | more than 11 years ago | (#5161719)

Let's hear it for FreeBSD, of all things, actually having better large file support. What were the Linux kernel hackers thinking? How sloppy.

Its not the size of the file... (1, Funny)

bananaape (542919) | more than 11 years ago | (#5161736)

Its how you use it.

Why not to learn from past? (2)

Libor Vanek (248963) | more than 11 years ago | (#5161767)

I just wonder why we don't learn from past (limits) and remove this limits "forever". E.g. 1 month ago I recieved question of possibility building 10 TB Linux cluster (physics are crazy ;-)).

There surely MUST be some way how to do this - I just imagine some file (e.g. defined in LSB) which would define this limits for COMPLETE system (from kernel, filesystems, utils to network daemons). I know there are efforts to things like this but if we'd say (for example) thay that distribution in 2004 won't be marked "LSB compatible" if ANY of programs will use any other limits I think it will create enough preasure on Linux vendors.

Just a crazy idea ;-)

The O/S should do it and do it well. (3, Interesting)

tjstork (137384) | more than 11 years ago | (#5161769)

1) Splitting up a big file turns an elegant solution into a an inelegant nightmare.

2) Instead of 10 different applications writing code to support splitting up an otherwise sound model, why not have 1 operating system have provisions for dealing with large files.

3) You are going to need the bigger files with all those 32 bit wchar_t and 64 time_ts you got!

BeOS Filesystem (2)

SixArmedJesus (513025) | more than 11 years ago | (#5161834)

I remember reading in the BeOS Bible that the BeOS filesystem could contain files as large as 18 petabytes. Makes you wonder two things: What's the biggest filesystem that you could use with a BeOS machine? and Why don't other OSs have filesystem like this. Espcecially with those awesome extended attributes. I weep for the loss of the BeOS filesystem...
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>