Journal vinsci's Journal: HOWTO: Thousands of open files on Linux 1
A project I worked on recently required splitting up more than ten million files, each with thousands of data points, into one file per type of data point and each with more than ten million entries. This seems straight-forward until you realize that by default, few kinds of systems allow having hundreds or thousands of files open simultaneously.
In my case I wrote a Python script to do the job and ended up with about 12,000 output files open at once, while attending to each of the more than ten million input files one at a time.
Fortunately, Debian GNU/Linux is a system that can be instructed to allow having that many open files, although you might get a different impression from the bash(1) manual page, which has the following to say for the "ulimit -n" command:
-n The maximum number of open file descriptors (most systems do not allow this value to be set)
The key here is that by "systems", the manual page really refers to "kinds of systems", not individual GNU/Linux systems. But even on GNU/Linux, it's a bit involved to actually get a non-root shell with an increased number of allowed open files. Here is one way to do it:
Let's begin with seeing how many files we are currently allowed (your number may vary, of course):
vinsci@debian $ ulimit -n
1024
Ok, now increase it. First, become root for example using su(1), sudo(8) or sux(1). Use sux(1) if you need an X application to open a large number of files:
vinsci@debian $ su
As root, we are by default allowed the same number of simultaneously open files:
root@debian # ulimit -n
1024
Now increase the number of allowed open files for this shell to 20,500 and check that it happened:
root@debian # ulimit -n 20500
root@debian # ulimit -n
20500
This increased limit only applies to the current shell and any child processes it has. We're still root, of course, but to get a user shell with this limit we can now simply su(1) again, this time to the desired user account and the new shell will inherit the newly set limit:
root@debian # su - vinsci
... and there it is:
vinsci@debian $ ulimit -n
20500
At this point you can then run the program(s) that requires more open files than normally allowed.
Keep in mind, though, that by exiting the current shell we become root again, so it would not be a good idea to hand over a shell set up this way to an untrusted user.
vinsci@debian $ exit
root@debian # exit
Afterwards, back in the original shell, nothing has changed so the old limit still applies:
vinsci@debian $ ulimit -n
1024
Do you have a more elegant way?
Don't forget file-max (Score:2)
The system-wide maximum number of handles is set in /proc/sys/fs/file-max This also includes network sockets that use file handles.
Systems using PAM for authentication/login processes can set the default per-user limits in /etc/security/limits.conf without having to go through all these steps, but if you just want a different limit for one time, you've got it.