Too many files error

Support
1

Brief description of the problem

Error received on login to Rockstor. This is the second occurrence. A reboot cleared it the last time.
I’m running Rockstor 4.1 and Leap 5.3.18-150300.59.68-default. Uptime 6 days. Rockons: Emby, nut-ups, netdata.

Maybe the answer is as simple as bumping these limits?

rockstor:~ # ulimit -Sn
1024
rockstor:~ # ulimit -Hn
524288
rockstor:~ #

Edit: I did find this, so there may not be a solution at this time. Error when log into web console - Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files

Detailed step by step instructions to reproduce the problem

Browse to rockstor & login

Web-UI screenshot

[Drag and drop the image here]

Error Traceback provided on the Web-UI

Traceback (most recent call last): File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception yield File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 42, in get_queryset self._update_hostname() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner return func(*args, **kwargs) File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 49, in _update_hostname cur_hostname = gethostname() File "/opt/rockstor/src/rockstor/system/osi.py", line 904, in gethostname o, e, rc = run_command([HOSTNAMECTL, "--static"]) File "/opt/rockstor/src/rockstor/system/osi.py", line 214, in run_command raise Exception("Exception while running command({}): {}".format(cmd, e)) Exception: Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files
2

@wdc Hello again.

The too many open files thing is usually an overload of sorts. It’s pretty much exactly what it states. However before, in v3, we tracked this down, we thing, to AFP which has now been removed. We have not since had this reported in v4. It may well be one of the Rock-ons is not cleaning up after it’s file handles sufficiently. What is the spec of this machine? As I think ulimits are set according to the memory of the machine. Not quite sure on this but I remember running into this some time ago.

The following command (list open files) may help you track what is keeping a handle on files (open files):

lsof

Let us know how you get on. I suspect netdata myself. Also you list nut-ups as a Rock-on. We don’t have one of those, is this a custom Rock-on you are running, and in which case why not used the built-in nut-ups service?

Note that once a system runs out of file handles it is basically temporary toast until some are freed-up. It may be you are just overloading this machine. Hence the hardware spec question. Also what size is the pool/s, that can have an additional burden on memory.

Another issue is using your system pool (which is small) for Rock-ons. Possible but not advised. This may also lead to a file handle exhaustion.

Hope that helps.

3

Well that was dumb. Why did I say rockon? I’m running the built-in nut-ups service, in client mode.

As for the machine - 8GB RAM; AMD Athlon 7750 processor; 1 4TB btrfs RAID1, 1 btrfs 1.5TB RAID1, 230GB single drive root filesystem, USB 2TB NTFS drive (secondary backup).

Thanks for lsof - I pulled a current list (Emby is #1 at 9555 files; netdata is a distant second at 5216). I’ll check it again in a few days and see what’s what. lsof will be a great help in tracking it down.

2 Likes