Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files - may be AFS related

[millerce (C Miller)]

Brief description of the problem

Cannot update Rockstor.

Detailed step by step instructions to reproduce the problem

[write here]

Web-UI screenshot

[Drag and drop the image here]

Error Traceback provided on the Web-UI

Traceback (most recent call last): File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception yield File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 41, in get_queryset self._update_hostname() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner return func(*args, **kwargs) File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 48, in _update_hostname cur_hostname = gethostname() File "/opt/rockstor/src/rockstor/system/osi.py", line 766, in gethostname o, e, rc = run_command([HOSTNAMECTL, '--static']) File "/opt/rockstor/src/rockstor/system/osi.py", line 113, in run_command 'Exception while running command({}): {}'.format(cmd, e)) Exception: Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files

[phillxnet (Philip Guyton)]

@millerce Welcome to the Rockstor community.

As you say this is suspected as related to our treatment of AFP or AFP itself and is an ongoing and long standing issue. Please see our GitHub issue on this for some background:

https://github.com/rockstor/rockstor-core/issues/1656
with a recent duplicate by way of:
https://github.com/rockstor/rockstor-core/issues/2020

This issue in turn link back to the following 2 forum threads:

Web UI errors (too many open files) while runing a balance

Hey folks, I added a couple of drives and started a balance last Friday. I noticed that it was proceeding slowly so I shutdown the rockons service and just let it run. Last night (Sunday), I noticed that I was receiving lots of errors just trying to navigate the Web UI, and that one CPU core was maxed at 100%, and when I was eventually able to load the status page, it showed only 9% complete! So, I poked around on here and found Balance is really disruptive - running the command quoted there …

and

Error 24 Too many open files causes scheduled scrub job to fail and prevents login to web UI

Hi My scheduled scrub job has not completed for the last few runs. It terminates with Error 24 Too many open files. Likewise, I occasionally get the same error message when logging in to the web UI. Rebooting fixes the logon problem, but the problem returns after some (up)time. I installed and ran lsof as recommended in in another topic, I could not make much sense of the output… I did however notice many lines like these for gunicorn and other packages: gunicorn 10307 10324 root 45u …

and

Support server down / GUI crashing

Hi; I’ve been trying to diagnose some user permission errors, that seem to coincide with the gui crashing. I tried to submit via support but the server was not responding? The traceback is here: Traceback (most recent call last): File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception yield File “/opt/rockstor/src/rockstor/storageadmin/views/appliances.py”, line 41, in get_queryset self._update_hostname() File “/opt/rockstor/eggs/Dj…

The nearest we have to a work around, resolution of sorts, is from @erisler’s excellent exposition in the following forum post:

Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files

@peter appears to be on to something - it may also be related to running AFP…read on… I ran the following in ssh while seeing this error in the webui to find the processes with the most open files (credit: https://stackoverflow.com/questions/21752067/counting-open-files-per-process): cd /proc; find -maxdepth 1 -type d -name '[0-9]*' -exec bash -c "ls {}/fd/ | wc -l | tr '\n' ' '" \; -printf "fds (PID = %P), command: " -exec bash -c "tr '\0' ' ' < {}/cmdline" \; -exec echo \…

However their proposed fix was I think though to be a ‘cover-up’ of sorts rather than a core fix: see @Flyer’s comments at the time in their associated pull request:

https://github.com/rockstor/rockstor-core/pull/1934

If we could have some more informed input / reviews on the pull request I could consider merging this pull request as a work around is better than a non existent ‘fix’. But I’m afraid on the web tech involved here I’m in need of more input to know the true ramifications and don’t currently have the time to research further due to other ongoing Rockstor concerns. So if anyone familiar with this could have a look at @erisler proposal and see if it ‘fits our bill’ that would be great. As mentioned @Flyer had reservations and I defer to their knowledge in this area so leaving as is until we have more input on this.

Hope that helps to clarify where we are on this issue and if anyone familiar with this area of Rockstor tech could chip in that would be much appreciated as it’s a long standing issue and quite a frustration for those needing AFP: which we are toying with the idea of dropping anyway, as per openmediavault, so that is a context of sorts.