Unifi? - Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files

[Please complete the below template with details of the problem reported on your Web-UI. Be as detailed as possible. Community members, including developers, shall try and help. Thanks for your time in reporting this issue! We recommend purchasing commercial support for expedited support directly from the developers.]

Brief description of the problem

All pages (dashboard, storage, system, etc.) Are unable to display with the ‘Houston, we have a problem’ error.

Detailed step by step instructions to reproduce the problem

Using multiple browsers on same local network as Rockstor NAS attempt to hit any page.

I think it’s the Unifi rock-on … it’s using an insane amount of files (it looks like the process is bouncing but not cleaning up after itself…)

[root@rockstor ~]# lsof | grep unifi | wc -l

103934

Web-UI screenshot

Error Traceback provided on the Web-UI

Traceback (most recent call last): File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception yield File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 41, in get_queryset self._update_hostname() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner return func(*args, **kwargs) File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 48, in _update_hostname cur_hostname = gethostname() File "/opt/rockstor/src/rockstor/system/osi.py", line 766, in gethostname o, e, rc = run_command([HOSTNAMECTL, '--static']) File "/opt/rockstor/src/rockstor/system/osi.py", line 113, in run_command 'Exception while running command({}): {}'.format(cmd, e)) Exception: Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files [root@rockstor ~]# lsof | grep unifi | wc -l lsof: no pwd entry for UID 911 <-- This line repeats at least 100 times. 106088 [root@rockstor ~]# ps -ef | grep unifi root 23299 23282 0 Nov19 ? 00:00:00 bash /usr/local/bin/docker-entrypoint.sh unifi root 23335 23299 0 Nov19 ? 00:00:00 unifi -nodetach -home /usr/lib/jvm/java-8-openjdk-amd64 -classpath /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/unifi/unifi.pid -procname unifi -outfile /unifi/log/unifi.out.log -errfile /unifi/log/unifi.err.log -cwd /usr/lib/unifi -Dunifi.datadir=/unifi/data -Dunifi.logdir=/unifi/log -Dunifi.rundir=/var/run/unifi -Xmx1024M -Djava.awt.headless=true -Dfile.encoding=UTF-8 com.ubnt.ace.Launcher start root 23336 23335 0 Nov19 ? 00:02:17 unifi -nodetach -home /usr/lib/jvm/java-8-openjdk-amd64 -classpath /usr/share/java/commons-daemon.jar:/usr/lib/unifi/lib/ace.jar -pidfile /var/run/unifi/unifi.pid -procname unifi -outfile /unifi/log/unifi.out.log -errfile /unifi/log/unifi.err.log -cwd /usr/lib/unifi -Dunifi.datadir=/unifi/data -Dunifi.logdir=/unifi/log -Dunifi.rundir=/var/run/unifi -Xmx1024M -Djava.awt.headless=true -Dfile.encoding=UTF-8 com.ubnt.ace.Launcher start root 23867 22659 0 11:47 pts/0 00:00:00 grep --color=auto unifi root 25255 23336 0 Nov19 ? 00:06:31 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms1024M -Xmx1024M -XX:ErrorFile=/usr/lib/unifi/logs/hs_err_pid%p.log -Dapple.awt.UIElement=true -jar /usr/lib/unifi/lib/ace.jar start root 26159 25255 0 Nov19 ? 00:15:18 bin/mongod --dbpath /usr/lib/unifi/data/db --port 27117 --unixSocketPrefix /usr/lib/unifi/run --logappend --logpath /usr/lib/unifi/logs/mongod.log --nohttpinterface --bind_ip 127.0.0.1

Do the docker container logs provide any insight as to why Unifi might be bouncing?

docker container logs <container name or id>

You can get the container name from

docker ps

You can also enter the container and run a local top with

docker container exec -it <container name or id> top
1 Like

Hi Haioken -

I stopped all containers other than plex (bare minimum I want working). Did see that OpenVPN server container was bouncing. Going to allow this to soak for some time and see if it comes up again.

Thanks for the gumdrops :wink: to find my way.

@sfinney

I’ve seen bouncing containers cause the ‘Too many open files’ issue a couple of times. (And witnessed myself a bouncing Gitlab eating all memory and swap on a 32Gb system!)

Regarding the oVPN container bouncing, have you completed the oVPN rockon post-install tasks as noted in the documentation?

The container will definately bounce until the ovpn-initpki script has been run, this is probably as good a place as any to start investigating further.

HTH,