Web UI errors (too many open files) while runing a balance

[nfriedly (Nathan Friedly)]

Hey folks, I added a couple of drives and started a balance last Friday. I noticed that it was proceeding slowly so I shutdown the rockons service and just let it run.

Last night (Sunday), I noticed that I was receiving lots of errors just trying to navigate the Web UI, and that one CPU core was maxed at 100%, and when I was eventually able to load the status page, it showed only 9% complete!

So, I poked around on here and found Balance is really disruptive - running the command quoted there to disable quotas helped the CPU load and greatly sped up the balance - this morning it had gone from 9% to > 50% complete.

However, but I’m still experiencing a lot of issues while just trying to navigate the web UI. Different commands and urls lead to the issues, but they basically all boil down to either “[Errno 24] Too many open files” or else nondescript “unknown internal errors.” (Presumably those are also be due to too many files errors.)

Here’s a couple of examples:

Houston, we’ve had a problem.
Exception while running command([‘/usr/bin/hostnamectl’, ‘–static’]): [Errno 24] Too many open files

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 41, in get_queryset
     self._update_hostname()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 48, in _update_hostname
    cur_hostname = gethostname()
  File "/opt/rockstor/src/rockstor/system/osi.py", line 760, in gethostname
    o, e, rc = run_command([HOSTNAMECTL, '--static'])
  File "/opt/rockstor/src/rockstor/system/osi.py", line 107, in run_command
    raise Exception(msg)
  Exception: Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files

or

Houston, we’ve had a problem.
Unknown internal error doing a GET to /api/shares?page=1&format=json&page_size=5000&count=&sortby=name

Lots of different commands and URLs get one of those errors, but it seems pretty consistent - like 4/5 refreshes get an error rather than actually displaying the page I want.

Is this a common / known issue, or is it something specific to my server?

[phillxnet (Philip Guyton)]

@nfriedly Hello again:

@nfriedly

“[Errno 24] Too many open files” or else …

@nfriedly

nondescript “unknown internal errors.” (Presumably those are also be due to too many files errors.)

Yes quite likely I would say.

@nfriedly

Is this a common / known issue, or is it something specific to my server?

It is a know issue but is not one that affects all configurations, and the main cause is as yet unknown. Any progress you can make on the cause would definitely help. We have another forum thread open that has other’s contributions in this area but as of yet no firm progress. I haven’t see it on any of my Rockstor instances as of yet.

Take a look at the following forum report by @peter:

Error 24 Too many open files causes scheduled scrub job to fail and prevents login to web UI

Hi My scheduled scrub job has not completed for the last few runs. It terminates with Error 24 Too many open files. Likewise, I occasionally get the same error message when logging in to the web UI. Rebooting fixes the logon problem, but the problem returns after some (up)time. I installed and ran lsof as recommended in in another topic, I could not make much sense of the output… I did however notice many lines like these for gunicorn and other packages: gunicorn 10307 10324 root 45u …

and I’ll link the currently open issue referenced in that thread here also for ease:

https://github.com/rockstor/rockstor-core/issues/1656

I’ve also just linked back to this and the above mentioned forum thread in that issue for context.

Hope that helps and please feel free to contribute any info / pointers on that issue.

[peter]

Hi

I can confirm that both scheduled and interactive scrub jobs now completes successfully on 3.9.2-1.

The Web UI problem “too many files open” is still there. On a newly rebooted system I can use the web UI but after several hours uptime the problem is back.

[wwwizzarrdry]

Kind of irritating that I can’t figure this out for myself. For what it’s worth, I just discovered that I can reliably trigger this error in the WebUI by opening Plex (via rockon) >> then triggering the “Analyze” function on my Movies, TV, Music and Photos libraries from the Plex WebUI.

So, perhaps some plex maintenance routines might be the culprit?

[TexasDex]

I just got this error on a new Rockstor VM with the following sequence of actions:

1. LUKS format two disks, and use them as a RAID1 pool
2. After a few hours, LUKS format a third disk, add it to the pool without changing RAID level.
3. Let it sit for a night or two.

The VM has 4GB of RAM, a 100GB base disk, 3x32GB pool disks, 2 CPUs, and the only Rock-on I have enabled is the http to https redirect.

Do we just need to up the ulimits? I’m handy with the CLI and can give that a shot if necessary.

[phillxnet (Philip Guyton)]

@TexasDex Welcome to the Rockstor community and thanks for the additional report.

Yes we still see this occasionally, you might like to take a look at the discussion in the earlier sighted open issue for this:

https://github.com/rockstor/rockstor-core/issues/1656

From that I suspected (in that thread) that it was AFP related. Do you also have AFP enabled?
Or did you have the Web-UI open over a long period?

I don’t think the LUKS element is relevant here though.

You might also like to take a look at the discussion / proposed fix by forum member @erisler in the following proposed pull request:

https://github.com/rockstor/rockstor-core/pull/1934

which in turn links back to the following forum thread:

Exception while running command(['/usr/bin/hostnamectl', '--static']): [Errno 24] Too many open files

@peter appears to be on to something - it may also be related to running AFP…read on… I ran the following in ssh while seeing this error in the webui to find the processes with the most open files (credit: https://stackoverflow.com/questions/21752067/counting-open-files-per-process): cd /proc; find -maxdepth 1 -type d -name '[0-9]*' -exec bash -c "ls {}/fd/ | wc -l | tr '\n' ' '" \; -printf "fds (PID = %P), command: " -exec bash -c "tr '\0' ' ' < {}/cmdline" \; -exec echo \…

Which also links in the AFP element.

I’m afraid my knowledge in these web tech areas is far surpassed by @suman, @Flyer, and I think we are awaiting @Flyer’s final comment on the above pull request. Please do take a look at these links to see where we are up to so far on this one as it would be great to finally get to the cause on this one.

Hope that helps.

[TexasDex]

Yeah, I had a tab with the web UI open the entire time.

And yes, I believe I do have AFP enabled.

[nandor]

+1

Also happening to me.