Internal error doing a POST to /api/rockons/update

[k0nsl]

Hello friends!

Below is my issue:

Brief description of the problem

Cannot update the rockstor “rock-ons”.

Detailed step by step instructions to reproduce the problem

I was trying to update the rockstor “rock-ons”.

Web-UI screenshot

Error Traceback provided on the Web-UI

##### Houston, we've had a problem.

Unknown internal error doing a POST to /api/rockons/update


Are there any issues with your server? This has been going on now for about a week, but now I decided to post at the forum about it. Everything else appears to be working on my end.

Faithfully,
-k0nsl

[phillxnet (Philip Guyton)]

@k0nsl Welcome to the Rockstor community and thanks for the report.
Re:

@k0nsl

I was trying to update the rockstor “rock-ons”.

So I’m assuming that all you did was press the “Update” button in the top right then?

@k0nsl

Unknown internal error doing a POST to /api/rockons/update

The “POST” part here indicates that the error was during an update of the database, when on occasions the server side has run out of connections we get a timeout message so I don’t think this is that. And we have made some changes in the more modern code to remove the number of redundant checks that we were making on the server side so that should happen less in the future.

@k0nsl

This has been going on now for about a week

This is quite curios. Does your rockstor_rockstor pool look to be rw still or has it gone read only as this would cause this same issue potentially as the post (db write) would be unable to complete. Or you system drive could be low on space, but again that would have other knock-on.

There is a pending issue related to an upstream update to python-tornado that could be breaking our background task manager that does some of the Rock-ons stuff. There may be some clues in the following file:

/opt/rockstor/var/log/supervisord_ztask-daemon_stderr.log

The command line tool “less” will help you view this file. But this issue has only been reported so far by @freaktechnik and is related to our “Build on openSUSE” variant:

Ztaskd broken after leap updates?

So I just restarted my NAS after 60 days of uptime. And weirdly ztask was very broken afterward. I’ve tried going through potential packages that may have broken it (pyzqm, tornado etc.) but I’ve only been able to fix it by essentially applying the patch from https://github.com/leapcode/bitmask_client/pull/932/files to django_ztask. Obviously this is a bandaid at best, but it made my ztask no longer crash loop. The stack trace is one that’s shown up on here quite a bit already: Traceback (most …

But it has also been seen on customised CentOS installs where additional repo additions have brought in newer versions of python-tornado. @maxhq tracked this one down quite some time ago:

Can't stop Rockons or install new ones

In case anyone gets the same message on a custom installation: My problem was another Yum repo (saltstack-repo) which installed a newer version of python-tornado that caused the error. I solved it with yum downgrade python-tornado.

But these look different from your report currently.

Have you applied any upstream updates that coincide with this recent failure?

Hope that helps, and let us know if you find anything suspicious in the logs that coincides with this message appearing in the Web-UI.