Rockon update fails

[DrC (Cam)]

Is there an issue with rockon update failing?

I just did:

• a fresh install of 3.9.1
• Updated
• Rebooted
• setup and enabled rockon
• went to the rockon page, then ‘all’, then clicked update.
• result: took ages and then popped up the empty message below

Then tried the same again (full reinstall, but i activated my rockstor with a key and enabled stable), same result.

In the logs I see:

 ==> /opt/rockstor/var/log/gunicorn.log <==
[2021-07-16 21:10:20 +0000] [2881] [CRITICAL] WORKER TIMEOUT (pid:4552)
[2021-07-16 21:10:20 +0000] [4552] [INFO] Worker exiting (pid: 4552)
[2021-07-16 21:10:20 +0000] [7117] [INFO] Booting worker with pid: 7117

==> /opt/rockstor/var/log/rockstor.log <==
[16/Jul/2021 21:10:21] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockstor_rockstor/home']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']

==> /opt/rockstor/var/log/gunicorn.log <==
[2021-07-16 21:10:27 +0000] [2881] [CRITICAL] WORKER TIMEOUT (pid:5552)
[2021-07-16 21:10:27 +0000] [5552] [INFO] Worker exiting (pid: 5552)
[2021-07-16 21:10:27 +0000] [7225] [INFO] Booting worker with pid: 7225

[DrC (Cam)]

I guess if noone else is having this issue it must be something about my setup then?

[Hooverdan (Dan W.)]

Can you check the file:
/opt/rockstor/var/log/supervisord_ztask-daemon_stderr.log

on the 3.9 version there will always be some error message about setuptools (since they are out of maintenance with the version that Rockstor 3.x uses), but there might be some other clues why you’re having the above “post” issue.

[DrC (Cam)]

Nope. Empty. Just reinstalled on a new box, didnt patch (so right off the iso) and then added a share for rockons, went there and hit update. Resultant (time sorted) logfiles are:

[root@macmini-old /]# ls -latr /opt/rockstor/var/log/
total 36
-rw-r--r--  1 root root    0 Jul 17 06:40 supervisord_nginx_stdout.log
-rw-r--r--  1 root root    0 Jul 17 06:40 supervisord_nginx_stderr.log
-rw-r--r--  1 root root    0 Jul 17 06:40 supervisord_gunicorn_stdout.log
-rw-r--r--  1 root root    0 Jul 17 06:40 supervisord_gunicorn_stderr.log
-rw-r--r--  1 root root    0 Jul 17 06:40 supervisord_data-collector_stdout.log
-rw-r--r--  1 root root    0 Jul 17 06:40 supervisord_ztask-daemon_stdout.log
-rw-r--r--  1 root root    0 Jul 17 06:40 supervisord_ztask-daemon_stderr.log
drwxr-xr-x. 1 root root  620 Jul 17 06:40 .
-rw-r--r--  1 root root  182 Jul 17 07:29 ztask.log
-rw-r--r--  1 root root 2657 Jul 17 07:30 supervisord.log
-rw-r--r--  1 root root 8356 Jul 17 07:36 supervisord_data-collector_stderr.log
-rw-r--r--  1 root root 8315 Jul 17 07:37 rockstor.log
-rw-r--r--  1 root root 1088 Jul 17 07:40 gunicorn.log

It is fetching something. tcpdump shows it dns’ing rockstor.com and then pulling stuff over http from rockstor.com (which is rockstor.com). It seems to download a number of things from there but still gets the same error message.

Im guessing the actual answer is ‘move to v4’?

[DrC (Cam)]

That was a bad bug report, sorry

Here is a better log dump. The qgroups error seems to be periodic and not related to the update request.

tail -f /opt/rockstor/var/log/* /var/log/*


========================= started update request =================================

==> /opt/rockstor/var/log/gunicorn.log <==
[2021-07-18 18:56:38 +0000] [2737] [CRITICAL] WORKER TIMEOUT (pid:2770)
[2021-07-18 18:56:38 +0000] [2770] [INFO] Worker exiting (pid: 2770)
[2021-07-18 18:56:38 +0000] [17117] [INFO] Booting worker with pid: 17117

==> /opt/rockstor/var/log/supervisord_data-collector_stderr.log <==
No handlers could be found for logger "smart_manager.data_collector"

==> /opt/rockstor/var/log/rockstor.log <==
[18/Jul/2021 18:56:39] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockstor_rockstor/home']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']

==> /opt/rockstor/var/log/gunicorn.log <==
[2021-07-18 18:57:18 +0000] [2737] [CRITICAL] WORKER TIMEOUT (pid:2765)
[2021-07-18 18:57:18 +0000] [2765] [INFO] Worker exiting (pid: 2765)
[2021-07-18 18:57:18 +0000] [17235] [INFO] Booting worker with pid: 17235

==> /opt/rockstor/var/log/rockstor.log <==
[18/Jul/2021 18:57:40] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockstor_rockstor/home']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']

[DrC (Cam)]

Just installed the 3.9.1 from an iso onto virtualbox and i get exactly the same behaviour. So i see no rockon updates working across 3 machines now.

Can someone confirm this?

• Install 3.9.1
• setup share for rockon, configure and turn on service
• goto rockons and click [update]

For me it just spins wheels for a while and then fails with a blank error message.

[DrC (Cam)]

Just built 4.0.7 in virtualbox and started it up. Same behaviour.

It must be me. I cant believe 3.9.* and 4.0.7 updates are broken and noone else is seeing it.

---

wget rockstor.com/rockons/root.json

Works on both machines (3.9.* and 4.0.7)

I am in Australia so im expecting we have more delay fetching this stuff. The wget of root.json took 1.716s

[phillxnet (Philip Guyton)]

@DrC Hello there.
Re:

@DrC

Just built 4.0.7 in virtualbox and started it up. Same behaviour.

Not necessarily. First off. Well done on building and installing 4. Nice. Now we have an oustanding issue in upstream with regards to docker not liking our IPv4 only environment. We are awaiting a fix in upstream for this. Docker has fixed it but openSUSE Leap is hopefully to get this fix soon. And thus we will too.

Take a look at the following forum thread for the work around (downgrading docker) for now:

Lost ability to start all my rockons after updating system

Brief description of the problem I Lost ability to start all my rockons after updating system. Detailed step by step instructions to reproduce the problem I clicked the blinking update logo Clicked update Closed browser Lost internet connection (due to Pi-hole shutting off) restarted machine after resetting my router (thought my router was bad) Web-UI screenshot [195_] Error Traceback provided on the Web-UI ``` [18/Jun/2021 17:04:07] ERROR [system.osi:196] non-zero code(125) returned b…

It’s quite frustrating for all concerned but downgrading the docker package and pinning / locking that for the time being should get you sorted. The Rockstor 3 development is now legacy and we no loner release updates to that version. But as you say we haven’t had any other reports; at least yet.

So take a look at @sanderweel work around in that thread:

Lost ability to start all my rockons after updating system

I have found the cause of our misery! Apparently by updating our system, a new version of Docker (20.10.6) is installed. I found out (hooray for google) that this version has a problem with disabled IPv6: it ignores this and tries to bind ports using IPv6, even if IPv6 has been disabled on the system. This causes all the errors. The only solutions are: enabling IPv6 or downgrading Docker. Because IPv6 on Rockstor is disabled intentionally, I chose to downgrade Docker and this worked! all my Roc…

and see if that gets you sorted. Hopefully this upstream docker fix will appear in a leap update soon and you can then remove the package lock (see later in that forum thread).

N.B. Re:

@DrC

The wget of root.json took 1.716s

That seems like a long time actually.

...
Length: 3333 (3.3K) [application/json]
Saving to: ‘root.json’

root.json                             100%[=========================================================================>]   3.25K  --.-KB/s    in 0.002s  

2021-07-18 20:21:48 (2.00 MB/s) - ‘root.json’ saved [3333/3333]

From Europe mainland.

So the root cause may be a timeout as root.json is only one of a series of files to retrieve, i.e. one for each Rock-on. To see the current file count see this repo:

GitHub - rockstor/rockon-registry: hosted registry for Rock-ons

And this is served from rockstor.com.

We do have mid term plans to add web server side compression to these files to speed up their download but that’s not going to make a tone of difference. But is should help a little.

Let us know how you get on with the docker downgrade on the Rockstor 4 and then hopefully folks here can come us with something to increase the time out or speed things up some how. What is the hardware you are using for this Rockstor instance?

Hope that helps and thanks for the report.

[DrC (Cam)]

Sorry i posted this in the wrong thread. Can an admin move it to [support]?

[DrC (Cam)]

Im in australia, so a ping to the UK is 330ms. So fetching anything from rockstor.com/rockons/ is at least 660ms because your site redirects to https which means: 1) establish tcp to port 80, get redirect, close 2) open https port and try again.

I see a redirect (http->https) for the root.json and some of the project.json files too.

[DrC (Cam)]

I couldnt help myself so i added a feature request:

Request: better rockon update protocol

Currently: root.json contains a list of: 'human project name' : 'project.json' for each rockon root.json and all project.json files are pulled every update. Failure to pull them all seems to result in an aborted update with no results. This data seems to be held in memory and not written to the rockons share (or maybe it isnt until its fully downloaded so any timeout aborts the whole process) rockon update pulls a meta list from /rockons/root.json (as http and gets redirected to https, …

[phillxnet (Philip Guyton)]

@DrC Hello again.
Re:

Im in australia, so a ping to the UK is 330ms.

Our main server is actually in Europe - Germany - Frankfurt. Was you ping to our server or a UK server.

… because your site redirects to https

Yes a recent inconvinience required for us to move to all https which was a long standing request, recently enacted. See my prior response in your larger post involving you multitude of improvement ideas for the Rock-ons retrieval update system.

Hope that helps and let us know in maybe a fresh post on ping times or the like.

[DrC (Cam)]

I just assumed it was UK because of the currency to subscribe. The ping was to rockstor.com from the bottom bit of australia.

I changed the setting.py url for rockon fetch from http to https and IT WORKED!! Rockon’s loaded! Woohoo! No failure and empty message! They loaded!!!

I submitted a pull request to fix the file conf/settings.conf.in but i didnt make it a branch (sorry, did it before reading the submitting guidelines). I cant work out how to cancel the pull request and fix it.

issue/pull: https://github.com/rockstor/rockstor-core/issues/2304 (warning, not in a branch, sorry)

(And i cant work out how to change the topic of this thread to [solved])

[DrC (Cam)]

For potential future users. The solution for me for ‘cant do rockon update, empty error message’

Try editing: /opt/rockstor/src/rockstor/settings.py

Change http to https in the following line (418 at this version)

    'remote_metastore': 'https://rockstor.com/rockons',