Web GUI after update to 3.9.2.49 not working properly

[phillxnet (Philip Guyton)]

@Flox @upapi_rockstor @Bazza @Warbucks More info on this one:

@phillxnet

Still a bit of a mystery this one as only seems to affect a few folks.

I may have a reproducer / more info here using chrome version:
Chrome Version 74.0.3729.169 (Official Build) (64-bit)
With repeated:
“WebSocket connection to ‘’ failed: WebSocket is closed before the connection is established.” in Dev console.

Where as the same Rockstor install via:
Firefox 60.8.0esr (64-bit)
Works as intended. But I have seen some dashboard freezes but no Web-UI pauses or blank Web-UI top right.

Their is an intention to completely replace the way the dashboard works as it has tripped us up a few times in the past. But for this we need to move to Python 3 and Django 3: both non trivial tasks. All doable in time though.

Can folks having issues with 3.9.2-49 showing a mostly blank top right Web-UI and a bunch of empty dashboard widgets:

Chrome:

and some really annoying 10-30 second pauses upon visiting any page !!!

Please report your findings using Firefox.

Firefox:

If you find, having visited the Web-UI with Chrome, that the CPU load in Rockstor is consistently high their after (data-collector spinning away to itself), I have seen this once so far, then a reboot or possibly a rockstor.service stop start, as in my previous post, should calm things down again.

@Flox I’m also seeing this Chrome incompatibility on our pre-release testing channel openSUSE rpms so fairly generic problem.

[Bazza (Barry Powell)]

My report of my installation working correctly was premature… it is working in firefox (but this has an existing browser cache. I’ll clear this shortly and retest) However in chrome which has never been used against my installation, exhibits the partial GUI problem. I also get the same issue when I test from an Android tablet.

[upapi_rockstor (Ulrich)]

Just to add to your findings: I’ve had this problem using

• Opera (63.0.3368.107)
• Chrome (77.0.3865.90 (Offizieller Build) (64-Bit))
• Edge (44.18362.387.0)

[Bazza (Barry Powell)]

If you would like any specific information, please let me know but it may take a day or two for me to respond. My environment, if this helps.

Client laptop running Fedora 30 :-

Firefox - 69.0.1 (64 bit) - Appears to be working, except that Web GUI is much slower than before and is causing firefox generate a high cpu load on my laptop. I am seeing the following recurring errors from the Firefox dev console (changed websocket IDs)

Firefox can’t establish a connection to the server at wss://rockstor/socket.io/?EIO=3&transport=websocket&sid=################. socket.io.js:6605:34
The connection to wss://rockstor/socket.io/?EIO=3&transport=websocket&sid=################# was interrupted while the page was loading. socket.io.js:6605:34

Chrome - Version 77.0.3865.90 (Official Build) (64-bit) - I am seeing the same errors in the dev console as reported.

My Rockstor server:

HP Gen 10 Microserver
Rockstor 3.9.2.49 however /etc/redhat-release still says “Rockstor release 3.9.23 (Core)”
Linux: 4.20.12-1.el7.elrepo.x86_64 (as latest Rockstor kernel is not compatible with this hardware)

Very high CPU usage by data-collector process

Recent updates

[root@rockstor ~]# yum history
Loaded plugins: changelog, fastestmirror
ID | Login user | Date and time | Action(s) | Altered

39 | root <root>              | 2019-10-05 06:25 | Update         |    1   
38 | root <root>              | 2019-10-02 16:37 | Update         |    1 EE  < Rockstor update
37 | root <root>              | 2019-09-27 06:50 | Update         |    1   
36 | root <root>              | 2019-09-18 08:42 | I, U           |  235   < sizeable CentOS changes

yum history info 38 - successful, except for all the eggs warning (not copied here)
Loaded plugins: changelog, fastestmirror
Transaction ID : 38
Begin time : Wed Oct 2 16:37:04 2019
Begin rpmdb : 570:7d89451c4119e4668c61b06988b6357481ae9fb5
End time : 16:37:11 2019 (7 seconds)
End rpmdb : 570:5362908611838f4d8830eb14ecda0eb8aac5c714
User : root
Return-Code : Success
Command Line : --setopt=timeout=600 -y update
Transaction performed with:
Installed rpm-4.11.3-40.el7.x86_64 @base
Installed yum-3.4.3-163.el7.centos.noarch @base
Installed yum-plugin-fastestmirror-1.1.31-52.el7.noarch @base
Packages Altered:
Updated rockstor-3.9.2-48.x86_64 @Rockstor-Stable
Update 3.9.2-49.x86_64 @Rockstor-Stable
Scriptlet output:
1 systemctl daemon-reload done

Best regards,

Barry

[Warbucks]

Yes firefox (android) looks fine and I is pretty snappy navigating the WebUI.

Chrome(windows or android) on the other hand is same as the OP and I had previously.

In firefox while the CPU load section does jump up to 95% initially. After a few seconds it seems to settle to ~5% with a few bumps up to 25% on one of the cores. Which with my Intel DN2800mt is pretty typical i believe.

[phillxnet (Philip Guyton)]

@upapi_rockstor Thanks for the other browser tests.
I believe they are all now based on webkit so essentially the same browser base as chrome.

@Bazza Thanks for the nice system report.

@Bazza

Rockstor 3.9.2.49 however /etc/redhat-release still says “Rockstor release 3.9.23 (Core)”

Yes, no worries their. The updating of redhat-release and other os version indicators was never really maintained. We are to leave these alone in our comming openSUSE bases as it just caused incompatibilities when folks tried to compile/install stuff and unsurprisingly that stuff new nothing of “rockstor 3” etc.

Your Firefox report is currently an anomaly but is a new shiny version so that may be triggering something that looks pretty similar.
Try closing all browsers accessing your Rockstor box, reboot the rockstor box, then re-try with your version of Firefox.

@Warbucks Re:

@Warbucks

Yes firefox (android) looks fine and I is pretty snappy navigating the WebUI.

Yes this is my experience on an openSUSE desktop system.

@Warbucks

In firefox while the CPU load section does jump up to 95% initially. After a few seconds it seems to settle to ~5% with a few bumps up to 25% on one of the cores. Which with my Intel DN2800mt is pretty typical i believe.

Yes, upon first visiting the Dashboard it does some yum / package checks and these often peak and then sustain a partial load on CPU but should then die off. 1% on moderate CPU cores is common for data-collector. But client CPU is hit pretty hard with our current dashboard.

Thank to all for your reports. I’m still not able to reproduce major slowdowns in Firefox and 3.9.2-49 should be noticeably more responsive than 3.9.2-48 and all for some time before it so any slowdown is most likely this browser python-enginio / gevent incompatibility or possibly quotas being re-enabled by a docker-ce install or the like. Normally only relevant on a re-install though and when then moving to our newer docker-ce from the prior version.

Linking for context to a prior pull request that ended up not being required as python-engineio ended up upgrading again and what problems we saw back then disappeared again. Looks like we may be hitting something similar again. This situation is complicated by our still using Python 2.7 so we do have some technical debt related issues here:

https://github.com/rockstor/rockstor-core/pull/1996

I’ll play with some of these library versions again and if a library pin or 2 works then I can roll out another stable rpm to hopefully get this sorted again.

If anyone else gets around to building from source to play with these library versions then do post here. I’m currently working on Web-UI upgrade within our openSUSE variants to help with moving in that direction but should be finished pretty soon. Their after I’ll have a look myself and if any joy will roll and release another stable rpm. My bet currently is on the same library pinning as was indicated in that pr of mine referenced above. So that’s probably where I’ll start when I’m done with my current task.

Apologies to all for the inconvenience, and the current work around looks to be:
“Use Firefox”,
at least mostly. But yes, almost all other browsers are webkit so we need to get this sorted.

Post here if you end up playing with source builds and newer / older library versions. But note that this should not be done on any production systems as a fresh source build will wipe the central Rockstor db.

[Flox]

@phillxnet

If anyone else gets around to building from source to play with these library versions then do post here. I’m currently working on Web-UI upgrade within our openSUSE variants to help with moving in that direction but should be finished pretty soon. Their after I’ll have a look myself and if any joy will roll and release another stable rpm. My bet currently is on the same library pinning as was indicated in that pr of mine referenced above. So that’s probably where I’ll start when I’m done with my current task.

A quick test here would seem to confirm your bet: pinning the version to 2.3.2 as per your PR #1996 seems to restore the header and dashboard activity (in chrome, after a clean source build):

I do not see any web socket error in Chrome dev console either (which I could reproduce prior to python-engineio version pinning).

I haven’t experienced high CPU usage by data-collector before so I can’t test whether these disappear as well, but given the header appears restored combined to the absence of websocket error, I would believe this would be fixed as well…

Still need more test to make sure nothing is broken but look like you were right.

[phillxnet (Philip Guyton)]

@Flox Thanks for the initial testing re:

… pinning the version to 2.3.2 as per your PR #1996 seems to restore the header and dashboard activity (in chrome, after a clean source build):

I’ve now re-opened that pr and commented their that it has re-occured. Might as well re-use it as it has quite a bit of detail in. If further testing with this pinning pans out as OK, I’ll pop it in and do a quick openSUSE testing release and if that check out we can release another Stable version with this included.

Cheers for confirming this as a potential fix, if all is well we should have this sorted in the next few days. Thanks folks for the feedback. I’m looking forward to having a ‘proper’ testing channel again soon. More on that in time .

[phillxnet (Philip Guyton)]

@upapi_rockstor, @Bazza, @Warbucks, @Flox, and @wyoham Quick update.

@phillxnet

… if all is well we should have this sorted in the next few days.

3.9.2-50 has just been released with the Chrome Go Slow fix in place.

Thank folks:

Linking to the release post:

3.9.2-# Stable Channel Changelog

3.9.2-50 Merged 6th October 2019 Released 7th October 2019 This one is predominantly a ‘HOT FIX’ for our Web-UI having 10-30 second pauses in Chrome and similar browsers. Firefox seemed to be unaffected, at least some versions. I have also included a couple of unrelated contributions to help with addressing our backlog of improvements. We welcome and thank @magicalyak as a first time rockstor-core contributor but as many of you will know, is very active in our Rock-ons rockon-registry repo. …

Hope that helps.