Howdy, title pretty much says it all. I’ve tried updating the Rockons but I don’t really get a helpful error message that I can trace down:
<strong>##### Houston, we've had a problem.</strong>
Unknown internal error doing a POST to /api/rockons/update
That’s literally all it says when I try to hit the update button.
Here is what the response is from the api/rockons/update command:
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.20.1</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
Some help would be appreciated, there isn’t an error I can see that I can trace down my self.
I don’t see any errors coming through /var/log/* or /opt/rockstor/var/log/*
Both remain errorless while I try to run the update; so I’m not sure where to look next.
Note, this is a fresh install, I tried to update to the new version but I’m running into hardware conflicts with Suse; it doesn’t see my hard drives and I’m not super interested in trying to over come this issue; so I had to roll back to CentOS since that is fully compatible with my hardware.
With this fresh install, I also ran into a certificate error trying to update the rockons for the first time; I had to rely off this post to help:
And I also had to update /opt/rockstor/src/rockstor/storageadmin/views/config_backup.py to have the fix line 255 isdir(cb_dir) fix so I can import from a backup. (You guys should also restrict the upload to tar.gz, so the user doesn’t extract the json from inside and try to upload that if an error occurs, like I did), it accepts the upload, it posts things that you can do with the upload but it really doesn’t work, the system can’t use the json and it just silently errors while the user waits. I only caught it because I was trying to figure out what was going on with the rockons.
@JustARockStorUser welcome to the Rockstor community forums. Are you just getting started on Rockstor? If yes, have you considered to start up with the new OpenSUSE 4.x version instead, downloads can be found here:
Mostly asking, since most of the focus now is on the new platform. But it also looks like you’ve might found the solution to your current issue on the forum already.
yes, my hardware doesn’t appear to be supported by Suse at least not without further work (even though Suse is supported by hardware). When I try to install, the first thing I get is an error mentioning it can’t find my drives; so I tried to find out what was going on, didn’t find a good solution … the only thing I could find is that I need to have an empty drive (which I tried, 3 or 4 times), so I just went to the last non-suse install.
And no, I’m not just getting started, I’ve been using Rockstor for a few years. I’m just trying to re-install to recover from like my 10th or 11lth btrfs complete system failure. These errors occur frequently and often require me having to re-do my system from scratch. It’s quite annoying but I have learned not to stick anything important on a btrfs filesystem as it’s easily corruptible (or appears to be in my case). I actually experienced a btrfs failure just after I made the initial post, so I’m having to re-do it again. My drives are fine, they all pass every check I can do on them.
All I have done is update existing packages; and run the backup file (with the line 255 fix to allow the backup file to be imported). I created my users and the shares. Ran the backup file again (to create the samba shares). I transferred my backups over (which is all my docker data) and tried to then add the rockons; at this point, I did the ca cert/sed stuff.
Now I’m back to Unknown internal error doing a POST to /api/rockons/update
I have run a tail -f /var/log/*
and a tail -f /opt/rockstor/var/log/*
Neither of these produces any data during this time.
I assume it is supposed to be hitting the guiunicorn thing (which says it is running and listening on 127.0.0.1:8000 … but I don’t know how to check to see if this is working or doing what it should be doing.
okay, I wiped it again, this time, I’m just not going to update, updating seems to be what the problem is … but between the ‘new install’ and ‘today’ there’s a lot of packages available for update, I think like 800; so trying to narrow down which package isn’t going to be fun. Would be super helpful if there was something I could do to enable debugging on this to see what is failing vs silently failing (as in not producing anything in a lot that could give anyone any idea what is going on).
I also checked /opt/rockstor/src/logs/ … this is a tz which contains a bunch of log files, I extracted this and saw nothing useful here that would help anyone understand what is failing.
The installer you are using is now 4 years old, hence the massive amount of updates, work needed on certificate, and the back-config bit. Any the core of your update problem is internet speed during the Rock-on update. Also we no longer support the older CentOS version and haven’t released any updates on our side since ages. Copied from the downloads page:
Last Testing Channel release (3.9.1-16) - November 2017
Last Stable channel released (3.9.2-57) - April 2020
Also the reasone we moved from our CentOS base was our failure to keep on top of the kernel updates. We now have openSUSE/SuSE on that one which is great, give they now default to using btrfs. So there are literally years of improvements between our 4.10 / 4.12 kernels on the CentOS elrepo side and our new upstream 5.12 kernels with btrfs backports.
Even so your report of many repeated btrfs failures is alarming and uncommon. But likely best approached by sorting the openSUSE compatibility bit. But you also likely have a memory issue on your machine. Take a look at our Pre-Install Best Practice (PBP):
specifically the memtest side of things. You really shouldn’t be having such difficulties.
So I suggest that you start a new thread explaining exactly your experience re the v4 installer. We may be able to address this or track it down at least as that where you need to be really.
As to the cause of the Rock-on update it’s as mentioned slow internet and a timeout getting the Rock-on info. Take a look at the following thread:
Basically you are experiencing a cummulative time out on the Rock-ons as we now redirect from the old http to the new https. This doubles the round trip chat time and break things on slow internet connections. You also are likely, on the btrfs front, either triggering weakneses in now very old btrfs or have bad memory.
Try starting a new thread on exactly what’s happening on your drives recognition issue re the v4 installer as I’d like to get that sorted for you as then all other things go away. You can import your existing pool and even the config if need be. But will then have the newer https rock-on update so not incure the timeout and have up-to-date certs and likely be OK on the config back-up restore issue. But it won’t address any bad memory of course ;). Also note that the new installer doesn’t, by default, show drives that are over a set size (5TB) to try and avoid folks accidentally installing on their data drives. I dought it that but that’s why I think we should concentrate on helping you get the new v4 in. Otherwise it’s all up-stream swimming there and is basically dealing with stuff that is either fixed years later (ago) and we can’t improve the older CentOS variant as we no longer release updates for it. If we can help get you on to v4 then it can hopefully help all others experiencing a similar incompatibility. By rights, given the newer kernel, the compatibility should be much better.
Try staring a new forum thread with the specifics of your hardware and how the v4 installer is failing you. There is also the more extreme option of modifing an openSUSE Leap 15.3 server install and adding the rockstor testing repo (see the downloads page for the details). You can then subscribe to stable is that is your wish.
But before anything else try a memory test. But your remaining problem is the http → https time outs but given the long list of effort just to get up-an-running with something that it’s difficult to help you with anyway it would seem that approaching the v4 install difficulties or a work around such as taking the openSUSE server install approach and retrofitting Rockstor will be far less frustrating and get you all new and shiny on the install front. A far more gratifying situation than hand patching a 4 year old install with as you say 800 updates. If you need a custom kernel options for instance to get the v4 installer working then you can now build your own with that. See our fairly new:
And once build, with your hypothetical required customnization, you then get zero updates required upon install as during the DIY installer build all pending updates are applied. Just a thought as you are now familiar with bare metal recovery and having to drop back 4 years is not good really. Especially since we are no longer able to build on the now way old CentOS base.
Hope that helps and lets see if we can get you on some form on openSUSE Leap first. Also note that there are 2 bases of the installer downloadable as pre-build isos. One based on Leap 15.2 and another based on Leap 15.3. It may be that one works where the other doesn’t. I did see you mention if you have tried them both. Just a thought.
My internet connection is 1.2gigbits; so it’s def. not in relation to my internet. I also upped the timeout on the nginx.conf to be 600 seconds instead of 75. If the slow round trip was the issue, this should have helped.
I also started a new thread for the other install … we will see where that goes.
Would there be a way to simply cut out the need to fetch remote rockons; only show me the rockons from the metastore folder? Every rockon I use has a custom json file in rockons-metastore; I actually don’t use any of the remote ones by default, I have to override them so I can make the required changes to make them function the way I want.
So, what I did, because I don’t use the remote rockons, I commented out the ‘remote calls’ in _get_available(self) from /opt/rockstor/src/rockstor/storageadmin/views/rockon.py
It’s more the round trip query’s from the 70+ calls made during this update to each and every rock-on definition. Out of interest what is your ping time from:
ping -c 5 rockstor.com
PING rockstor.com (68.183.215.172) 56(84) bytes of data.
64 bytes from rockstor.com (68.183.215.172): icmp_seq=1 ttl=45 time=84.3 ms
64 bytes from rockstor.com (68.183.215.172): icmp_seq=2 ttl=45 time=83.7 ms
64 bytes from rockstor.com (68.183.215.172): icmp_seq=3 ttl=45 time=76.2 ms
64 bytes from rockstor.com (68.183.215.172): icmp_seq=4 ttl=45 time=95.7 ms
64 bytes from rockstor.com (68.183.215.172): icmp_seq=5 ttl=45 time=94.2 ms
--- rockstor.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 76.247/86.859/95.783/7.261 ms
The server is in germany.
If these times are too large they end up adding up to too long and we get a time out. It’s not just your connection to your providers gateway to the internet unfortunately. We also have some improvements to be made on our end. And now we have the https in place we can consider doing some server side compression.
Well done on the work around by the way. I suspect if you also changed the url to https for the rock-ons you would also avoid the double round trip as each http was in turn redirected to it’s https version.