[SOLVED] $ yum update -y breaks rockstor services

phillxnet · January 3, 2017, 5:52pm

@exelan OK, so you now have Rockstor version 3.8.12 which, as suspected, is well old. You are probably going to be better off just re-installing using a 3.8.16-1.iso and re-importing your pool (see Reinstalling Rockstor and specifically the Data Import sub-section) . But you could try the procedure detailed in that post for updating versions prior to 3.8.15 and make sure that you reboot after every yum operation on the rockstor package. But it may be that you simply don’t have the required yum configuration: here I’m assuming you hadn’t first selected an update channel prior to loosing web-ui functionality.

A fresh 3.8.16 install prior to update channel selection has the following in /etc/yum.repos.d/

ls -la /etc/yum.repos.d/
total 32
drwxr-xr-x. 1 root root  276 Nov 15 07:30 .
drwxr-xr-x. 1 root root 3550 Jan  3 09:33 ..
-rw-r--r--. 1 root root 1624 Dec 19  2015 CentOS-Base.repo
-rw-r--r--. 1 root root  649 Dec 19  2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  290 Dec 19  2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 1331 Dec 19  2015 CentOS-Sources.repo
-rw-r--r--. 1 root root  156 Dec 19  2015 CentOS-Vault.repo
-rw-r--r--. 1 root root  957 Jul 23 14:37 epel.repo
-rw-r--r--. 1 root root 1056 Jul 23 14:37 epel-testing.repo
-rw-r--r--. 1 root root  184 Dec 19  2015 Rockstor-Base.repo

And after activating Testing Updates channel it would have:

ls -la /etc/yum.repos.d/
total 36
drwxr-xr-x. 1 root root  318 Jan  3 09:38 .
drwxr-xr-x. 1 root root 3550 Jan  3 09:33 ..
-rw-r--r--. 1 root root 1624 Dec 19  2015 CentOS-Base.repo
-rw-r--r--. 1 root root  649 Dec 19  2015 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  290 Dec 19  2015 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 1331 Dec 19  2015 CentOS-Sources.repo
-rw-r--r--. 1 root root  156 Dec 19  2015 CentOS-Vault.repo
-rw-r--r--. 1 root root  957 Jul 23 14:37 epel.repo
-rw-r--r--. 1 root root 1056 Jul 23 14:37 epel-testing.repo
-rw-r--r--. 1 root root  184 Dec 19  2015 Rockstor-Base.repo
-rw-------  1 root root  213 Jan  3 09:38 Rockstor-Testing.repo

So the Rockstor-Testing.repo has been added which contains the following:

cat /etc/yum.repos.d/Rockstor-Testing.repo 
[Rockstor-Testing]
name=Subscription channel for testing updates
baseurl=http://updates.rockstor.com:8999/rockstor-testing
enabled=1
gpgcheck=1
gpgkey=file:///opt/rockstor/conf/ROCKSTOR-GPG-KEY
metadata_expire=1m

Hope that helps.

exelan · January 3, 2017, 6:13pm

Ah, I see. Yes my /etc/yum.repos.d/ dir is exactly what you posted. Including timestamps. I’m not exactly seeing in the linked documentation on how to activate the Rockstor-Testing.repo? Do you happen to have a direct link to that or perhaps a quick SOP on the steps I should take to try and restore? I’m a little nervous about re-installing via the ISO, but if thats the cleanest and most efficient way to try and solve the problem, I’ll do my best to go that route.

Also, I remember doing a config backup in July, is that just a json reference file or something? Would that help anything?:

$ ll /opt/rockstor/static/config-backups/
total 4
4 -rw-r–r-- 1 root root 623 Jul 27 18:59 backup-2016-07-27-185945.json.gz

Thank you again!

phillxnet · January 3, 2017, 7:58pm

@exelan Normally the appropriate repo is enabled via the Web-UI as described in Update Channels as is activated via the System - Software Update Web-UI page. But given you appear not to have done this you could try:

nano /etc/yum.repos.d/Rockstor-Testing.repo

and cut and paste the contents as I previously pasted for that file: starting with the line "
[Rockstor-Testing]".
Then:

yum clean all
yum update rockstor-3.8.15

As per the instructions linked you would then reboot into 3.8.15 for it to do it’s setup. However in proving this here I then found that rockstor-3.8.15 is not available in the given Testing repo.

yum list available rockstor --showduplicates
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.sax.uk.as61049.net
 * epel: epel.check-update.co.uk
 * extras: mirror.sov.uk.goscomb.net
 * updates: mirror.sax.uk.as61049.net
Available Packages
rockstor.x86_64                                            3.8-12                                               rockstor        
rockstor.x86_64                                            3.8.16-1                                             Rockstor-Testing
rockstor.x86_64                                            3.8.16-2                                             Rockstor-Testing

@suman Is this a Stable channel only version or was it intended to be in Rockstor-Testing repo also? @haoto in the following forum thread seems to have run into the same issue:
Upgrading Rockstor from a broken 3.8-14?

@exelan So for the time being your options are now limited to re-installing. But it might be worth holding out for further information in this thread as if the 3.8.15 package appears in the Testing repo soon then you can continue on from here with the linked instructions.

But it is worth noting that when earlier I installed 3.8-12 and then, in an attempt to reproduce your issue, did a ‘yum update -y’ my test install was not broken and continues to function after a reboot. So given your experience is quite different with missing dirs etc you may have run into something else, given your reported actions.

Sorry to not have been more help and well done on making a config backup, the docs for this can be found in the official docs as Configuration Backup and Restore and yes the file is simply a gzip json format so you could even unzip it a read the settings by eye if it came to it. Although the file ‘as is’ can be uploaded and ‘played back’ to the system in question.

Make sure you retrieve that settings file from the system drive prior to any re-installing though.

Currently given I was unable to reproduce you missing dir’s it may well be. If you first disconnected your data drives after shutting down then there would be no risk of choosing the wrong drive during re-install. Then once you’ve got the install and first reboot and setup done you can then shutdown, re-attached the data drives, bootup and import via the given doc references.

Hope that helps at least a bit.

exelan · January 3, 2017, 11:11pm

@phillxnet

It definitely does help in terms of understanding what happened, I really appreciate it!

I guess I just don’t know where to go from here but perhaps wait for some more information as to what happened to …/bin. I did what you mentioned about adding the testing repo and then doing a rockstor update:

$ yum info rockstor
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile

base: centos.gbeservers.com

epel: ftp.osuosl.org

extras: mirror.tocici.com

updates: mirror.tocici.com
Available Packages
Name : rockstor
Arch : x86_64
Version : 3.8.16
Release : 2
Size : 15 M
Repo : Rockstor-Testing
Summary : RockStor – Store Smartly
License : GPL
Description : RockStor – Store Smartly

So now I’m at 3.8.16 R2, and I haven’t rebooted yet because I’m terrified haha. I checked the /opt/rocktor dir and alas, there is no /bin directory created :\

I also have some verbage in my .bash_profle to save my history and perhaps that contains some extra information about what might have happened?:

https://p.ngx.cc/2c

I tried to create a concise account of what I did, try not to laugh at my admin steps :\

Regards,

Exelan

phillxnet · January 4, 2017, 12:03pm

@exelan Some nice reporting there.

But from your reported steps you should not yet be on 3.8.16-2 as the command ‘yum update rockstor-3.8.15’ fails due to no package by that version currently existing in the testing repo. Unfortunately that Rockstor package version is a requirement to avoid a known db issue that breaks Rockstor when it is upgraded from a pre 3.8.15 version as you had to a post 3.8.15 version (hence my embedded note to @suman). But there is still the missing /bin dir!

Also re the email config it may have been easier to first configure this via enabling email notifications within Rockstor, then adjust what is then generated postfix config wise to your requirements. Just a side note really.

To double check your /opt/rockstor/bin issue here, I deleted my existing 3.8.12 install’s /opt/rockstor/bin and then did a ’ yum update’ to try and duplicate your issue as the upgrade should restore all missing files and in my case at least it did. The install then displays the known db issue and after a reboot complains within the Web-UI accordingly. But still the bin dir was restored as expected.

From this I can only assume you have a corrupt fs as your bin was not restored as expected, and observed here, yet the inadvertent Rockstor package upgrade (putting aside the skipping of the needed 3.8.15 version) went through OK as is evident from your last ‘yum info rockstor’ report.

I at least think that we are at the stage of diminishing returns as the behaviour of your package and it’s associated files are not what I see here on a clean system as described. And I was then able to reset my on purpose 3.8.15 skip via the very heavy handed and config dangerous .initrock method (delete then reboot) but if your install has no bin then you have larger concerns that are likely to hamper further attempts to fix stuff. Easy for me to say I know but given you have the chance to retrieve your previous config via the json.gz file (which you must first scp off to another machine) and that all data is on separate drives (assuming you stored nothing in either of the home or root shares) it may very well be more beneficial to start afresh. Just detach any data drive with the machine powered off prior to following the re-install doc. And make sure to reconnect them all with the machine off after first completing the install and initial first reboot and sign in / hostname setup stage.

But it may of course be worth waiting around a little more to see if we get any more informed approaches to take you from where you are now. But with no bin dir even after a successful reinstall it looks tricky.

As always I hope that helps.

exelan · January 4, 2017, 8:47pm

@phillxnet Hmm, I see. Well, I guess like you said there are 2 options: Patience, or do an OS re-install. I’m not opposed to the re-install since I have a pretty good idea of what my drives are (hoping that they don’t rename them during the install proccess ie; sda->sdb etc based on boot order):

$ lsblk --output NAME,FSTYPE,LABEL,MODE
NAME FSTYPE LABEL MODE
sda btrfs data_pool1 brw-rw----
sdb btrfs data_pool1 brw-rw----
sdc btrfs data_pool1 brw-rw----
sdd btrfs data_pool1 brw-rw----
sde brw-rw----
├─sde1 ext4 brw-rw----
├─sde2 swap brw-rw----
└─sde3 btrfs rockstor_rockstor brw-rw----

What would you recommend to do? I’m not that concerned about getting this back up in a certain timeframe and would be willing to wait for other input. But of course, the longer I stare at empty pools, the more concerned I get that it may just vaporize haha

Thank you so much for all your support!

Ex

phillxnet · January 4, 2017, 10:26pm

@exelan Hello again.

You could take a look at the output from the following command:

ls -la /dev/disk/by-id/

Those are the device names that newer versions of Rockstor use. They are generated by udev and point to the temp-dev names and are re-regenerated on each boot. They are just symbolic links to the canonical sda type names but with the added bonus of being boot to boot stable. You could find your boot devices serial from there (it’s in the name) that way you will know which drive is the system drive from the hardware perspective.

Also from the look of it your system drive is the only one with partition on. So that’s another marker.

Another command that may be of interest is:

btrfs fi show

It should show all the btrfs pools of the drives attached to the system and their device members and their attached or otherwise status, along with some basic size/used info.

exelan · January 8, 2017, 10:11pm

@phillxnet Hi again, and I’m really sorry to bug you again, but I had a quick question.

If I pursue the route of the OS re-install, would I do a fresh install on the OS drive, then boot to it with the RAID pool drives plugged in, then do the config restore that I have saved from July?

I just want to make sure I don’t affect the data on the pools.

Thanks!

Exelan

phillxnet · January 9, 2017, 10:12am

@exelan Hello.

First thing is to copy off your previous config file as it’s currently stored on the system drive so will be lost during the re-install. Not that this config save system doesn’t yet cover all of Rockstor’s settings so it’s not perfect. See the Configuration Backup and Restore for a list of those things it does and doesn’t contain.

Yes, but to guard against mistakes you could disconnect the data drives during this re-install, noting how they are currently connected of course; that way they won’t show up in the installer at all, which removes the possibility of one of them being mistaken for the prior system drive.

After the initial install is complete and after the first reboot where you then setup up the hostname and admin user. You could then upload your config to see how it goes. Some of the share related configs may not be happy, ie samba config re shares, as there wouldn’t be any shares yet due to no pool import yet but you can always shutdown, re-attach the data drives, then boot up and import via the Disks page.

Do note however that we have an outstanding issue with samba config restore:

So this may affect your samba config restore anyway.

But you can always resort to the rather draconian visual inspection of the config save so all is not lost.

The config restore will also error on restoring existing users, ie if you use the same admin username as before; but it should continue on regardless. If you take careful note on what works for you and what doesn’t you could open issues for each so we can get that side of things improved some.

If you are nervous you could always go through all the process, bar the drive re-connection and import that is, in a Virtual Machine environment and upload your config to that so you can see what to expect. Ie on your desktop machine using Something like VMM (linux) or Virtual Box. At least that way you will be more familiar with what to expect and the process in general.You could also submit bugs re you config restore on that basis. The Quick Start section has a slightly dated but still relevant Rockstor install in Virtual Box install video and there is also a link to to the Rockstor in Virtual Machine Manager guide howto.

I like the virtual machine trial idea myself. You only need an 8GB system drive in the VM to install Rockstor and make sure to use the latest Rockstor iso installer. You might find that your particular setup is straight forward enough to setup from scratch anyway and remember that Rockstor still need a reboot occasionally to get things straight after such things as update. This is getting better but it’s worth remembering.

Hope that helps.

exelan · January 11, 2017, 1:22am

@phillxnet Thanks as always. How possible would it be to try and just restore the /bin dir? I assume that the contents are just binaries and not configuration details.

phillxnet · January 11, 2017, 11:11am

@exelan My concern here is that this should already have happened during the package upgrade, ie my own test of deleting my /opt/rockstor/bin then upgrading and it re-appearing, so the fact that it doesn’t show up again on your install is what lead to my concerns. That is that there is something bigger going on.

exelan · January 11, 2017, 5:22pm

@phillxnet Whelp, it looks like rockstor wasn’t even installed somehow? Maybe when messing around with the sendmail / postfix stuff it was removed?

https://p.ngx.cc/8e

So I’m not sure if I should reboot here or not… I doubt it would do anything to the data pools, but I don’t know! Do you happen to have any of your mounts in your /etc/fstab file? Should I be concerned about versioning issues or anything?

EDIT:

Okay, so I got the GUI / Rockstor service up and I’m logged into the updated front end (looks a little different from my previous version) and I have this item loaded:

You are running an unsupported kernel(4.6.0-1.el7.elrepo.x86_64). Some features may not work properly. Please reboot and the system will automatically boot using the supported kernel(4.8.7-1.el7.elrepo.x86_64)

And under pools I can see the primary pool “intact”:

data_pool1 3.62 TB 2.87 TB (79.22 %) raid5 lzo None ata-SAMSUNG_HD204UI_S2H7JD1B109709,ata-WDC_WD1001FALS-00J7B0_WD-WMATV0504493,ata-WDC_WD1001FALS-00J7B0_WD-WMATV0326624,ata-SAMSUNG_HD204UI_S2H7JD1B109708

All of my shares are listed properly and the only thing that doesn’t seem to be listed are my NFS shares, which you mentioned would be removed.

phillxnet · January 11, 2017, 6:26pm

@exelan Well done.
As for the kernel version warning you should find that on next reboot Rockstor’s start up scripts should have reset the boot to use the newer kernel that should also have been installed.

exelan · January 11, 2017, 6:37pm

@phillxnet Thank you for all of your help and leading me to a successful resolution, I (and my data) are in your debt.

Trying a graceful reboot now and once everything gets back online I’ll do a massive backup **whew

EDIT:

Okay so everything seems to be okay, but there are a couple issues popping up here and there. For instance, when trying to click on most of the links under the dashboard, I get an error with it running some of the python scripts as well as what it looks like the OS DB:

Houston, we’ve had a problem.

relation “storageadmin_pincard” does not exist LINE 1: SELECT COUNT(*) AS “__count” FROM “storageadmin_pincard” WHE… ^

        Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 40, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/user.py”, line 105, in get_queryset
return combined_users()
File “/opt/rockstor/src/rockstor/storageadmin/views/ug_helpers.py”, line 69, in combined_users
temp_uo.pincard_allowed, temp_uo.has_pincard = pincard_states(temp_uo)
File “/opt/rockstor/src/rockstor/system/pinmanager.py”, line 135, in pincard_states
pincard_present = has_pincard(user)
File “/opt/rockstor/src/rockstor/system/pinmanager.py”, line 120, in has_pincard
pins = Pincard.objects.filter(user=int(uid_field)).count()
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py”, line 318, in count
return self.query.get_count(using=self.db)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/query.py”, line 466, in get_count
number = obj.get_aggregation(using, [’__count’])[’__count’]
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/query.py”, line 447, in get_aggregation
result = compiler.execute_sql(SINGLE)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py”, line 840, in execute_sql
cursor.execute(sql, params)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py”, line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
ProgrammingError: relation “storageadmin_pincard” does not exist
LINE 1: SELECT COUNT(*) AS “__count” FROM “storageadmin_pincard” WHE…
^

phillxnet · January 13, 2017, 7:31pm

@exelan Hello again, and you are welcome.

Yes pretty sure this is the known issue of db failure to update from too older version of Rockstor such as you had previously. But the fix I referenced earlier in this thread:

requires an interim Rockstor package version of rockstor-3.8.15, which was not available in the testing channel repositories. Hence my previous:

As at some point you upgraded from 3.8.14 to 3.8.16 and ran into this very db migration issue.

Again leaving you back at:

As from my VM here to replicate your issue (bar the missing bin) we see that 3.8.15 is not available in testing updates channel repo still:

yum clean all
yum list available rockstor --showduplicates
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base: repo.uk.bigstepcloud.com
 * epel: www.mirrorservice.org
 * extras: repo.uk.bigstepcloud.com
 * updates: repo.uk.bigstepcloud.com
Available Packages
rockstor.x86_64                                           3.8-12                                             rockstor        
rockstor.x86_64                                           3.8.16-1                                           Rockstor-Testing
rockstor.x86_64                                           3.8.16-2                                           Rockstor-Testing
rockstor.x86_64                                           3.8.16-4                                           Rockstor-Testing
rockstor.x86_64                                           3.8.16-5                                           Rockstor-Testing
rockstor.x86_64                                           3.8.16-6                                           Rockstor-Testing

The changes in db management between 3.8.14 to 3.8.16 were fairly major, hence the inadvertent breakage in testing channel updates. You may well still be required to reinstall using the latest iso given the missing (possibly impractical to add) specific interim package versions for testing. Heeding the tips / warning we covered earlier in this post.

I know this is frustrating but I hope you can persevere as your input here has already been very valuable:

I particularly like you reverse proxy / lets encrypt post:

But the testing channel updates are rather bleeding edge and need to be if we are to realise more of what we are aiming at.

Hope that helps.

Flyer · January 14, 2017, 12:53am

Hi @exelan,
after @phillxnet excellent infos about Rockstor upgrade/downgrade/upgrade here we come with the daily hack, hoping this will help you (and other “bad” guys not having regular updates! ) till we get 3.8.14 and .15 on testing channel by @suman

Preparatory operations

Roll up your sleeves and have a cofee/tee mug
If having a running 3.8.16 Rockstor to be downgraded disable running tasks (replication, snapshots, scrubs, balances, etc) to avoid a bunch of mails with errors
Yoda says NEVER join the dark side of the Force (rpm install instead of yum localinstall → mess up yum)

Rockstor-lifebelt is an ad hoc Github repo I’ve just created to host Rockstor officials 3.8.14 and 3.8.15 rpms, please remember this is an hack and those rpms will be deleted asa @suman add them to Rockstor Testing!

Moving Rockstor to latest the hard way

ssh to your Rockstor environment (avoid WebUI during all process, not required)
download Rockstor rpms from Rockstor-lifebelt:
curl -O -J https://raw.githubusercontent.com/MFlyer/rockstor-lifebelt/master/rockstor-3.8-14.x86_64.rpm
curl -O -J https://raw.githubusercontent.com/MFlyer/rockstor-lifebelt/master/rockstor-3.8.15-0.x86_64.rpm
Downgrade Rockstor to 3.8.14 with yum downgrade ./rockstor-3.8-14.x86_64.rpm
Gently reboot with shutdown -r now
Back to Rockstor version 3.8.14 don’t perform yum update or similar (this will move 3.8.14->3.8.16 and you’ll have to downgrade again)
Upgrade to Rockstor 3.8.15 with yum localinstall ./rockstor-3.8.15-0.x86_64.rpm (remember Yoda vs Dark Side: no rpm installation)
Once again reboot with shutdown -r now
Open your Rockstor WebUI and enjoy 3.8.15 with alerts about available updates: update and reboot
Back to your WebUI, welcome to Rockstor 3.8.16.* with all required db tables/dependencies

Finally: never take sweets from a stranger ( Those 2 rpms on Github are official Rockstors 3.8.14 and 3.8.15 downloaded from Rockstor Stable Branch so take them from this Github repository )

Mirko

exelan · January 14, 2017, 3:06am

@Flyer You are a Wizard, a Jedi Master… Tron level of understanding the relationship between you and your user. I can’t thank you enough, and wish I could hold a digital medal ceremony involving you walking through the hall of appreciative, silently observing processes that run on my server as John Williams trumpets play resounding triumphant praise.

In case you (or anyone) forgot:

Wookies don’t care about medals

Again, thank you for your help. My system is now happily upgraded and working.

exelan

Flyer · January 14, 2017, 12:17pm

Well, tons of LOLs with this:

Rockstor maintainer @suman is a 2 meters height wookie, 2 contributors are a jedi whose father was first a jedi too then a sith cutting his hand (@phillxnet) and the other one (me) an intergalactic smuggler whose son will become a sith killing his father: i like this group eheh

@exelan : happy to see you solved it

Mirko

suman · January 18, 2017, 8:40pm

Oh man, I came late to the party here. @Flyer, thanks a lot for saving the day here! I have finally managed to update the testing repo with 3.8.15-1.

@exelan you made my day!

exelan · January 19, 2017, 7:49pm

@suman Haha, glad to hear! I can’t thank the community enough for coming together on this one and for many other issues.