Manual db migration required after 5.1.0 -> 5.5.4 on Slowroll

Troubleshooting
1

After a system upgrade that included moving from rockstor-5.1.0 to 5.5.4 on x86_64 Slowroll, the postgresql service failed to start because the postgresql13 binaries were missing. Looking into this, it appears that postgresql13-server has now been removed from the TW/SR repos, and furthermore there is a file conflict with the latest version of the postgresql noarch package that forces postgresql13-server to be uninstalled during the upgrade. The procedure I ended up following that allowed me to get everything running again was:

  • use snapper rollback to undo the upgrade
  • make backup copies of /usr/lib/postgresql13 and /usr/share/postgresql13
  • upgrade rockstor and accept the solution to remove postgresql13-server
  • restore the backups of /usr/lib/postgresql13 and /usr/share/postgresql13

At this point, postgresql would start, but rockstor-pre still failed on the initrock step, because it appears the version of django in the venv now requires postgresql>=14 and my database was still 13. So, I then did the following:

  • stop the postgresql service
  • rename ~postgres/data to ~postgres/data13
  • start/stop the postgresql service to let it recreate the ~/postgres/data directory structure
  • manually migrate the db with sudo -u postgres pg_upgrade -b /usr/lib/postgresql13 -B /usr/lib/postgresql17 -d /var/lib/pgsql/data13 -D /var/lib/pgsql/data -r
  • manually optimize with sudo -u postgres /usr/lib/postgresql17/bin/vacuumdb --all --analyze-in-stages (this was recommended by pg_upgrade, but probably wasn’t strictly necessary)
  • start the postgressql service

At this point, my database was version 17, and rockstor-bootstrap and rockstar-pre both ran successfully, so I re-ran the system upgrade and everything seems happy.

I hope this is helpful. I’m not really sure what steps rockstor could take to avoid this situation where a required package is forcibly uninstalled by the OS, but it’s possible it might be necessary to start keeping backups of the currently-used postgresql binaries in case they are needed for a migration.

Thanks for reading.

3 Likes
2

@malphite42 Thanks for the feedback on this update. We have yet to release the next stable so have also yet to establish a working procedure in docs akin to our prior how-tos here:

Re. Distribution update from * to *.

And those guides have, thus-far, included only Leap to subsequent Leap version.

This feedback should help with developing said guide, however this failure may be a Slowroll only issue. Where we currently have Slowroll and Tumbleweed, form the downloads page, as:

Development/Advanced-user/Rescue use only

When using the Web-UI update, we started, in V5.5.0-0 of the current and still ongoing testing phase, to use zypper’s “–force-resolution” see:

Legacy postgresql 13 devel packages blocking update

Issue: Legacy postgresql 13 devel packages blocking update · Issue #3020 · rockstor/rockstor-core · GitHub
Pull Request: Legacy postgresql 13 devel packages blocking update #3020 by phillxnet · Pull Request #3025 · rockstor/rockstor-core · GitHub

This fixed part of what you may have experienced, especially if you did not use the Web-UI to do the update. As on Tumbleweed and Slowroll we use “zypper dup” as is recommended.

Yes, as per our rockstor-rpmbuild repo this is our newer postgres dependency:

Definitely. And as more folks use our Tumblewed or Slowroll repos we can use such feedback and our own expedrimentation once we have the next stable RPM released, to hopefully ensure a smoother transition. Which is something we aim for form Leap to Leap when subscribed to the Stable channel.

Indeed, the assumption has been, possibly incorrectly in this case, that older Postgres binaries would still be accessble. They have been in tests of Leap to Leap - but Slowroll and Tumbleweed are their own flavour with their own challenges.

For context we use the following script of our own authorship, to do DB migrations, again based on upstream (Postgres) advice:

In any future write-up for Slowroll-to-Slowroll hickups in update, as you have kindly shared, we would likely want to resource that script if possible, as it takes some precautions to ‘stash’ the original DB: just in case. As you can see, it takes a similar approach as you took regarding the update.

It’s a shame you ran into this and we will want to approach this in time - but likely now only after we have released our last testing package (next Stable rpm) so that we know where folks are coming from and going to. We generally can only focus on Stable to Stable updates (the indicated howtos) and do a best effort as we go on testing to testing. Mainly as we run into these types of issues as we go along and try to address them in code as best we can. That is where the need for that script arose as we had to update Django and so needed to handle a DB format migration also.

The main Rockstor developers were concerned about such failures re TW & SR - and this is why, in-part, we sitlll have the advanced use only badge on these OS versions. But they are super hand to see what is coming in the future, and of great interest to the more technical users such as yourself. But as it has turned out, Leap 15.6 to Leap 16.0 as been a similar challenge re updating - which it sharing some of our TW failings/incompatibilities.

Thanks for the contribution here. And I’m please you got this sorted in the end and thankful for your reporting back on your finding and associated work-around.

Thanks again for this feedback - much appreciated.

3 Likes