As an update, I ended up working around this by nuking my PostgreSQL database:
systemctl stop postgresql
rm -rf /var/lib/pgsql
Then re-initialize the database by hand-executing the commands under sections [postgres-conf] and [postgres-setup] in buildout.cfg.
After these steps the rockstor service starts fine and prompts me to create a new user. Using the existing user turned out to be the trickiest part that required me to hack the rockstor source. It might have been easier if I just removed my existing user temporarily.
Hope this helps anyone else having similar problems.
I think the necessary steps should be all there for everyone who is comfortable enough with command lines to do that. I didn’t list the commands in buildout.cfg because it might be modified anytime in the future; so better just look at the most recent version. For anyone else with a mostly stock system, a clean re-install would be quicker.
And the showstopper of my system wasn’t even related to the database! Turned out the current version of initrock doesn’t like my network configuration and complains that the default interface isn’t configured. I still had to modify the source to get around that.
@Tomasz_Kusmierz Oops, didn’t think it was relevant to anyone else. The problem I had was that initrock.py looks for ‘default via’ from the output of ip route, which does not match any of my interfaces. (It’s actually ‘default dev’ if that’s what I think it’s looking for.) So I just hacked it to use one of my interfaces (which isn’t the default dev anyway.) Hope this helps!
< strikethrough > As an aside, nuking the database still turned out to be a PITA because I’m left setting up all the services again I still have the old database so hopefully someone chips in so I can upgrade temporarily to 3.8.15!< /strikethrough > So it turns out my old database somehow still works (of course I didn’t check first), and that that network interface bug was the only problem. Duh!!
I think that you actually found the problem ( seem from where I sit ). We’ve had a problem where
if you would set up a teaming interface the system would fail to start (even with dhcp) unless you would add a normal interface with dhcp.
system would not start if no cable was connected to any interface and there was no lease from dhcp -> system would timeout after ~10 minutes and only bailout was to log in through physical console and reboot it or use “the button”
I did not engage with solving this any further because I’ve had other stuff to do (don’t ask :/)
Might be completelly different script, but your scenario would explain why my systems didn’t get up when I was setting stuff in manual IP without gateway where there was not other interface with gateway in … and I didn’t occur to me that route might be used as check …
Really useful thread, sorry for chipping in very late. Thanks for your input @haoto. I think you are right about the initrock bug, and it does run on every boot – updating default interface part of it anyway. We’ll get to it. Please feel free to open a github issue.
Regarding 3.8.15 not being available on testing channel, I just added 3.8.15.-1. I know this is too little too late for you, but others may find it useful. Typically, we don’t remove the previous major version’s testing updates from the repo, but looks like I was a bit overzealous cleaning up.