Rockstor 4 Installer Recipe - call for beta testers

phillxnet · September 30, 2020, 3:53pm

Are yes, sorry; forgot about that ‘safety’ feature. You can remove it as an option via the grub kernel command line edit intervention. We should add a doc entry for this come to think of it.

We purposefully only show < 500 GB to help folks avoid accidentally installing over their data drives. Trying to play it safe give I’m keen on having this very accessible and mostly folks use a small drive for system and lager than 500 GB for data drives so we have our safety net.

Here is where we build in this option:

github.com

rockstor/rockstor-installer/blob/master/rockstor.kiwi#L72


<!--Re AArch64: https://github.com/OSInside/kiwi/issues/1491-->
<type
        image="oem"
        primary="true"
        initrd_system="dracut"
        filesystem="btrfs"
        fsmountoptions="noatime"
        bootloader="grub2"
        firmware="efi"
        installiso="true"
        kernelcmdline="ipv6.disable=1 plymouth.enable=0 rd.kiwi.oem.maxdisk=500G"
        bootpartition="false"
        devicepersistency="by-label"
        btrfs_root_is_snapshot="true"
        btrfs_quota_groups="false"
        efipartsize="33"
>
    <systemdisk>
        <volume name="home"/>
        <volume name="root"/>
        <volume name="tmp"/>

So you can just grub edit the command kernel command line and remove the:

rd.kiwi.oem.maxdisk=500G

entry.

Let us know how this goes.

Cheers.

phillxnet · September 30, 2020, 3:56pm

@magicalyak

Trying to encourage seperate system and data drives as much as we can, hence say a small ssd for system and big hdds for data.

But we still have feature parity with being able to create subvols on the system drive in our “Built on openSUSE” variant.

magicalyak · September 30, 2020, 3:56pm

I modified the command line on the boot menu (pressing e) and that worked. Not sure it makes sense to limit (better to suggest I think?).

phillxnet · September 30, 2020, 3:58pm

@magicalyak Re:

Agreed, but there is not a suggestion option unfortunately. We use a completely vanilla kiwi-ng so have to stick to their capabilities re options available. They may be up for pull requests on this but I’m unlikely to find the time for this myself for quite a while by the looks of things in the Rockstor camp.

Thanks.

magicalyak · September 30, 2020, 3:58pm

I’ll take a look, I think I can make the PR for it.

phillxnet · September 30, 2020, 4:01pm

I’ve had a few interactions with the main dev during our development of this installer config and they were both super helpful. Hence my credit to them in the docs. This is the upstream project:

Essentially a re-working in bash/python of the old kiwi (perl I think). And it now pretty much as feature parity bar a few things they just dropped due to lack of use, ie re-partitioning etc.

Hope that helps.

Hooverdan · October 22, 2020, 9:59pm

Hi,
I finally got around to try out the installer for 4.0.4. Everything went well installing it on a VM. Post-install I noticed an error/warning message popping up related to rpm…

which apparently is harmless, and related to rpm moving from a Berkeley DB to SQLite DB implementation (rpm.org - Releases from 9/30/2020).
Somewhere I found that to correct this is by rebuilding the rpm database so it aligns with the configuration, running the following command

rpmdb –rebuilddb

Worked for me, and the warning disappeared…

Not sure, whether it has any bearing on the rockstor installer setup/configuration, but I figured I mention it here, in case anybody else runs into this. In any case, it’s only a warning, so also not very high priority.

Cheers,

GeoffA · October 22, 2020, 10:16pm

@Hooverdan Thats an interesting one, and glad it’s not anything fatal. I have to admit that I have not seen that error after having done several (too many probably) installs over the past week or so while testing the new Rockstor 4 installer.
Good find regarding the potential cause as well (is there a ‘thumbs up’ emoji type thing here?).

Hooverdan · October 26, 2020, 10:55pm

One other thing, again, not fatal, but curious. The initial boot completes, the IP addresses are put out, and the login prompt shows up (like in the “olden” days). However, 10 seconds later, additional processes are spawned and the login prompt/IP info disappears off the screen:

And maybe that’s just a VirtualBox “idiosyncrasy”, once the last processes seem to have launched (in my case qgroup scan completed) the only way to get back to the login prompt is to hit Enter.

and then of course, one has to log in and run the myip command to get the IP info…

Doesn’t prevent me from logging into Rockstor via the WebUI and performa

phillxnet · October 27, 2020, 8:07am

@Hooverdan Thanks for the report:

Yes, this is normal behaviour but does vary somewhat with machine speeds. It’s systemd doing a parallel start up and it get the login going sooner than the rest. A bit inconvenient, and it’s why on initial boot I added the prompt about pressing the enter key to get a new login prompt. And on login the messages is displayed again so a little bit of a juggling act really.

I know, again a juggle really. The early login prompt activation is too blame but I’d rather not mess with the default systemd arrangement of starting the shell early. The myip script I added is at least another capability here.

There may be scope for us re-configuring our own systemd service dependencies but they currently stand as our minimum requirement so I’d rather stick with that really.

Thanks again for the feedback.

Hooverdan · October 27, 2020, 10:35pm

Thanks @phillxnet for reviewing the observations … since the intent for the appliance really is to be managed and viewed through the WebUI in normal operations …

HarryHUK · December 10, 2020, 6:53pm

Hi,

I’ve just built the installer using the top-level instructions, (having forgotten about vagrant, I created my own Leap15.2 virtualbox VM by hand)
Edit Got the iso out by shared folder, installing to a VM to try it out.

Then I tried to build with vagrant, but I panicked and thought I’d missed a prerequisite and killed the vagrant build. Now it times out trying to connect to the vagrant VM, and I don’t quite know how to clean it up.

Not sure I’ll have time to install it before Christmas, anyway, though

HarryHUK · December 10, 2020, 9:26pm

For the disappearing prompt, could you do something like mail does, a post a message after every command the user types?

phillxnet · December 15, 2020, 12:29pm

@HarryHUK Hello again.
Re:

This seems a bit spamy to me. I’d personally like to get our network discovery ‘story’ sorted once and for all so folks can just find their installs auto style. But all in good time and if you fancy popping in a pull request with your idea as a proof of concept it may well end up receiving a positive review and getting merged. But we should also be careful not to over complicate things. Especially at the command line, but as you state there is an existing mechanism in the mail message example.

Good luck with your install by the way and well done getting the 4 DIY installer build working. Not sure myself of what might be ‘stuck’ in the vagrant approach but there is some suggested commands within the lower level readme that might help.

Cheers.

HarryHUK · December 15, 2020, 7:57pm

Fair point. Maybe just after it posts that everything is started, use the “write” command to send a message to root.

HarryHUK · December 16, 2020, 12:19am

There was another thing I wanted to ask. In testing the install of 4.0.4 over 3.9.2, I see that importing the pool works well, picking up the shares, but the Rock-Ons seemed to need re-installing, is there a way of exporting/importing/recovering the config so they pick up where they left off?

Flox · December 16, 2020, 12:57am

I’ll be able to answer that one: Yes, what you are looking for is the “Config Backup & Restore” feature:
http://rockstor.com/docs/config_backup/config_backup.html#configuration-backup-and-restore

As of 3.9.2-52, rock-ons are included in the config backup and will be re-installed with the same settings upon Restore. All details should be explained in the documentation above, but feel free to have a look at the corresponding PR for a demonstration, for instance:
https://github.com/rockstor/rockstor-core/pull/2100

It is important to note, however, that because all rock-ons depend on the same shares as those present in the backup, one needs to import pools and shares (as you did) before restoring the config backup. Note also that restoring rock-ons can take some time depending on how many are to be restored, as the procedure equals to re-installing all the rock-ons that were installed at the time of the config backup. As a result, make sure to give it a few minutes to propagate.

Hope this helps, and let us know if you have any other question.

HarryHUK · December 16, 2020, 5:51pm

That’s brilliant, thanks. It helps a great deal.

Marenz · January 8, 2021, 6:14pm

I just built the installer image put it on an USB stick and installed rockstor from it. Had no major issues.

As I don’t have a openSuse system, I used docker. That was not completely straight forward, but it worked eventually. Here are the commands I used for anyone interested:

# Important part here is the --privileged flag which is required to make mount --bind /proc ... work
docker run --privileged -it --mount src=/home/marenz/projects/rockstor-installer,target=/rockstor,type=bind opensuse/leap:15.2

zypper addrepo http://download.opensuse.org/repositories/Virtualization:/Appliances:/Builder/openSUSE_Leap_15.2/ appliance-builder

zypper install python3-kiwi btrfsprogs gfxboot

cd /rockstor
kiwi-ng --profile=Leap15.2.x86_64 --type oem system build --description ./ --target-dir installers/

Thank you for the hard work. Will experiment a bit with it.

A few things I noticed in the WebInterface:

I had the already known error message about some javascript db problem which had no real effects and I think it was gone after a hard-refresh
Upon enabling services (SMB, NFS), the “enable” switch was not centered after I clicked on it but was then again centered later.
Enabling Rock-ons first didn’t work. With that I mean, I clicked the switch, it showed the “processing” animation but then stayed on “off”. After a second try it worked.
When editing a user that has /bin/nologin as shell, in the edit mask it shows /bin/bash . If you don’t change anything and go back it is still bin/nologin in the overview.
Tiny details like that make it feel a bit unpolished. But none of those are actual problems…

I am now gonna try to setup jellyfin as a rock-on. Will report my progress (if interest?)

Another detail: I still had the usb stick attached for installing and in the rockstor log I constantly see this error:

[08/Jan/2021 19:49:22] ERROR [storageadmin.views.disk:461] Error running a command. cmd = /usr/sbin/smartctl --info /dev/disk/by-id/usb-Kingston_DataTraveler_2.0_6CF049E31FC2BD4039440109-0:0. rc = 1. stdout = ['smartctl 7.0 2019-05-21 r4917 [x86_64-linux-5.3.18-lp152.57-default] (SUSE RPM)', 'Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/disk/by-id/usb-Kingston_DataTraveler_2.0_6CF049E31FC2BD4039440109-0:0: Unknown USB bridge [0x0951:0x1665 (0x100)]', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = ['']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/disk.py", line 458, in _update_disk_state
    do.name, do.smart_options
  File "/opt/rockstor/src/rockstor/system/smart.py", line 338, in available
    [SMART, "--info"] + get_dev_options(device, custom_options)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 198, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/sbin/smartctl --info /dev/disk/by-id/usb-Kingston_DataTraveler_2.0_6CF049E31FC2BD4039440109-0:0. rc = 1. stdout = ['smartctl 7.0 2019-05-21 r4917 [x86_64-linux-5.3.18-lp152.57-default] (SUSE RPM)', 'Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/disk/by-id/usb-Kingston_DataTraveler_2.0_6CF049E31FC2BD4039440109-0:0: Unknown USB bridge [0x0951:0x1665 (0x100)]', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = ['']

phillxnet · January 8, 2021, 7:47pm

@Marenz Hello again.
Re:

That’s a good start at least :).

The alternative vagrant/virtualbox based method contributed by @mikemonkers may have been easier than using docker I suspect:

Thanks for sharing your docker approach here, interesting. The kiwi-ng system actually has a build in virtualisation function now so it may be useful overall to try and move to that in our instructions. But it’s still a little new and I like that we use the core kiwi-ng functionality only currently.

Yes, there is an annoying lag in the switches some times. They appear to get stuck for a bit. This I think is long standing and likely a low level javascript issue that we may be able to fix by updating our dependencies. But we may end up getting more that is broken so holding off on that until we have our os transition in place.

OK, again this may be down to delay, we need to revamp hole elements of our UI really but all in good time. We should keep an eye on this one however.

Could you make a GitHub issue for that in our rockstor-core repo. That way we have attribution which is always good. I’ve not noticed this one myself.

Agreed. We do want to get to all these paper cuts in the end and we hope to do some of the already identified ones before the next stable release. But as always it’s down to resources and we are rather short on those. Developer time/attention is a major shortage actually. So if you fancy taking a look at any of these then that would be great. But getting all we can in clearly defined GitHub issues with clear reliable reproducers is the start of this.

Yes, all reports are welcome. Though best to report in a dedicateed thread with an indication of the exact version of Rockstor you are running. I.e. 4.0.4 or the just released 4.0.5 which has quite a few changes in comparison to prior releases.

Yes, this is a limitation of smrtmontools and it’s ability to ‘read’ the device. Many USB sticks have incomplete smart support. The message is harmless but we do have a doc guide on -d options you can try:

http://rockstor.com/docs/smart/smart.html#disk-custom-s-m-a-r-t-options

It has and example of this exact error message as it’s very common with USB sticks that have partial support or need a -d option to be properly recognised. Many will simply not work with smart no matter what options you put in.

Thanks for the detailed feedback. And agreed we have lots of paper cuts, but far fewer than we use to have and we have transitioned to a more appropriate OS in the interim so once that process is done we can settle down to dealing with all the reports as and when developer time/attention is available.

Hope that helps.