3.1.3 installer on HP Proliant microserver gen8

Installation
1

On a brand spanking new HP Proliant microserver, I boot off the CD, get as far as choosing the installation target (a USB drive), and it falls over in a big ugly mess with an “unknown error”. Further info has a load of python stacktrace.


If I click quit, the system just reboots.

If I click debug, it dumps me in to a python debugger in tmux.

If I click report error, it gives me a bit more info: “ValueError: cannot remove non-existent subvolume home - anaconda”.

If I click “upload the problem data to a server” it says “report uploader is not properly configured”.

Pretty poor experience so far. Is anybody actually using rockstor? (aside from people that work there, I mean).
2

I tried installing to a HDD instead, figuring mebbe it’s the USB drive that’s the problem.


Nope, falls over in exactly the same way.
3

This is strange as I am personally aware of HP microservers running Rockstor and none of them ran into this problem.

Sorry that it’s not even installing for you, we can help troubleshoot.

I think that the installer/anaconda is running into a buggy code path because of the particular state of the drive you are trying to install it to. My speculation is that it is because it’s not happy with the existing partition table on the drive.

What was on the drive before?

Is it possible to zero out the drive using dd or wipefs and try installing again? You may use Knoppix for that.

One more thing to try is to install stock CentOS 7 and see if that works, both with and without btrfs partitioning scheme.

I agree that it indeed is a poor install experience. We’ll need to reproduce the error in order to fix it. But it’s clear that the problem is in the anaconda installer.

4

in the installer, there’s a bit where you choose the installation target drive. On my setup, first is an HD, then the USB drive, then the other HD. The first HD is reported as something like 986GB capacity, 1.8 MB free, whereas the second HD drive is “968GB capacity, 968GB free” or something in that ballpark. (These aren’t the exact numbers just a ballpark, my server is at home now).


The first HD is selected by default. If I choose either of the others, I run into the aforementioned problems. 

However, if I leave it on default and don’t change anything, it installs fine (albeit to one of the hard discs). I’m going to attempt to migrate the ext4 boot and swap partitions on to the USB stick, followed by the BTRFS volume…altho I am new to BTRFS so not entirely confident how to do this.

I’ll try your wipefs approach, it is possible that HP’s Intelligent Provisioning has done something funky to the drives.

I’ll happily send over some screenshots if that helps.
5

Happy to help. HP microservers are one of the first hardware configurations we started using Rockstor for real. I take it yours is the latest(gen8) unlike our older ones.

Do you have experience running Linux with root filesystem on a USB drive? I gather your goal is to run Rockstor from usb so you can free up one more HDD for data? I have an old HP desktop here on which I run Rockstor from a usb stick, but I am not very happy with the results. First, the system runs significantly slower and unpredictably, in a matter of few months the usb died, however, reinstall worked fine. But that has another problem in that after you reinstall, you can’t just import your previous state (pools, shares, users…). This last problem is something we’ll fix(https://github.com/rockstor/rockstor-core/issues/392) soon.

I built another DIY NAS box(http://rockstor.com/blog/?p=425#comment-5) and the installer did show the usb stick as the second(non-default) drive. It installed fine on it.

Yes, please send screenshots and any other helpful information. You can email files to support@rockstor.com


6

Yes, it’s a gen8. You’re correct: I want to put rockstor on a USB stick so that both hard drives that came with the server can participate in the array.


Well this is weird: after the installer crashes yesterday, I let the rockstor installer run with its default settings: ie it installed itself to sda. Tonight when I came to repeat the sequence to generate some screenshots for you, it didn’t bomb out in the usual place. This time it asked me if I wanted to reclaim space from the USB stick, then installed on to it fine. I used the rockstor disk tools icon to blow away the old rockstor install on sda, and suddenly we’re in business!

I suspect that Intelligent Provisioning left one or both of the HDs in a weird state, which the first rockstor install corrected, allowing the second one to succeed.

7

I am glad you were able to troubleshoot it. I’ll try to reproduce it by trying to install on a hard drive with ext or xfs filesystem on it.

But more importantly, keep us posted on how the behaviour is running from the USB stick.

Also, do you mind sharing exact specs of your hardware including HDDs and USB? It could be useful for others in the community.

Last but not least, I am interested in helping you get Rockstor running from the micro sd card. You’ll need to spend some time mucking with your precious box and I understand if you don’t want to deal with it since things are working now. But it can really help others to figure this out. Either way, I appreciate your involvment so far.

8

You’re not wrong about the USB by the way. Dreadfully slow. I’ve got a 16GB sansdisk extreme on order (has a proper SSD controller) - hopefully that improves things. Otherwise I’m going to replace the DVD drive with a proper SSD.

9

Fair enough. I’ve updated the title of this discussion to it’s more useful to others trying Rockstor on HP microservers.

10

I just wanted to add that I have also seen this issue, and I was not installing on an HP microserver, so it’s not limited just to that scenario. I tried the steps that worked for blowback (install to HDD, then reinstall on the USB drive), and that worked, but before that I was getting the same error about being unable to delete a non-existent subvolume.

For reference, the hardware I was using was a machine I built myself: ASRock C2750D4I motherboard/CPU with an 8 GB Verbati Store N Stay drive as the target. Not sure which of those was at fault, but clearly something made the installer unhappy.

11

6 months later, and I’m back for another go. This time with Rockstor-3.7.9.iso, to see if the installation process is any less dreadful.


TL;DR: No, it isn’t.

I copied the install image to a brand new 64BG Kingston USB key, installed a brand new SansDisk Extreme 16GB USB key in the internal slot, and fired up the install.

First attempt was unsuccessful. GUI installer fell over with "unable to delete non-existent subvolume home00"

Second attempt was unsuccessful, I told it to reclaim the space on the SansDisk drive, and it died trying to format.

Third time around, I wiped the SansDisk on a different computer first, and again “unable to delete non-existent subvolume”.

This time I learned a new trick, instead of clicking “Quit” which just reboots the box and starts the whole interminable 5 minutes POST+date config + lang config + keyboard config + disk config, I just ctrl-alt-f1’d over to the console, typed ctrl-b-1 to get to the shell, and ran anaconda from there. It gives you a text interface, but at least I didn’t have to wait for the reboot.

This time the install completed successfully. I rebooted. More waiting for POST/boot, and then…“grub: error attempting to read or write outside of hd0”.

I rebooted and reinstalled once again, this time with the GUI, and this time around the “reclaim space” option showed what looked like a correctly configured boot+swap+xfs setup, and it was actually able to delete them without falling over, which was a nice surprise.

I let the install complete, rebooted, waited some more…same grub error.

from grub’s rescue prompt, I can see the following partitions: (hd0), (hd0,msdos3), (hd0,msdos2), (hd0,msdos1), (hd1), (hd1,msdos1), (hd2), (hd3)

Of these, only one has a filesystem that grub can read: (hd0,1) is ext2.

I tried:

set root=(hd0,1)
set prefix=(hd0,1)/grub
insmod normal

and got "attempt to read or write outside of disk hd0"

I tried:

set root=(hd0,1)
set prefix=(hd0,1)/grub2
insmod normal

and got "attempt to read or write outside of disk hd0"

ls (hd0,1) gives me lost+found, grub, grub2, initrd-pymouth.img, a bunch of System*, image, config and initramfs* files.

I can’t ls  the grub or grub2 dirs, without getting the “attempt to read or write outside if disk hd0” error, so that’s why the insmod fails also.

At this point I gave up.

I think you;ve got a couple of problems:

1) your installer is incredibly flakey. If you try and do anything even slightly off the “happy path”, it chucks an error and gives up. Also, even when it does complete a “successful installation” for anything other than the setup it is expecting, all it ends up doing is trashing the disk.

2) given that your installer is so incredibly flakey, you really out to have an option other than “Quit”. Because that just reboots the system. This is annoying, because each time around you have to configure all the settings again. On an HP Proliant it is infuriating because the total round-trip time for a reboot is about 5 minutes.
12

blowback, I feel your pain and really appreciate the detailed comment. Besides just thanking you, I think I have something helpful. I’ve created a new iso with latest anaconda bits from centos and it’s a lot less flaky. I tried a few paths especially with custom partitioning and no crash! I did manage to crash it trying to toggle on/off switch of a nic that’s not connected. 


Anyway, I’d really like you to try the new iso/installer. It’s not on sourceforge yet as I want to do some more testing. You can get it from here: http://rockstor.com/Rockstor-3.8-0.iso

Could you please give it a try and perhaps report back some good news :slight_smile: ?
13

After a cup of tea, I decided I wasn’t quite done. I downloaded Super Grub2 Disk from here http://www.supergrubdisk.org/category/download/supergrub2diskdownload/super-grub2-disk-stable/, burned it to a CD, and booted off that.


It looks like hd0 is one of my BTRFS volumes, and hd2 is the other (both with label “filestore”). hd1 is my boot/system disk (with label “VID”). Attempting to boot off (hd1) gives me an “Illegal opcode” panic, and attempting to boot from “(hd1,msdos1)” gives me "This is not a bootable disk."

I downloaded knoppix 7.4 and booted to that. Poking around reveals some conflicting information, now I’ve got the two big BTRFS platters on /dev/sda and /dev/sdb, some weird internal HP iLO thing on /dev/sdc, and the RockStor image on /dev/sdd. 

/dev/sdd has three partitions, boot, swap and btrfs with label “rockstor_rockstor”. I mounted boot, and everything looks fine, altho the grub2 device.map maps (hd0) to /dev/sde, which seems wrong. Tried changing it to /dev/sdd, but system still will not boot without grub error. I suspect that installing with two USB drives inserted and then attempting to boot after removing one of them has thrown a spanner in the works.

To rule this out I burned Rockstor 3.7.9 to a CD and booted from that, reinstalling over the SanDisk USB drive.

grub error: “attempt to read or write outside of disk hd0”.

grub’s ls now shows me: (hd0), (hd0,msdos3), (hd0,msdos2), (hd0, msdos1) (hd1) and (hd2). Boot partition is (hd0,1) as you’d expect.

Then I found this: http://h30499.www3.hp.com/t5/HP-BladeSystem-Server-Blades/Using-16GB-Sandisk-USB-drives-for-VMware/td-p/6450768#.VWn6Ed-c1rY

This seems to suggest that gen8’s won’t boot from certain SanDisk USB keys! Arghhhh!  :-0

I’ve checked for new BIOS updates, and I’m up to date (J06), so it looks like that’s the end of that.

14

blowback! did you see my previous comment? Have you tried the new 3.8-0 iso? I am curious if the problem persists with it.

15

Suman, sorry didn’t see the post, was too busy investigating! Earlier posts are a bit long-winded and curmudgeonly I’m afraid, a bit of a stream-of-consciousness rant as I tried to get that particular set-up to work! I tried to edit them for tone and length but the edit page doesn’t seem to work!


I’m pretty sure now that the issue is basically that the HP ProLiant Gen8 BIOS reports incorrect drive geometry for the SanDisk Extreme USB card, and simply will not boot from it. A running system can read the USB stick fine.

My previous experience with a regular USB thumb drive was the same as yours: it’s slow, and dies completely after a couple of months.

I’ve given up and decided to do it the prescribed way: it’s cheaper and less effort to throw more platters at the problem.

I think the only legitimate take-away from this is the CentOS installer being a bit poor when you go off-script, and the thing about “Quit” (and subsequent lengthy reboot) being the only option when something goes wrong, and it sounds like you guys are doing your best to address them.

If I can find another small-volume USB thumb drive with a proper SDD controller I might be tempted to have another go!
16

That giant non-resizable BTRFS partition doesn’t really sit well with me, so I’m back with a new non-standard setup, for which I’ll try your new installer image.


New setup: a Lycom PE-125 4xmSATA PCIe RAID card, with a pair of Transcend 16GB mSATA drives on it (has room for 4!)

I want to configure the two mSATA drives as a RAID1 array, and boot off that (so that I have some redundancy in /boot). I figure I have a couple of options here:

1) use the card’s hardware RAID to merge the two mSATA drives into 1 logical volume, and install on to that
2) some sort of LVM/mdadm mucking about: don’t know if it will boot off that
3) mirror the two drives with BTRFS, but I’m presuming grub2 won’t boot from btrfs?

any recommendations? (1) seems the easiest route, but it’s a marvel controller, and a few years ago there were all sorts of problems with linux and hardware RAID - I don’t know if these are solved now, or if this setup is still best avoided.


17

so I decided after reading around not to rely on the Marvell hardware RAID, and configure it as JBOD.

Booted RockStor 3.8.0, and selected the two mSATA drives for installation (sde, sdd).

I manually created a /boot partition, and switched it over to RAID1.
I added some swap on sdd
I added a BTRFS volume to take up the remaining space on sde, then configured it as RAID1.

Anaconda did not fail me once during this entire process, set up performed flawlessly. So that’s the good news: the 3.8.0 installer is a massive, massive improvement.

The bad news tho: it won’t boot. I get:

BTRFS: failed to read the system array on sdd2
BTRFS: open_ctree failed
18

Second time around, I configured /boot as a RAID-1 across both sdd/sde, swap on sdd, / as btrfs on sdd only. figuring I’ll be able to add the remaining space on sde to the rockstor_rockstor pool once the system’s booted.


Again, the installer worked flawlessly.
This time the system boots!

In storage, I have: sdd2 = btrfs ‘rockstor_rockstor’
                              sde (disk is unusable because it has another file system on it - click to wipe)

clicking the wipe icon fails with “device busy”: makes sense as sde is part of /boot mirror.

More worryingly, one of the two big magnetic drives (sda) is also mounted (it’s got a previous rockstor installation on) and this is also mounted and can’t be wiped: it seems to have merged with my ‘rockstor_rockstor’ pool…
19

Third time around, I booted 3.8.0, and used one of the spare shells to wipefs sda,sdc (magnetics) and sdd, sde (msatas).


Ananconda didn’t notice the change, so I selected all disks for install and used the custom partitioning to erase them.
Then I went back and removed sda,sdc and did custom partitioning again; ie install just to the two msata drives.

I went with /boot as ext4 RAID-1, swap on both drives, and a BTRFS volume in RAID1 on both (and just to be sure I renamed it “rockstor_boot”).

Again, can’t fault the 3.8 installer: it’s starting to feel like something I can rely on, rather than something I need to be wary off!

Again tho, system won’t boot: same error as before.


20

I think I fell foul of the Rockstor rule “rockstor only works on unpartitioned drives”; so while I can set up / as btrfs in a partition at install time, I can’t add another partition to it at run time.


Fourth time around, I configured /boot as md-raid1 across the two msata drives, and also /. with swap space on one drive. The installer looks like it will let you set up lvm+mdraid or btrfs-raid for /boot, but later tells you you can’t use these for /boot.

This config boots fine. I’m free to use the two magnetic disks however I want in rockstor, and the two msatas give me some boot/root redundancy. 

There’s a slightly worrying message about the md device in rockstor’s ‘disks’ page: "

Warning! Disk serial number or UUID is not legitimate or unique.

Disk names may change unfavorably upon reboot, errors and replacement resulting in data loss. This error is caused by the source of these disks such as your Hypervisor or SAN. Please ensure that disks are provided with unique serial numbers before proceeding futher"

but i’m presuming I can just ignore that :wink:
finally I’m ready to use it as a NAS!