I am in a situation now where my root mounts and 3 pools mount but two of the pools (my 2 that run all my rockons and collections) don’t mount subvolumes. I’ve run scrub and am now running a btrfs check (I’m too afraid to run --repair). Currently the check hits the memory error so I’m using btrfs check --mode lowmem /dev/sdc (one of the RAID10 disks) and it’s running…very slowly
Reboots haven’t worked. I think it may be a qgroup issue and I tried /opt/rockstor/bin/qgroup-clean but that didn’t change much.
rockstor-bootstrap fails and it’s just strange. the root volume mounts and a single disk pool called backup mounts but my two big pools (tv, movies) only mount the main pool. What is very interesting is I can run a docker container for plex (because that is like a nightmare for my little kids when it’s down) by creating a new name container and using /mnt2/movies/radarr-movies:/radarr-movies instead of the /mnt2/radarr-movies:/radarr-movies for example.
I really really don’t want to reinstall rockstor considering I don’t see what’s wrong with it. I was also on experimental just fine and possibly the -13 might have affected it? I downgraded to 12 and 11 and then back to stable but all resulted in the same issue.
from journalctl when manually running rockstor-bootstrap
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: BTRFS device scan complete
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com bootstrap[4125]: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: ['Internal Server Error: No JSON object could be decoded']
Sep 21 08:44:23 rocky.gamull.com systemd[1]: rockstor-bootstrap.service: main process exited, code=exited, status=1/FAILURE
Sep 21 08:44:23 rocky.gamull.com systemd[1]: Failed to start Rockstor bootstrapping tasks.
Sep 21 08:44:23 rocky.gamull.com systemd[1]: Unit rockstor-bootstrap.service entered failed state.
Sep 21 08:44:23 rocky.gamull.com systemd[1]: rockstor-bootstrap.service failed.
from journalctl starting bootstrap also:
Sep 21 08:42:28 rocky.gamull.com systemd[1]: Starting Rockstor bootstrapping tasks...
Sep 21 08:42:28 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:28,870 CRIT Supervisor running as root (no user in config file)
Sep 21 08:42:28 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:28,885 INFO RPC interface 'supervisor' initialized
Sep 21 08:42:28 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:28,886 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Sep 21 08:42:28 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:28,886 INFO supervisord started with pid 4124
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label tv devid 1 transid 26786 /dev/sdc
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label tv devid 2 transid 26786 /dev/sdi
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label tv devid 3 transid 26786 /dev/sdh
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label tv devid 4 transid 26786 /dev/sdb
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label movies devid 6 transid 112693 /dev/sdj
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label movies devid 1 transid 112693 /dev/sdf
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label movies devid 2 transid 112693 /dev/sdd
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label movies devid 3 transid 112693 /dev/sdg
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label movies devid 5 transid 112693 /dev/sde
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label movies devid 4 transid 112693 /dev/sdk
Sep 21 08:42:29 rocky.gamull.com kernel: BTRFS: device label backup devid 1 transid 3257 /dev/sdm
Sep 21 08:42:29 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:29,889 INFO spawned: 'nginx' with pid 4136
Sep 21 08:42:29 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:29,891 INFO spawned: 'gunicorn' with pid 4137
Sep 21 08:42:29 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:29,893 INFO spawned: 'data-collector' with pid 4138
Sep 21 08:42:29 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:29,895 INFO spawned: 'ztask-daemon' with pid 4139
Sep 21 08:42:31 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:31,910 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Sep 21 08:42:31 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:31,910 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Sep 21 08:42:32 rocky.gamull.com kernel: device-mapper: uevent: version 1.0.3
Sep 21 08:42:32 rocky.gamull.com kernel: device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
Sep 21 08:42:33 rocky.gamull.com kernel: BTRFS info (device sdm): use lzo compression
Sep 21 08:42:33 rocky.gamull.com kernel: BTRFS info (device sdm): disk space caching is enabled
Sep 21 08:42:33 rocky.gamull.com kernel: BTRFS info (device sdm): has skinny extents
Sep 21 08:42:34 rocky.gamull.com kernel: BTRFS info (device sdk): use no compression
Sep 21 08:42:34 rocky.gamull.com kernel: BTRFS info (device sdk): disk space caching is enabled
Sep 21 08:42:34 rocky.gamull.com kernel: BTRFS info (device sdk): has skinny extents
Sep 21 08:42:34 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:34,914 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Sep 21 08:42:34 rocky.gamull.com supervisord[4124]: 2017-09-21 08:42:34,914 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Sep 21 08:42:46 rocky.gamull.com kernel: BTRFS info (device sdb): use no compression
Sep 21 08:42:46 rocky.gamull.com kernel: BTRFS info (device sdb): disk space caching is enabled
Sep 21 08:42:46 rocky.gamull.com kernel: BTRFS info (device sdb): has skinny extents
Sep 21 08:42:51 rocky.gamull.com kernel: BTRFS error (device sdb): qgroup generation mismatch, marked as inconsistent
Sep 21 08:42:51 rocky.gamull.com kernel: BTRFS info (device sdb): checking UUID tree
btrfs fi show
[root@rocky log]# btrfs fi show
Label: 'rockstor_rockstor' uuid: 3e3b17e7-2490-484f-9c8a-f402b2a51517
Total devices 1 FS bytes used 3.57GiB
devid 1 size 122.66GiB used 7.06GiB path /dev/md126
Label: 'backup' uuid: 8ac79908-cb09-4568-bfc8-b0fd377dcf15
Total devices 1 FS bytes used 67.91GiB
devid 1 size 3.64TiB used 71.02GiB path /dev/sdm
Label: 'movies' uuid: c77c9722-7a5d-458c-bb1f-c077a950771d
Total devices 6 FS bytes used 3.51TiB
devid 1 size 1.82TiB used 1.35TiB path /dev/sdf
devid 2 size 1.82TiB used 1.35TiB path /dev/sdd
devid 3 size 1.82TiB used 1.35TiB path /dev/sdg
devid 4 size 1.82TiB used 1.35TiB path /dev/sdk
devid 5 size 1.82TiB used 1.35TiB path /dev/sde
devid 6 size 1.82TiB used 1.35TiB path /dev/sdj
Label: 'tv' uuid: 76b16cb2-0f13-401d-8395-c408cfc0fdfe
Total devices 4 FS bytes used 3.72TiB
devid 1 size 2.73TiB used 1.86TiB path /dev/sdc
devid 2 size 2.73TiB used 1.86TiB path /dev/sdi
devid 3 size 2.73TiB used 1.86TiB path /dev/sdh
devid 4 size 2.73TiB used 1.86TiB path /dev/sdb
Only other caveat is the rockstor root is on mdraid following instructions from documentation. Again, this isn’t fun to reinstall.