Rock-ons can't be enabled [SOLVED], Lost access to shares & data [SOLVED], running Rockstor 3.8.16-1

glenngould · January 12, 2017, 8:12am

After a restart, I can’t access my files anymore.

Rock-ons is disabled and can’t be enabled anymore
The terminal shows “systemctl: Unit nslcd.service could not be found”
the different widgets on the dashboard show on the one hand only 12,9GB being used on the other hand still the correct 900GB being used out of the 7TB share

Please - can you help me?!
What has happened to my system?

My last backup was done a few months ago
Fu**, I had in mind to run another backup after New Year…

Editing / problem partly solved:
Samba Service was OFF due to a normal reboot? strange
I can access my files, BUT Rock-ons can’t be switched on - same error “nslcd.service could not be found”

Any ideas how to get Rock-ons again staying enabled?

exelan · January 12, 2017, 6:29pm

@glenngould Hey there, glad you were able to figure out your data “problem”. Do you have ssh/command line access to your server?

If not, just click on ‘System Shell’ under the rock-ons tab:

Then type, sudo docker ps and see if anything outputs. If you weren’t aware, Docker containers are your Rock-On plugins. Just re-branded for Rockstor theme consistency.

Here’s a sample output:

[1:0:4][root@rockstor:pts/0][~]
# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                            NAMES
ef6503e10b29        rocketchat/rocket.chat    "node main.js"           5 months ago        Up 18 hours         0.0.0.0:3000->3000/tcp                           rocketchat
4cffd3e72328        mongo                     "/entrypoint.sh mongo"   5 months ago        Up 18 hours         27017/tcp                                        mongodb.rocketchat
8ca614e38742        linuxserver/plex          "/sbin/my_init"          5 months ago        Up 18 hours                                                          plex-linuxserver.io
3065a6cefc34        linuxserver/headphones    "/sbin/my_init"          5 months ago        Up 18 hours         0.0.0.0:8181->8181/tcp                           linuxserver-headphones
a90e1c05443a        linuxserver/couchpotato   "/sbin/my_init"          5 months ago        Up 18 hours         0.0.0.0:5050->5050/tcp                           couchpotato
7c360ccfe37b        linuxserver/sickrage      "/sbin/my_init"          5 months ago        Up 18 hours         0.0.0.0:8081->8081/tcp                           sickrage
24cdfb8793ae        linuxserver/sabnzbd       "/sbin/my_init"          5 months ago        Up 18 hours         0.0.0.0:9090->9090/tcp, 0.0.0.0:8084->8080/tcp   sabnzb
524d1f34889c        owncloud                  "/entrypoint.sh apach"   5 months ago        Up 18 hours         0.0.0.0:8085->80/tcp, 0.0.0.0:8085->80/udp       owncloud-official

glenngould · January 13, 2017, 6:40am

Thanks for your reply.

I’ve tried “sudo docker ps” but no output…it looks like the command is hanging after hitting enter…

I had to cancel it with “Strg+c”

Any ideas?

exelan · January 13, 2017, 7:02am

@glenngould Yep, that means your docker service isn’t running and probably is hanging on some dependent modules.

First off, based on your shell status, you’re running as root.

I’ve tried “sudo docker ps” but no output…

So you don’t need sudo (superuser do). You can just run the commands below without that.

Try this:

systemctl --failed

It should output all the failed units within systemd:

[1:0:2][root@rockstor:pts/0][~]
# systemctl --failed
  UNIT          LOAD   ACTIVE SUB    DESCRIPTION
● kdump.service loaded failed failed Crash recovery kernel arming

kdump.service “failing” is normal and will only activate when trying to capture a kernel dump, which you aren’t experiencing at the moment.

The Docker / Rockstor services rely on the rockstor.service and the rockstor-bootstrap.service, so if these are failing then you have big problems. But since you’re able to get to the dashboard I think you’re okay here.

There are 2 sysd units that should be running related to Docker / Rock-ons:

● docker.socket - Docker Socket for the API
   Loaded: loaded (/etc/systemd/system/docker.socket; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-01-11 17:20:23 MST; 1 day 6h ago
   Listen: /var/run/docker.sock (Stream)

And

● docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-01-11 17:20:38 MST; 1 day 6h ago\

These both should be Active status. To check these individually, run this:

systemctl status docker.socket
systemctl status docker.service

Regards,

exelan

glenngould · January 13, 2017, 7:50am

this is the output I get - socket running but service inactive (dead)

[root@rockstor ~]# systemctl status docker.socket
● docker.socket - Docker Socket for the API
Loaded: loaded (/etc/systemd/system/docker.socket; disabled; vendor preset: disabled)
Active: active (running) since Thu 2017-01-12 12:30:56 CET; 20h ago
Listen: /var/run/docker.sock (Stream)
[root@rockstor ~]# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: inactive (dead)

exelan · January 13, 2017, 7:58am

@glenngould Okay, just to be clear, attempting to activate Rockons from the GUI is failing?

If so, try this:

systemctl stop docker.service

then do a

systemctl start docker.service

If it fails or is still in an inactive (dead) state, then check your sysd logs to see what is happening during that time:

journalctl -xe

Check the timestamps of when you restarted docker.service and what is in your journalctl file.

glenngould · January 13, 2017, 9:12am

thx again -> again inactive (dead)

Here is the log which tells us a problem with the bootstrap service

[root@rockstor ~]# journalctl -xe
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping.
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping.
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping.
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping.
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping.
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping.
Jan 13 10:08:15 rockstor bootstrap[8616]: Max attempts(15) reached. Connection e
Jan 13 10:08:15 rockstor systemd[1]: Received SIGCHLD from PID 8616 (bootstrap).
Jan 13 10:08:15 rockstor systemd[1]: Child 8616 (bootstrap) died (code=exited, s
Jan 13 10:08:15 rockstor systemd[1]: Child 8616 belongs to rockstor-bootstrap.se
Jan 13 10:08:15 rockstor systemd[1]: rockstor-bootstrap.service: main process ex
Jan 13 10:08:15 rockstor systemd[1]: rockstor-bootstrap.service changed start ->
Jan 13 10:08:15 rockstor systemd[1]: Job rockstor-bootstrap.service/start finish
Jan 13 10:08:15 rockstor systemd[1]: Failed to start Rockstor bootstrapping task
Jan 13 10:08:15 rockstor systemd[1]: Sent message type=signal sender=n/a destina
Jan 13 10:08:15 rockstor systemd[1]: Sent message type=signal sender=n/a destina
Jan 13 10:08:15 rockstor systemd[1]: Job docker.service/start finished, result=d
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1
Jan 13 10:08:15 rockstor polkitd[2127]: Unregistered Authentication Agent for un
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=or
lines 979-1001/1001 (END)
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be
Jan 13 10:08:15 rockstor bootstrap[8616]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be
Jan 13 10:08:15 rockstor bootstrap[8616]: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: [‘Internal Server Error: No JSON object could be decoded’]
Jan 13 10:08:15 rockstor systemd[1]: Received SIGCHLD from PID 8616 (bootstrap).
Jan 13 10:08:15 rockstor systemd[1]: Child 8616 (bootstrap) died (code=exited, status=1/FAILURE)
Jan 13 10:08:15 rockstor systemd[1]: Child 8616 belongs to rockstor-bootstrap.service
Jan 13 10:08:15 rockstor systemd[1]: rockstor-bootstrap.service: main process exited, code=exited, status=1/FAILURE
Jan 13 10:08:15 rockstor systemd[1]: rockstor-bootstrap.service changed start -> failed
Jan 13 10:08:15 rockstor systemd[1]: Job rockstor-bootstrap.service/start finished, result=failed
Jan 13 10:08:15 rockstor systemd[1]: Failed to start Rockstor bootstrapping tasks.
Jan 13 10:08:15 rockstor systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=13 reply_cookie=0 error=n/a
Jan 13 10:08:15 rockstor systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=6990 reply_cookie=0 error=n/a
Jan 13 10:08:15 rockstor systemd[1]: Job docker.service/start finished, result=dependency
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1.1 destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=6990 reply_cookie=0 error=n/a
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1.1 destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=6991 reply_cookie=0 error=n/a
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1.1 destination=n/a object=/org/freedesktop/systemd1/unit/rockstor_2dbootstrap_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged co
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=:1.1 destination=n/a object=/org/freedesktop/systemd1/unit/rockstor_2dbootstrap_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged co
Jan 13 10:08:15 rockstor polkitd[2127]: Unregistered Authentication Agent for unix-process:8816:7782387 (system bus name :1.113, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Jan 13 10:08:15 rockstor systemd-logind[1814]: Got message type=signal sender=org.freedesktop.DBus destination=n/a object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameOwnerChanged cookie=239 reply_cookie=0 error=n/a
~

exelan · January 13, 2017, 5:22pm

@glenngould Yeah, there’s something else going on.

Do a:

ls -larth in your /opt/rockstor/ directory.

exelan

glenngould · January 13, 2017, 5:39pm

again the output

[root@rockstor ~]# cd /opt/rockstor/
[root@rockstor rockstor]# ls -larth
total 108K
drwxr-xr-x. 1 root root 6 Apr 8 2016 var
drwxr-xr-x 1 root root 92 Apr 8 2016 certs
-rw-r–r-- 1 root root 0 Apr 8 2016 .initrock
drwxr-xr-x 1 root root 0 Jun 22 2016 rockons-metastore
drwxr-xr-x. 1 root root 16 Nov 5 16:38 …
drwxr-xr-x. 1 root root 16 Dec 17 02:23 src
-rw-r–r-- 1 root root 3.1K Dec 17 02:23 setup.py
-rw-r–r-- 1 root root 3.0K Dec 17 02:23 prod-buildout.cfg
-rw-r–r-- 1 root root 251 Dec 17 02:23 manage.py
-rw-r–r-- 1 root root 639 Dec 17 02:23 install.py
-rw-r–r-- 1 root root 35K Dec 17 02:23 COPYING
-rw-r–r-- 1 root root 7.0K Dec 17 02:23 buildout.cfg
-rw-r–r-- 1 root root 6.1K Dec 17 02:23 bootstrap.py
-rw-r–r-- 1 root root 6.7K Dec 17 02:23 base-buildout.cfg
-rw-r–r-- 1 root root 81 Dec 17 02:23 AUTHORS
drwxr-xr-x. 1 root root 0 Dec 17 02:23 parts
drwxr-xr-x. 1 root root 84 Dec 17 02:27 static
-rw-r–r-- 1 root root 25K Dec 17 02:27 .installed.cfg
drwxr-xr-x. 1 root root 930 Dec 17 05:03 bin
drwxr-xr-x. 1 root root 34 Dec 17 05:03 develop-eggs
drwxr-xr-x. 1 root root 1.2K Dec 17 05:03 conf
drwxr-xr-x. 1 root root 54 Dec 17 05:04 etc
drwxr-xr-x. 1 root root 374 Dec 17 05:04 .
drwxr-xr-x. 1 root root 2.1K Dec 17 05:04 eggs

exelan · January 13, 2017, 10:06pm

@glenngould Yep, there are some serious issues.

This is your problem right here:

Try running this to check the dependencies of your rockstor-bootstrap.service:

systemctl list-dependencies rockstor-bootstrap.service

Let me know which ones show up red.

exelan

glenngould · January 14, 2017, 8:58am

These are all showing up in red

● ├─rhel-autorelabel-mark.service
● ├─rhel-autorelabel.service
● ├─rhel-configure.service

● ├─rhel-loadmodules.service
● ├─rockstor-hdparm.service
● ├─selinux-policy-migrate-local-changes@targeted.service

● ├─rhel-autorelabel-mark.service
● ├─rhel-autorelabel.service
● ├─rhel-configure.service

● ├─rhel-loadmodules.service
● ├─rockstor-hdparm.service
● ├─selinux-policy-migrate-local-changes@targeted.service

● │ ├─plymouth-read-write.service
● │ ├─plymouth-start.service

● │ ├─sys-fs-fuse-connections.mount

● │ ├─systemd-ask-password-console.path
● │ ├─systemd-binfmt.service
● │ ├─systemd-firstboot.service
● │ ├─systemd-hwdb-update.service
● │ ├─systemd-journal-catalog-update.service

● │ ├─systemd-machine-id-commit.service
● │ ├─systemd-modules-load.service

● │ ├─systemd-update-done.service

I’ve tried again to enable Rock-ons which ended in the following message via a pop-up “Unknown internal error doing a POST to /api/sm/services/docker/start”

exelan · January 14, 2017, 5:47pm

@glenngould Yeah, that’s quite strange.

I bet that restarting it will fail as well, but go ahead and try it anyway:

systemctl restart rockstor-bootstrap.service

If thats the case, lets take a look at the contents of that unit and see if everything is the way it should be:

systemctl cat rockstor-bootstrap.service

Here’s mine, for example:

[1:0:49][root@portal:pts/0][~]
# systemctl cat rockstor-bootstrap.service

[Unit]
Description=Rockstor bootstrapping tasks
After=rockstor.service
Requires=rockstor.service

[Service]
ExecStart=/opt/rockstor/bin/bootstrap
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

glenngould · January 16, 2017, 6:27am

@exelan yes, restarting fails

[root@rockstor ~]# systemctl restart rockstor-bootstrap.service
Job for rockstor-bootstrap.service failed because the control process exited with error code. See "systemctl status rockstor-bootstrap.service" and "journalctl -xe" for details.

Running “systemctl status rockstor-bootstrap.service” as stated above shows the following

● rockstor-bootstrap.service - Rockstor bootstrapping tasks
   Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2017-01-16 07:21:09 CET; 26min ago
  Process: 1289 ExecStart=/opt/rockstor/bin/bootstrap (code=exited, status=1/FAILURE)
 Main PID: 1289 (code=exited, status=1/FAILURE)

Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 16 07:21:09 rockstor bootstrap[1289]: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: ['Internal Server Error: No JSON object could be decoded']

and “journalctl -xe” this

Jan 16 07:21:09 rockstor systemd-logind[1814]: Got message type=signal sender=:1
Jan 16 07:21:09 rockstor systemd-logind[1814]: Got message type=signal sender=:1
Jan 16 07:21:09 rockstor systemd-logind[1814]: Got message type=signal sender=or
Jan 16 07:21:09 rockstor polkitd[2127]: Unregistered Authentication Agent for un
Jan 16 07:21:15 rockstor udevadm[1709]: calling: info
Jan 16 07:21:15 rockstor udevadm[1710]: calling: info
Jan 16 07:21:15 rockstor udevadm[1711]: calling: info
Jan 16 07:21:15 rockstor udevadm[1712]: calling: info
Jan 16 07:21:16 rockstor udevadm[1731]: calling: info
Jan 16 07:21:16 rockstor udevadm[1732]: calling: info
Jan 16 07:21:16 rockstor udevadm[1733]: calling: info
Jan 16 07:21:16 rockstor udevadm[1738]: calling: info
Jan 16 07:22:22 rockstor udevadm[2269]: calling: info
Jan 16 07:22:22 rockstor systemd[1]: Got notification message for unit systemd-j
Jan 16 07:22:22 rockstor systemd[1]: systemd-journald.service: Got notification
Jan 16 07:22:22 rockstor systemd[1]: systemd-journald.service: got WATCHDOG=1
Jan 16 07:22:22 rockstor udevadm[2270]: calling: info
Jan 16 07:22:22 rockstor udevadm[2271]: calling: info
Jan 16 07:22:22 rockstor udevadm[2272]: calling: info
Jan 16 07:22:22 rockstor udevadm[2293]: calling: info
Jan 16 07:22:22 rockstor udevadm[2294]: calling: info
Jan 16 07:22:22 rockstor udevadm[2295]: calling: info
Jan 16 07:22:22 rockstor udevadm[2298]: calling: info
lines 979-1001/1001 (END)
Jan 16 07:21:09 rockstor systemd-logind[1814]: Got message type=signal sender=:1.1 destination=n/a object=/
Jan 16 07:21:09 rockstor systemd-logind[1814]: Got message type=signal sender=:1.1 destination=n/a object=/
Jan 16 07:21:09 rockstor systemd-logind[1814]: Got message type=signal sender=org.freedesktop.DBus destinat
Jan 16 07:21:09 rockstor polkitd[2127]: Unregistered Authentication Agent for unix-process:1283:32698025 (s
Jan 16 07:21:15 rockstor udevadm[1709]: calling: info
Jan 16 07:21:15 rockstor udevadm[1710]: calling: info
Jan 16 07:21:15 rockstor udevadm[1711]: calling: info
Jan 16 07:21:15 rockstor udevadm[1712]: calling: info
Jan 16 07:21:16 rockstor udevadm[1731]: calling: info
Jan 16 07:21:16 rockstor udevadm[1732]: calling: info
Jan 16 07:21:16 rockstor udevadm[1733]: calling: info
Jan 16 07:21:16 rockstor udevadm[1738]: calling: info
Jan 16 07:22:22 rockstor udevadm[2269]: calling: info
Jan 16 07:22:22 rockstor systemd[1]: Got notification message for unit systemd-journald.service
Jan 16 07:22:22 rockstor systemd[1]: systemd-journald.service: Got notification message from PID 1138 (WATC
Jan 16 07:22:22 rockstor systemd[1]: systemd-journald.service: got WATCHDOG=1
Jan 16 07:22:22 rockstor udevadm[2270]: calling: info
Jan 16 07:22:22 rockstor udevadm[2271]: calling: info
Jan 16 07:22:22 rockstor udevadm[2272]: calling: info
Jan 16 07:22:22 rockstor udevadm[2293]: calling: info
Jan 16 07:22:22 rockstor udevadm[2294]: calling: info
Jan 16 07:22:22 rockstor udevadm[2295]: calling: info
Jan 16 07:22:22 rockstor udevadm[2298]: calling: info

Running “systemctl cat rockstor-bootstrap.service” shows

# /etc/systemd/system/rockstor-bootstrap.service
    [Unit]
    Description=Rockstor bootstrapping tasks
    After=rockstor.service
    Requires=rockstor.service

    [Service]
    ExecStart=/opt/rockstor/bin/bootstrap
    Type=oneshot
    RemainAfterExit=yes

    [Install]
    WantedBy=multi-user.target

So no difference compared to your example

glenngould · January 19, 2017, 7:22am

I want to thank you @exelan for your support so far. I’m not sure how to move on and get this solved, so I ask you to possible comment on @suman or @Flyer or @phillxnet - who could support best to get this Rock-ons service again up and running on my machine?

Maybe the upcoming next release 3.9.0 and the upgrade itself will solve this issue (?) - I’m have my doubts
Or is the only solution to do a fresh/new installation of Rockstor (last option to me)

Thank you commenting or directing where to go with this.

Regards,
Roland

exelan · January 19, 2017, 7:44pm

@glenngould Hey Roland,

Sorry for the delay, I was on vacation and tried to avoid anything digital. So to answer your first reply:

That’s what I figured, so there must be some issue with the binaries themselves in /bin. The Rock-ons should be secondary to your bigger problem of rockstor-bootstrap.service failing to start, this is a serious deal that will make the Rock-on plugins moot in the bigger picture of your system. I’m sure @phillxnet, @Flyer, and @suman would tend to agree.

What is the current version of your Rockstor build?:

yum info rockstor

exelan

Flyer · January 19, 2017, 10:17pm

Hi guys, here we are although you don’t see us (@suman, @phillxnet and me too)

Starting from @glenngould first message:

Share/Pool : actually Pool usage computes snapshots and metadata too, while share usage is wrong (well, right and wrong: it reports only current data, not considering snapshots data -> open issue, working to patch it )
nslcd.service -> nss & samba so 1) check you’re running Rockstor latest version (thanks @exelan for great support!) 2)IMPORTANT: have yum update to get latest samba&other packages fixing major bug related nss/name resolving (check around forum posts, full of this)

Some basic checking:
systemctl -l and service name (ex systemctl -l docker) returns more infos
systemctl show and service name (ex systemctl show docker) returns other infos (ex ExecStart to check what docker runs on startup)

Guessing: probably yum updates will solve it

Mirko

glenngould · January 20, 2017, 10:45am

@Flyer and @exelan

thanks for getting back.

The Rockstor release/version I’m using is 3.8.16-1 stable as stated in the heading of this thread.
A few minutes ago (before I’ve seen your reply) I thought about trying the latest testing version in order to see if there is any difference - no, no improvments. System is still on 3.8.16-8 now.

Another thing I’ve noted, but also somehow clear with all the troubles we see on my machine…also the bootstrap service is showing up being disabled and can’t be turned on, like Rock-ons.

systemctl -l provides

proc-sys-fs-binfmt_misc.automount
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.0-net-eno1.device
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.1-net-eno2.device
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda.d
  sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb.d
  sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc.d
  sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd.d
  sys-devices-platform-serial8250-tty-ttyS0.device
  sys-devices-platform-serial8250-tty-ttyS1.device
  sys-devices-platform-serial8250-tty-ttyS2.device
  sys-devices-platform-serial8250-tty-ttyS3.device
  sys-module-configfs.device
  sys-subsystem-net-devices-eno1.device
  sys-subsystem-net-devices-eno2.device
  -.mount
  boot.mount
  dev-hugepages.mount
  dev-mqueue.mount
  home.mount
  mnt2-rm_pool.mount
  mnt2-rockstor_rockstor.mount
  proc-fs-nfsd.mount
  run-user-0.mount
  sys-kernel-config.mount
  sys-kernel-debug.mount
  tmp.mount
  var-lib-nfs-rpc_pipefs.mount
  brandbot.path
  systemd-ask-password-plymouth.path
  systemd-ask-password-wall.path
  session-2.scope
  atd.service
  auditd.service
  avahi-daemon.service
  chronyd.service
  crond.service
  dbus.service
  getty@tty1.service
  gssproxy.service
  irqbalance.service
● kdump.service
  kmod-static-nodes.service
  lm_sensors.service
  network.service
  NetworkManager-wait-online.service
  NetworkManager.service
  nmb.service
  polkit.service
  postfix.service
  postgresql.service
  rhel-dmesg.service
  rhel-import-state.service
  rhel-readonly.service
● rockstor-bootstrap.service
  rockstor-pre.service
  rockstor.service
  rsyslog.service
...skipping...
  UNIT
  proc-sys-fs-binfmt_misc.automount
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.0-net-eno1.device
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.1-net-eno2.device
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda.d
  sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb.d
  sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc.d
  sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd.d
  sys-devices-platform-serial8250-tty-ttyS0.device
  sys-devices-platform-serial8250-tty-ttyS1.device
  sys-devices-platform-serial8250-tty-ttyS2.device
  sys-devices-platform-serial8250-tty-ttyS3.device
  sys-module-configfs.device
  sys-subsystem-net-devices-eno1.device
  sys-subsystem-net-devices-eno2.device
  -.mount
  boot.mount
  dev-hugepages.mount
  dev-mqueue.mount
  home.mount
  mnt2-rm_pool.mount
  mnt2-rockstor_rockstor.mount
  proc-fs-nfsd.mount
  run-user-0.mount
  sys-kernel-config.mount
  sys-kernel-debug.mount
  tmp.mount
  var-lib-nfs-rpc_pipefs.mount
  brandbot.path
  systemd-ask-password-plymouth.path
  systemd-ask-password-wall.path
  session-2.scope
  atd.service
  auditd.service
  avahi-daemon.service
  chronyd.service
  crond.service
  dbus.service
  getty@tty1.service
  gssproxy.service
  irqbalance.service
● kdump.service
  kmod-static-nodes.service
  lm_sensors.service
  network.service
  NetworkManager-wait-online.service
  NetworkManager.service
  nmb.service
  polkit.service
  postfix.service
  postgresql.service
  rhel-dmesg.service
  rhel-import-state.service
  rhel-readonly.service
● rockstor-bootstrap.service
  rockstor-pre.service
  rockstor.service
  rsyslog.service
  UNIT
  proc-sys-fs-binfmt_misc.automount
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.0-net-eno1.device
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.1-net-eno2.device
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda.d
  sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb.d
  sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc.d
  sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd.d
  sys-devices-platform-serial8250-tty-ttyS0.device
  sys-devices-platform-serial8250-tty-ttyS1.device
  sys-devices-platform-serial8250-tty-ttyS2.device
  sys-devices-platform-serial8250-tty-ttyS3.device
  sys-module-configfs.device
  sys-subsystem-net-devices-eno1.device
  sys-subsystem-net-devices-eno2.device
  -.mount
  boot.mount
  dev-hugepages.mount
  dev-mqueue.mount
  home.mount
  mnt2-rm_pool.mount
  mnt2-rockstor_rockstor.mount
  proc-fs-nfsd.mount
  run-user-0.mount
  sys-kernel-config.mount
  sys-kernel-debug.mount
  tmp.mount
  var-lib-nfs-rpc_pipefs.mount
  brandbot.path
  systemd-ask-password-plymouth.path
  systemd-ask-password-wall.path
  session-2.scope
  atd.service
  auditd.service
  avahi-daemon.service
  chronyd.service
  crond.service
  dbus.service
  getty@tty1.service
  gssproxy.service
  irqbalance.service
● kdump.service
  kmod-static-nodes.service
  lm_sensors.service
  network.service
  NetworkManager-wait-online.service
  NetworkManager.service
  nmb.service
  polkit.service
  postfix.service
  postgresql.service
  rhel-dmesg.service
  rhel-import-state.service
  rhel-readonly.service
● rockstor-bootstrap.service
  rockstor-pre.service
  rockstor.service
  rsyslog.service
  shellinaboxd.service
  smartd.service
  smb.service
  sshd.service
  session-2.scope
  atd.service
  auditd.service
  avahi-daemon.service
  chronyd.service
  crond.service
  dbus.service
  getty@tty1.service
  gssproxy.service
  irqbalance.service
● kdump.service
  kmod-static-nodes.service
  lm_sensors.service
  network.service
  NetworkManager-wait-online.service
  NetworkManager.service
  nmb.service
  polkit.service
  postfix.service
  postgresql.service
  rhel-dmesg.service
  rhel-import-state.service
  rhel-readonly.service
● rockstor-bootstrap.service
  rockstor-pre.service
  rockstor.service
  rsyslog.service
  shellinaboxd.service
  smartd.service
  smb.service
  sshd.service
  UNIT
  proc-sys-fs-binfmt_misc.automount
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.0-net-eno1.device
  sys-devices-pci0000:00-0000:00:1c.4-0000:03:00.1-net-eno2.device
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-s
  sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda.d
  sys-devices-pci0000:00-0000:00:1f.2-ata2-host1-target1:0:0-1:0:0:0-block-sdb.d
  sys-devices-pci0000:00-0000:00:1f.2-ata3-host2-target2:0:0-2:0:0:0-block-sdc.d
  sys-devices-pci0000:00-0000:00:1f.2-ata4-host3-target3:0:0-3:0:0:0-block-sdd.d
  sys-devices-platform-serial8250-tty-ttyS0.device
  sys-devices-platform-serial8250-tty-ttyS1.device
  sys-devices-platform-serial8250-tty-ttyS2.device
  sys-devices-platform-serial8250-tty-ttyS3.device
  sys-module-configfs.device
  sys-subsystem-net-devices-eno1.device
  sys-subsystem-net-devices-eno2.device
  -.mount
  boot.mount
  dev-hugepages.mount
  dev-mqueue.mount
  home.mount
  mnt2-rm_pool.mount
  mnt2-rockstor_rockstor.mount
  proc-fs-nfsd.mount
  run-user-0.mount
  sys-kernel-config.mount
  sys-kernel-debug.mount
  tmp.mount
  var-lib-nfs-rpc_pipefs.mount
  brandbot.path
  systemd-ask-password-plymouth.path
  systemd-ask-password-wall.path
  session-2.scope
  atd.service
  auditd.service
  avahi-daemon.service
  chronyd.service
  crond.service
  dbus.service
  getty@tty1.service
  gssproxy.service
  irqbalance.service
● kdump.service
  kmod-static-nodes.service
  lm_sensors.service
  network.service
  NetworkManager-wait-online.service
  NetworkManager.service
  nmb.service
  polkit.service
  postfix.service
  postgresql.service
  rhel-dmesg.service
  rhel-import-state.service
  rhel-readonly.service
● rockstor-bootstrap.service
  rockstor-pre.service
  rockstor.service
  rsyslog.service
  shellinaboxd.service
  smartd.service
  smb.service
  sshd.service
  systemd-fsck@dev-disk-by\x2duuid-689fbd0f\x2dc129\x2d444e\x2da839\x2d2784053b4
  systemd-journal-flush.service
  systemd-journald.service
  systemd-logind.service
  systemd-random-seed.service
  systemd-readahead-collect.service
  systemd-readahead-replay.service
  systemd-remount-fs.service
  systemd-sysctl.service
  systemd-tmpfiles-setup-dev.service
  systemd-tmpfiles-setup.service
  systemd-udev-trigger.service
  systemd-udevd.service
  systemd-update-utmp.service
  systemd-user-sessions.service
  systemd-vconsole-setup.service
  yum-cron.service
  -.slice
  system-getty.slice
  system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice
  system-systemd\x2dfsck.slice
  system.slice
  user-0.slice
  user.slice
  avahi-daemon.socket
  dbus.socket
  docker.socket
  rpcbind.socket
  systemd-initctl.socket
  systemd-journald.socket
  systemd-shutdownd.socket
  systemd-udevd-control.socket
  systemd-udevd-kernel.socket
  dev-disk-by\x2duuid-7e64af26\x2d03e8\x2d4944\x2da704\x2dd27f8a1f42e3.swap
  basic.target
  cryptsetup.target
  getty.target
  graphical.target
  local-fs-pre.target
  local-fs.target
  multi-user.target
  network-online.target
  network.target
  nfs-client.target
  paths.target
  remote-fs-pre.target
  remote-fs.target
  slices.target
  sockets.target
  swap.target
  sysinit.target
  timers.target
  systemd-readahead-done.timer
  systemd-tmpfiles-clean.timer

systemctl show docker provides

Type=simple
Restart=no
NotifyAccess=none
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
WatchdogUSec=0
WatchdogTimestampMonotonic=0
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=success
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
lines 1-23...skipping...
Type=simple
Restart=no
NotifyAccess=none
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
WatchdogUSec=0
WatchdogTimestampMonotonic=0
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=success
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/opt/rockstor/bin/docker-wrapper ; argv[]=/opt/rockstor/bin/docker-wrapper /mnt2/rock-ons-root3 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
MemoryCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=1048576
LimitAS=18446744073709551615
LimitNPROC=1048576
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=39736
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
MountFlags=524288
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=basic.target docker.socket rockstor-bootstrap.service
Wants=system.slice
WantedBy=multi-user.target
ConsistsOf=docker.socket
Conflicts=shutdown.target
Before=multi-user.target shutdown.target
After=system.slice rockstor-bootstrap.service network.target docker.socket systemd-journald.socket basic.target
TriggeredBy=docker.socket
Documentation=http://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/etc/systemd/system/docker.service
UnitFileState=enabled
UnitFilePreset=disabled
InactiveExitTimestampMonotonic=0
ActiveEnterTimestampMonotonic=0
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=no
AssertResult=no
ConditionTimestampMonotonic=0
AssertTimestampMonotonic=0
Transient=no

systemctl -l docker
Unknown operation ‘docker’.

Flyer · January 20, 2017, 11:00am

Hi @glenngould,
sorry, my fault, missing argument:

systemctl status -l rockstor-bootstrap to check Rockstor Bootstrap
systemctl status -l docker to check docker

M.

glenngould · January 20, 2017, 11:26am

Hi @Flyer,

thanks for your support - no problem for missing arguments…

systemctl status -l rockstor-bootstrap

● rockstor-bootstrap.service - Rockstor bootstrapping tasks
   Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2017-01-20 11:00:10 CET; 1h 25min ago
  Process: 15353 ExecStart=/opt/rockstor/bin/bootstrap (code=exited, status=1/FAILURE)
 Main PID: 15353 (code=exited, status=1/FAILURE)

Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: ['Internal Server Error: No JSON object could be decoded']
Jan 20 11:00:10 rockstor bootstrap[15353]: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: ['Internal Server Error: No JSON object could be decoded']

systemctl status -l docker

● docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: http://docs.docker.com

Flyer · January 20, 2017, 1:17pm

Ok,
this doesn’t give us more info, but we can debug it: boostrap service runs scripts/bootstrap.py and your boostrap fails after 15 attempts, so we’re failing here:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/scripts/bootstrap.py#L51-L71


num_attempts = 0
while True:
    try:
        aw = APIWrapper()
        time.sleep(2)
        aw.api_call('network')
        aw.api_call('commands/bootstrap', calltype='post')
        break
    except Exception as e:
        # Retry on every exception, primarily because of django-oauth
        # related code behaving unpredictably while setting
        # tokens. Retrying is a decent workaround for now(11302015).
        if (num_attempts > 15):
            print('Max attempts(15) reached. Connection errors persist. '
                  'Failed to bootstrap. Error: %s' % e.__str__())
            sys.exit(1)
        print('Exception occured while bootstrapping. This could be '
              'because rockstor.service is still starting up. will '
              'wait 2 seconds and try again. Exception: %s' % e.__str__())
        time.sleep(2)
        num_attempts += 1

My idea, considering bootstrap.py try...except code

I assume APIWrapper call to network is ok and commands/bootstrap failing, with bootstrap command. Can you provide rockstor logs with cat /opt/rockstor/var/log/rockstor.log ? fortunately bootstrap under commands view has lot of logging helping us

Mirko