Reinstall, docker won't start

Troubleshooting
1

I did a reinstall of rockstor and updated to 3.8-2. For the most part everything went smoothy a is working. However docker won’t start.

[root@san01 log]# systemctl status -l docker
docker.service - Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/docker.service; enabled)
Active: failed (Result: exit-code) since Tue 2015-07-14 14:19:52 CDT; 26s ago
Docs: http://docs.docker.com
Process: 14771 ExecStart=/opt/rockstor/bin/docker-wrapper /mnt2/rockons (code=exited, status=1/FAILURE)
Main PID: 14771 (code=exited, status=1/FAILURE)

Jul 14 14:19:52 san01 docker-wrapper[14771]: Traceback (most recent call last):
Jul 14 14:19:52 san01 docker-wrapper[14771]: File “/opt/rockstor/bin/docker-wrapper”, line 37, in
Jul 14 14:19:52 san01 docker-wrapper[14771]: sys.exit(docker_wrapper.main())
Jul 14 14:19:52 san01 docker-wrapper[14771]: File “/opt/rockstor/src/rockstor/docker_wrapper.py”, line 29, in main
Jul 14 14:19:52 san01 docker-wrapper[14771]: run_command([DOCKER, ‘-d’, ‘-s’, ‘btrfs’, ‘-g’, mnt_pt])
Jul 14 14:19:52 san01 docker-wrapper[14771]: File “/opt/rockstor/src/rockstor/system/osi.py”, line 79, in run_command
Jul 14 14:19:52 san01 docker-wrapper[14771]: raise CommandException(out, err, rc)
Jul 14 14:19:52 san01 docker-wrapper[14771]: system.exceptions.CommandException: ’ time=“2015-07-14T14:19:52.163434527-05:00” level=info msg=“Listening for HTTP on unix (/var/run/docker.sock)” time=“2015-07-14T14:19:52.181625633-05:00” level=fatal msg=“Error starting daemon: Couldn’t create Tag store: EOF” '
Jul 14 14:19:52 san01 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jul 14 14:19:52 san01 systemd[1]: Unit docker.service entered failed state.

2

Looks like this is a known issue with docker: https://github.com/docker/docker/issues/10553

Are you repurposing /mnt2/rockons from previous install? Something to try is rm /mnt2/rockons/repositories-btrfs.

Also, I’d make sure you have enough free space on /mnt2/rockons… speaking of which, there are some qgroup related bugs that docker is easily managing to trigger :slight_smile: I am pushing the phase 1 of these fixes with the next update. phase 2 fixes would come with 4.2 kernel. This is only fyi, not to go on too much of a tangent.

3

Originally it was, then I deleted the share and created a new one in the gui.

Tried deleting repositories-btrfs but no difference.

Then I tried btrfs quota enable /mnt2/rockons followed by rm repositories-btrfs which spit out ‘cannot remove disk quota exceeded’. Fun stuff but possibly heading in the right direction. My usage is in the single digits so free space shouldn’t be an issue.

4

I think I know what’s happening. This definitely points to incomplete qgroup management in Rockstor. When you import pools from a previous install, things get even more out of control. You need to enable quota on the pool. This is what I’d do.

  1. btrfs quota enable /mnt2/<pool_name> (for each pool)
  2. remove /mnt2/rockons share
  3. run /opt/rockstor/bin/qgroup-clean (this will free up unnecessary qgroups that cause quota exceed errors)
  4. create new Share for rockons, repeat :slight_smile:
5

I can’t actually rn that script directly as apparently some django variables are undefined.

I’m able to get docker started after deleting the subvolume, recanning all qgroups, and creating a new subvolume.

In the UI however I’m running in to an error when clicking on update to list the available rockons.

“Unknown internal error doing a POST to /api/rockons/update” I will email the .tgz

6

I might know what the problem is. I have 2 interfaces, one directly connected and dedicated to a hypervisor. The UI forces a gateway setting when configuring an interface, and the metric for this link is lower in the routing table than the real interface that can reach the internet.

EDIT: Yup manually editing the interface, deleting the gateway, and systemctl restart network fixed it.

7

New issue!!

Owncloud fails to install. I’ll try otherrs.

[root@san01 log]# tail -f rockstor.log
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py”, line 126, in install
globals()’%s_install’ % rockon.name.lower()
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py”, line 263, in owncloud_install
run_command(cmd)
File “/opt/rockstor/src/rockstor/system/osi.py”, line 79, in run_command
raise CommandException(out, err, rc)
CommandException: ‘9448fd28f08e72b4f20714b5114ea9770a2270efde645e637c4ab67f9657e0b0 Error response from daemon: Cannot start container 9448fd28f08e72b4f20714b5114ea9770a2270efde645e637c4ab67f9657e0b0: Cannot link to a non running container: /owncloud-postgres AS /owncloud/db ‘
[14/Jul/2015 16:16:52] DEBUG [storageadmin.views.rockon_helpers:80] Attempted to remove a container(owncloud). out: [‘owncloud’, ‘’] err: [’’] rc: 0.
[14/Jul/2015 16:16:55] DEBUG [storageadmin.views.rockon_helpers:80] Attempted to remove a container(owncloud-postgres). out: [‘owncloud-postgres’, ‘’] err: [’’] rc: 0.
[14/Jul/2015 16:16:57] DEBUG [storageadmin.views.rockon_helpers:262] docker cmd = [’/usr/bin/docker’, ‘run’, ‘-d’, ‘–name’, u’owncloud-postgres’, ‘-e’, u’POSTGRES_USER=admin’, ‘-e’, u’POSTGRES_PASSWORD=WowPasswordsAreBeingStoredInTheClearInThisLogThatShouldProbablyBeChanged’, ‘-v’, u’/mnt2/owncloud:/var/lib/postgresql/data’, u’postgres’]
[14/Jul/2015 16:22:11] ERROR [storageadmin.views.rockon_helpers:273] Waited too long(300 seconds) for postgres to initialize for owncloud. giving up.
[14/Jul/2015 16:22:11] DEBUG [storageadmin.views.rockon_helpers:262] docker cmd = [’/usr/bin/docker’, ‘run’, ‘-d’, ‘–name’, u’owncloud’, ‘-p’, u’8080:443/tcp’, ‘–link’, u’owncloud-postgres:db’, ‘-v’, u’/mnt2/owncloud:/var/www/owncloud/config’, ‘-v’, u’/mnt2/owncloud:/var/www/owncloud/data’, ‘-v’, ‘/opt/rockstor/certs/rockstor.key:/etc/ssl/private/owncloud.key’, ‘-v’, ‘/opt/rockstor/certs/rockstor.cert:/etc/ssl/certs/owncloud.crt’, ‘-e’, ‘HTTPS_ENABLED=true’, ‘-e’, u’DB_USER=admin’, ‘-e’, u’DB_PASS=WowPasswordsAreBeingStoredInTheClearInThisLogThatShouldProbablyBeChanged’, u’pschmitt/owncloud’]
[14/Jul/2015 16:22:14] DEBUG [storageadmin.views.rockon_helpers:128] exception while installing the rockon
[14/Jul/2015 16:22:14] ERROR [storageadmin.views.rockon_helpers:129] '47e636621aa56d5edbbdbda7dba80583cdc63fdb7706c360678a9f42899da2a6 Error response from daemon: Cannot start container 47e636621aa56d5edbbdbda7dba80583cdc63fdb7706c360678a9f42899da2a6: Cannot link to a non running container: /owncloud-postgres AS /owncloud/db '
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py”, line 126, in install
globals()’%s_install’ % rockon.name.lower()
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py”, line 263, in owncloud_install
run_command(cmd)
File “/opt/rockstor/src/rockstor/system/osi.py”, line 79, in run_command
raise CommandException(out, err, rc)
CommandException: '47e636621aa56d5edbbdbda7dba80583cdc63fdb7706c360678a9f42899da2a6 Error response from daemon: Cannot start container 47e636621aa56d5edbbdbda7dba80583cdc63fdb7706c360678a9f42899da2a6: Cannot link to a non running container: /owncloud-postgres AS /owncloud/db ’

8

This is the key error. ownCloud Rock-on has two containers, one of which is postgres. Postgres container setup is expected to take several seconds but not that long. We have an arbitrary 300 sec wait which expired and postgres is still not setup, hence the install failed. You can try the following:

  1. reinstall the Rock-on
  2. docker logs owncloud-postgres (this could show you why postgres container is taking a long time to setup)
9

docker logs owncloud-postgres turns up…

disk quota exceeded!!

Will I break rockstor if I just globally shut off quotas?
I’m also running into problems with, and may be causing quota problems by, generating a lot of snapshots with my backup scripts.

10

Thanks for alerting us. However, I have removed this already in one of the branches going into next release. But please continue to alert us on anything wrong. given enough eyeballs, all bugs are shallow

11

You are better off waiting for the next release than turning quotas off. But qgroup-clean should help you partially in the meantime. What is the error when you run it?

12

[root@san01 weekly]# /opt/rockstor/bin/qgroup-clean
Traceback (most recent call last):
File “/opt/rockstor/bin/qgroup-clean”, line 34, in
import qgroup_clean
File “/opt/rockstor/src/rockstor/qgroup_clean.py”, line 24, in
from storageadmin.models import Pool
File “/opt/rockstor/src/rockstor/storageadmin/models/init.py”, line 19, in
from pool import Pool
File “/opt/rockstor/src/rockstor/storageadmin/models/pool.py”, line 19, in
from django.db import models
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/init.py”, line 5, in
from django.db.models.query import Q
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/query.py”, line 17, in
from django.db.models.deletion import Collector
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/deletion.py”, line 4, in
from django.db.models import signals, sql
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/sql/init.py”, line 4, in
from django.db.models.sql.subqueries import *
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/sql/subqueries.py”, line 12, in
from django.db.models.sql.query import Query
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/sql/query.py”, line 22, in
from django.db.models.sql import aggregates as base_aggregates_module
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/sql/aggregates.py”, line 9, in
ordinal_aggregate_field = IntegerField()
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/db/models/fields/init.py”, line 116, in init
self.db_tablespace = db_tablespace or settings.DEFAULT_INDEX_TABLESPACE
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/conf/init.py”, line 54, in getattr
self._setup(name)
File “/opt/rockstor/eggs/Django-1.6.2-py2.7.egg/django/conf/init.py”, line 47, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

13

export DJANGO_SETTINGS_MODULE=“settings”; /opt/rockstor/bin/qgroup-clean

14

Got a little farther.

[root@san01 weekly]# export DJANGO_SETTINGS_MODULE=“settings”
[root@san01 weekly]# /opt/rockstor/bin/qgroup-clean
qgroup 261 not in use. deleting
qgroup 262 not in use. deleting
qgroup 263 not in use. deleting
qgroup 264 not in use. deleting
Traceback (most recent call last):
File “/opt/rockstor/bin/qgroup-clean”, line 37, in
sys.exit(qgroup_clean.main())
File “/opt/rockstor/src/rockstor/qgroup_clean.py”, line 37, in main
o, e, rc = run_command([BTRFS, ‘qgroup’, ‘show’, mnt_pt])
File “/opt/rockstor/src/rockstor/system/osi.py”, line 79, in run_command
raise CommandException(out, err, rc)
system.exceptions.CommandException: " ERROR: can’t perform the search - No such file or directory ERROR: can’t list qgroups: No such file or directory "

15

Found the problem, working too fast and doing too many things in parallel. I disabled quotas on my backup pool so my backup could proceed. After re-enabling the script runs fine.

1 Like