Docker won't start: device or resource busy

Support
1

Brief description of the problem

Docker daemon won’t start as it indicates that the device or resource are busy while deleting tmp-old folder and creating tmp folder.
I also have problems when stopping Rockons, as they won’t stop. Also If I uninstall a Rockon it will break it and I need to manually delete the container, the image, reboot and then delete the rockon metadata on the DB using the script in /opt/rockstor/bin. I think that this might be due to the device or resource busy.

I used lsof to see if something was writing on the folder, but nothing was being done.

Detailed step by step instructions to reproduce the problem

Restarting Rockstor will break it, the only way of fixing it is restarting until it starts properly. Sometimes the restart just hangs and the server won’t restart until a hard reset is done.

Web-UI screenshot

No web UI error, it can only be seen on output of systemctl status docker -l
Right now I have some services running, will provide screenshot as soon as possible.

Error Traceback provided on the Web-UI

No UI error it just shows that the Rockon service is turned OFF

2

@DarkYamik, what version are you currently running?

yum info rockstor

Also, can you check whether you might have a really long scrubbing or balancing job running in BTRFS?

1 Like
3

Hello @Hooverdan,

Thank you for your reply.

I currently have version 3.9.2 Stable running. Here is the full output of the command

yum info rockstor

Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
base: ftp.csuc.cat
epel: ftp.fau.de
extras: ftp.csuc.cat
updates: ftp.csuc.cat
Installed Packages
Name : rockstor
Arch : x86_64
Version : 3.9.2
Release : 57
Size : 85 M
Repo : installed
From repo : Rockstor-Stable
Summary : Btrfs Network Attached Storage (NAS) Appliance.
URL : http://rockstor.com/
License : GPL
Description : Software raid, snapshot capable NAS solution with built-in file integrity protection.

: Allows for file sharing between network attached devices.

A month ago I created another issue regarding having a Quota rescan running for more than a month (now two months) which I think it might be creating the issue. Quotas appear as disabled on Rockstor but the reescan is running and disabling quotas from SSH freezes Rockstor.

No balance or scrub is running on the pool that has the Rockon share.

I just launched a manual Balance with the Force setting will launch a scrub to see if this has some impact on the issue.

Please let me know if I can provide more information.

Kind regards

4

Thanks for the additional info @DarkYamik,

Maybe the system logs might give some info:

systemctl status docker

and/or

journalctl -u docker

Hope this helps,

1 Like
5

Hello @Flox,

Thank you for tuning in!

I’ve run the Balance on the pool and rebooted Rockstor. It froze during reboot and had to hard reset it.

Upon rebooting the issue appeared again, here are the outputs of the commands:

systemctl status docker
docker.service - Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-06-09 20:56:19 CEST; 1min 26s ago
Docs: http://docs.docker.com
Process: 6142 ExecStart=/opt/rockstor/bin/docker-wrapper /mnt2/rockon_service (code=exited, status=1/FAILURE)
Main PID: 6142 (code=exited, status=1/FAILURE)

Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: File “/opt/rockstor/bin/docker-wrapper”, line 44, in
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: sys.exit(scripts.docker_wrapper.main())
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: File “/opt/rockstor/src/rockstor/scripts/docker_wrapper.py”, line 52, in main
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: run_command(cmd)
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: File “/opt/rockstor/src/rockstor/system/osi.py”, line 176, in run_command
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: raise CommandException(cmd, out, err, rc)
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: system.exceptions.CommandException: Error running a command. cmd = /usr/bin/dockerd --log-driver=journald --storage-driver btrfs --storage-opt btrfs.min_space=1G --dat…l=info msg="libc
Jun 09 20:56:19 yaminasamdv2 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jun 09 20:56:19 yaminasamdv2 systemd[1]: Unit docker.service entered failed state.
Jun 09 20:56:19 yaminasamdv2 systemd[1]: docker.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

journalctl -u docker
– Logs begin at Tue 2020-06-09 20:55:02 CEST, end at Tue 2020-06-09 21:11:35 CEST. –
Jun 09 20:56:17 yaminasamdv2 systemd[1]: Started Docker Application Container Engine.
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: /opt/rockstor/eggs/setuptools-46.1.3-py2.7.egg/pkg_resources/py2_warn.py:21: UserWarning: Setuptools will stop working on Python 2
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: ************************************************************
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: You are running Setuptools on Python 2, which is no longer
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: supported and
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: >>> SETUPTOOLS WILL STOP WORKING <<<
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: in a subsequent release (no sooner than 2020-04-20).
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: Please ensure you are installing
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: Setuptools using pip 9.x or later or pin to setuptools<45
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: in your environment.
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: If you have done those things and are still encountering
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: this message, please follow up at
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: https://bit.ly/setuptools-py2-warning.
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: ************************************************************
Jun 09 20:56:17 yaminasamdv2 docker-wrapper[6142]: sys.version_info < (3,) and warnings.warn(pre + “" * 60 + msg + "” * 60)
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: Traceback (most recent call last):
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: File “/opt/rockstor/bin/docker-wrapper”, line 44, in
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: sys.exit(scripts.docker_wrapper.main())
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: File “/opt/rockstor/src/rockstor/scripts/docker_wrapper.py”, line 52, in main
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: run_command(cmd)
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: File “/opt/rockstor/src/rockstor/system/osi.py”, line 176, in run_command
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: raise CommandException(cmd, out, err, rc)
Jun 09 20:56:18 yaminasamdv2 docker-wrapper[6142]: system.exceptions.CommandException: Error running a command. cmd = /usr/bin/dockerd --log-driver=journald --storage-driver btrfs --storage-opt btrfs.min_space=1G --data-root /mnt2/rockon
Jun 09 20:56:19 yaminasamdv2 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jun 09 20:56:19 yaminasamdv2 systemd[1]: Unit docker.service entered failed state.
Jun 09 20:56:19 yaminasamdv2 systemd[1]: docker.service failed.

After this, I rebooted the server again and Docker started correctly with all the containers running.

Please let me know If I can provide more information

6

Maybe that one might have provided some hint, but I should have detailed to use the -l flag as well… my apologies.

I’m glad you’re sorted now, at least!

Thanks a lot for the offer… I’m not sure we’ll be able to go back to that exact error unless we can reproduce it. Even then, we have changed quite a bit how we implement Docker in our openSUSE base to rely as much as possible on its default setup to improve its robustness; hopefully it’ll help with these kind of situations by now as well.

Thanks again for your help!

1 Like