Rock-on stuck in "loading" state

Rock-ons
1

Hi,

I encountered an odd behavior with one of my Rock-ons resulting in a somewhat “zombie” container: seems apparently frozen, but cannot be stopped, restarted, removed, etc…, from the command line with either docker tools. Everything else on the machine was running fine and based on a quick Google search seems to happen on non-Rockstor boxes as well (thus suggesting a Docker-related issue).
The fix is easy as rebooting the machine does the trick, but in the meantime, I believe I managed to get the Rockstor database confused with regards to this specific Rock-on. Indeed, in my attempts at stopping the Rock-on, I naturally toggled the Rock-on switch on the web-UI, but since then, this Rock-on remains stuck on the “loading/refreshing” state. This is on the webUI only, as after rebooting the machine everything is running smoothly, including this container.
This problem remains after deleting browser cache or using another machine to connect to the webUI, so this is the reason why I’m suspecting that something is stuck in the Rockstor Rock-on database.

Is there a way to inspect the relevant info in the database and manually fix these, or should I simply use the delete_rockon() script described below?

Would the latter even fix this problem?

I thought I would ask for feedback before trying the delete_rockon script, so thanks in advance for any input!

2

@Flox Hello again.

I think it’s worth a try, all it does is create a reference to the Rock-on db object, by name (if it exists), and use it to stop, rm, and rmi, all related docker bits. It carries out these later steps in order even if one fails. And finally it deletes the RockOn db object. Take a look at the programs source here:

You should then have a clean slate docker and Rock-on db wise with regard to the named Rock-on. Allowing for a fresh install. Does seem like overkill when it’s working as is but given that the data and config are in unaffected shares, assuming you configured it as per the docs, then a re-install should put docker and Rock-ons db representation back in step at least where this Rock-on is concerned. Bit of a pain this and happens less often than it use to so we are heading in the right direction but we definitely need to tighten things up but given we also have such behaviour as you have indicated in base docker and for example this:

It’s all very much a work in progress that has leapt into mass use probably a little ahead of it’s time. Oh well; more of a challenge I suppose.

Hope that helps and let us know how it goes.

The ‘hand editing of the db’ approach is far more prone to error and a lot more fiddly, plus we have a number of ForeignKey relationships between the related models that could easily get missed. You can take a look at the models definitions here:

3

Hi @phillxnet,

I haven’t had time to spend time on this since the last message here, but when I tried this morning, I can’t seem to figure out how to use that script.

 [root@rockstor scripts]# pwd
/opt/rockstor/src/rockstor/scripts
[root@rockstor scripts]# python rockon_delete.py
Traceback (most recent call last):
  File "rockon_delete.py", line 22, in <module>
    from storageadmin.models import (RockOn, DContainer)
ImportError: No module named storageadmin.models

I remember trying this script a long time ago for a different situation and have it worked but this time I feel stupid not being able to repeat that again.
Am I missing something obvious here or was some dependency altered in some way by update?

Here’s my system version:

yum info rockstor
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.dal10.us.leaseweb.net
 * epel: reflector.westga.edu
 * extras: mirror.teklinks.com
 * updates: centos.aol.com
Installed Packages
Name        : rockstor
Arch        : x86_64
Version     : 3.9.2
Release     : 11
Size        : 77 M
Repo        : installed
From repo   : Rockstor-Stable
Summary     : RockStor -- Store Smartly
License     : GPL
Description : RockStor -- Store Smartly

Available Packages
Name        : rockstor
Arch        : x86_64
Version     : 3.9.2
Release     : 23
Size        : 15 M
Repo        : Rockstor-Stable
Summary     : RockStor -- Store Smartly
License     : GPL
Description : RockStor -- Store Smartly
4

@Flox Hello again.

I think what you are looking for is:

 /opt/rockstor/bin/delete-rockon

Delete metadata, containers and images of a Rock-on
	Usage: /opt/rockstor/bin/delete-rockon <rockon name>

ie an installed (built) system has the above convenience wrapper that sets up the required environment for the script. And with no parameters give it outputs the above ‘guide’ as to it’s use.

Hope that helps. You may have to play with getting the exact Rock-on name but pretty sure it’s the displayed name.

Also it looks like you are several versions of Rockstor behind, I haven’t noticed any issues with the interim versions re updates so you might like to upgrade to the latest as 3.9.2-11 was 3 months ago now.

5

Thanks a lot, I thought about that but somehow failed to find it :frowning:

Yes, it’s on my list… I simply hadn’t had the time to do anything to it lately until now.

Thanks a lot for the help, I’ll try again and get back to you to hopefully confirm it worked.

6

@Flox Yes: we really need to document it’s use somewhere, maybe we could have a wiki entry given it’s kind of an emergency ‘brute force’ type thing and as it’s currently only cli it’s a little unfriendly. But comes in handy with the occasional ‘confused’ Rock-on state.

I’ve just looked up a prior post indicating it’s origin:

That forum post has the details of when @suman added this emergency rock-on removal script.

Any ideas on how / where we might best provide a Web-UI trigger for this same script would be welcome? I think it should be independent of the existing Rock-on tab listings as the ‘installed’ tab entries are what seem to end up breaking when this script is required so another location that also makes sense would be good. Maybe we need some kind of advance section that surfaces such things?

7

Thanks for your help @phillxnet, this worked and I could remove the Rock-on that was causing problem.

I agree with you that it would be a nice addition in a location separate from Rock-ons. If a web-UI trigger for other scripts like that could prove useful, how about creating a separate section under the “System” menu? maybe something labeled “Advanced tasks” with a warning on the potential danger of each script?

8

@Flox Thanks of the update / feedback.

Yes I like this idea, although maybe “Advanced Utils” to avoid the association with Scheduled tasks?

Cheers and glad that script worked for you. We will have to do a little more debugging on the Rock-on stuff going forward as this does keep happening occasionally. Although I’m not completely sold on docker’s maturity / stability as of yet anyway, ie:

Oh well, bit by bit.

1 Like
9

Agreed, it is much better and avoids unnecessary confusion.