Unable to access web ui, and shares not mounted to file system after reboot

Troubleshooting
1

I have had consistent problems with rockstor since upgrading from 3.8. The web ui has not been functional and when it is I would have the issue of the network shares being available but not connected to the raid. I am now suffering from both issues and have attempted to solve these issues but to no avail.

I believe these issues are all related. After rockstor version 3.8 I
have had nothing but issues in regards to accessing the web ui and
getting shares to mount. First it appears rockstor-boostrap will not
start. Below is the output of systemctl list dependencies that will not
start.
rockstor-bootstrap.service

●   ├─rhel-autorelabel-mark.service
●   ├─rhel-autorelabel.service
●   ├─rhel-configure.service
●   ├─rhel-dmesg.service
●   ├─rhel-loadmodules.service
●   ├─selinux-policy-migrate-local-changes@targeted.service
●   │ ├─plymouth-read-write.service
●   │ ├─plymouth-start.service

Attempting to access shares also does not work. I have two shares and
neither appear to be mounted properly so none of my files are
accessible.
Even creating a new initrock does not solve the problem.

2

Hi @djevo1,
can you provide further infos? ex. Rockstor current version, some logs from /opt/rockstor/var/log/rockstor.log, etc? :slight_smile:

Thanks
Mirko

3

Here is a link to the file https://www.dropbox.com/s/udpwfggffszea5n/rslog.rtf?dl=0

4

@djevo1 and @Flyer Just picking out this bit:

[12/Mar/2017 22:42:09] DEBUG [scripts.scheduled_tasks.snapshot:107] created snapshot at shares/Content2016/snapshots/Content2k16_201703122242
Package upgrade
[20/Mar/2017 20:27:21] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /api/commands/bootstrap method: POST
[20/Mar/2017 20:27:29] ERROR [storageadmin.middleware:33] column storageadmin_share.pqgroup_rusage does not exist LINE 1: …n_share".“rusage”, “storageadmin_share”.“eusage”, "storagead…

Ie the known update problem. I’m wondering if since this has persisted since then, we have some kind of db too many connections, but hidden due to the other db migration issue: @Flyer ?

5

This will drive us crazy :confounded:

Collecting infos around Rockstor forum/Github/etc to catch that small code line mocking us! :

  • Issue related to a db migration
  • that is the first db migration after South to pure Django migration after Django updates

This week end will stop yum updates via WebUI coding and check this hateful one! :rage:
Mirko

6

Ok, I’m curious if this could be the cause to both the web ui issue and the drive mounting issue. Is there a workaround similar to zfs to remount the share properly?

7

@djevo1 just had an upgrade from old Rockstor to 3.9.0-1 and it seems ok, so not helping.

Can you help us with some checks? :slight_smile:
From a shell type this : psql -U rocky -d storageadmin -c "\d storageadmin_share"
When asked for password : rocky (Rockstor default values)

Please provide output with table schema (first part)

Thanks
Mirko

8 
                                       Table "public.storageadmin_share"
      Column      |           Type           |                            Modifi
ers                            
------------------+--------------------------+----------------------------------
-------------------------------
 id               | integer                  | not null default nextval('storage
admin_share_id_seq'::regclass)
 pool_id          | integer                  | not null
 qgroup           | character varying(100)   | not null
 name             | character varying(4096)  | not null
 uuid             | character varying(100)   | 
 size             | bigint                   | not null
 owner            | character varying(4096)  | not null
 group            | character varying(4096)  | not null
 perms            | character varying(9)     | not null
 toc              | timestamp with time zone | not null
 subvol_name      | character varying(4096)  | not null
 replica          | boolean                  | not null
 compression_algo | character varying(1024)  | 
 rusage           | bigint                   | not null
 eusage           | bigint                   | not null
 pqgroup          | character varying(32)    | not null
 pqgroup_eusage   | bigint                   | not null
 pqgroup_rusage   | bigint                   | not null
1 Like
9

Ok, storageadmin_share.pqgroup_rusage does not exist…but exists!, so Django migration had new storageadmin_share new fields, but we still fail :confused:

Who wants to be a millionaire mode on, I “phone a friend” :grin: : @suman any idea about this ? Seems like a kind of “ghost db migration” ?!?

10

Some guessing starting from @djevo1 log:
we have un update, we initrock and then this failing:

[23/Mar/2017 08:50:23] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /api/commands/bootstrap method: POST [23/Mar/2017 08:50:23] ERROR [storageadmin.middleware:33] column storageadmin_share.pqgroup_rusage does not exist LINE 1: ...n_share"."rusage", "storageadmin_share"."eusage", "storagead... ^ Traceback (most recent call last): File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) [etc etc etc]
and finally this

File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 90, in post import_shares(p, request) File "/opt/rockstor/src/rockstor/storageadmin/views/share_helpers.py", line 73, in import_shares shares = [s.name for s in Share.objects.filter(pool=pool)] File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 162, in __iter__ self._fetch_all() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 965, in _fetch_all self._result_cache = list(self.iterator()) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 238, in iterator results = compiler.execute_sql() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 840, in execute_sql cursor.execute(sql, params) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) ProgrammingError: column storageadmin_share.pqgroup_rusage does not exist LINE 1: ...n_share"."rusage", "storageadmin_share"."eusage", "storagead... ^

To @djevo1, can you try this from a shell?
psql -U rocky -d storageadmin -c "SELECT * FROM django_migrations;"
(same rocky pass) - Thanks :slight_smile:

Mirko

11 
 id |       app       |                 name                  |            applied            
----+-----------------+---------------------------------------+-------------------------------
  1 | contenttypes    | 0001_initial                          | 2017-03-23 06:14:44.103263-07
  2 | auth            | 0001_initial                          | 2017-03-23 06:14:44.124905-07
  3 | oauth2_provider | 0001_initial                          | 2017-03-23 06:14:44.155001-07
  4 | storageadmin    | 0001_initial                          | 2017-03-23 06:14:44.179394-07
  5 | storageadmin    | 0002_auto_20161125_0051               | 2017-03-23 06:14:47.579102-07
  6 | storageadmin    | 0003_auto_20170114_1332               | 2017-03-23 06:14:52.97075-07
  7 | contenttypes    | 0002_remove_content_type_name         | 2017-03-23 06:15:10.426857-07
  8 | auth            | 0002_alter_permission_name_max_length | 2017-03-23 06:15:10.512811-07
  9 | auth            | 0003_alter_user_email_max_length      | 2017-03-23 06:15:10.575783-07
 10 | auth            | 0004_alter_user_username_opts         | 2017-03-23 06:15:10.622553-07
 11 | auth            | 0005_alter_user_last_login_null       | 2017-03-23 06:15:10.830232-07
 12 | auth            | 0006_require_contenttypes_0002        | 2017-03-23 06:15:10.855935-07
(12 rows)
12

@phillxnet & @suman checking djevo1 django_migrations table over storageadmin & previous info we can say :

  • share table has required new fields pqgroup_rusage/e_usage
  • Django knows about this migration ( 0003_auto_20170114_1332 over migration table)

…and it should be all fine.

I think it’s more a “weirdo” with some special conditions rather then a pure coding error.

@djevo1 once again thanks for your help, can you provide other outputs?
echo -e 'select * from storageadmin_share; \n select * from storageadmin_pool;' | psql -U rocky -d storageadmin same rocky pass

we echo two sql statements to grab share and pool data from db, psql not supporting multiple command with -c param so we use a pipe

Mirko

13 
 id | pool_id | qgroup | name | uuid | size | owner | group | perms | toc | subvol_name | replica | compression_algo | rusage | eusage | pqgroup | pqgroup_eusage | pqgroup_rusage 
----+---------+--------+------+------+------+-------+-------+-------+-----+-------------+---------+------------------+--------+--------+---------+----------------+----------------
(0 rows)

 id | name | uuid | size | raid | toc | compression | mnt_options | role 
----+------+------+------+------+-----+-------------+-------------+------
(0 rows)
14

?? No pools and no shares?

15

I have a single pool, and two shares.

16

@suman :

but db is empty/seems empty?!? see previous message from @djevo1

@djevo1 do you see your pool and your shares via WebUI?

M.

17

I can see the shares available via afp, but they do not lead to the pool. If I ssh into the box I can find the pool via the mnt2 directory. As stated above I have not been able to access the web-ui.

18

Thanks @djevo1, did not catch it :slight_smile:

Your infos help me on debugging, will try to reproduce it by flushing all pools and shares (my suspicion it’s a missing check when system has no pools/shares)

Mirko

19

Note to @suman & @phillxnet : any idea how we can have no pools & no shares with Rockstor defaults too? (I mean home & root shares plus rockstor default pool) :confused:

M.

20

Hi @djevo1 your no pools no shares is an odd, so had some tests on this.

First one, flushing all shares from PostgreSQL db : our Rockstor is supposed to be self-healing, so on every share rescan we restore our data. First time I test it with an “hard delete” and it’s fine, restoring system shares like root and home too :slight_smile:

Next step, testing: flushing pools too

M.