[SOLVED] Rockstor WebUI won't start after reboot due to problems with initrock

Tried to ‘upgrade’ a NextCloud Rockon (that wraps the undeadhunter/nextcloud-letsencrypt docker container), by uninstall/install the rockon.

As it did not have any effect, I thought a reboot of the server would not hurt.

But the webui would not start any more.

Checking which services failed it showed the that the rockstor-pre.service had failed.

Looking into that it showed that the initrock python script had failed:

[root@rocky rockons-metastore]# /opt/rockstor/bin/initrock
2017-04-20 02:07:18,373: Supported kernel(/boot/vmlinuz-4.10.6-1.el7.elrepo.x86_64) is already the default
2017-04-20 02:07:18,605: /etc/rc.d/rc.local looks correct. Not updating.
2017-04-20 02:07:18,606: Checking for flash and Running flash optimizations if appropriate.
2017-04-20 02:07:20,816: Updating the timezone from the system
2017-04-20 02:07:20,817: system timezone = Europe/Berlin
2017-04-20 02:07:20,820: Changed timezone from America/Chicago to Europe/Berlin
2017-04-20 02:07:20,826: Updating sshd_config
2017-04-20 02:07:20,830: sshd_config already has the updates. Leaving it unchanged.
2017-04-20 02:07:20,831: Running app database migrations…
Traceback (most recent call last):
File “/opt/rockstor/bin/initrock”, line 45, in
sys.exit(scripts.initrock.main())
File “/opt/rockstor/src/rockstor/scripts/initrock.py”, line 421, in main
run_command(migration_cmd + [‘storageadmin’])
File “/opt/rockstor/src/rockstor/system/osi.py”, line 114, in run_command
raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /opt/rockstor/bin/django migrate --noinput storageadmin. rc = 1. stdout = [‘Operations to perform:’, ’ Apply all migrations: storageadmin’, ‘Running migrations:’, ’ Rendering model states… DONE’, ’ Applying storageadmin.0003_auto_20170114_1332…’]. stderr = [‘Traceback (most recent call last):’, ’ File “/opt/rockstor/bin/django”, line 44, in ‘, " sys.exit(djangorecipe.manage.main(‘rockstor.settings’))", ’ File “/opt/rockstor/eggs/djangorecipe-1.9-py2.7.egg/djangorecipe/manage.py”, line 9, in main’, ’ management.execute_from_command_line(sys.argv)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/init.py”, line 354, in execute_from_command_line’, ’ utility.execute()’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/init.py”, line 346, in execute’, ’ self.fetch_command(subcommand).run_from_argv(self.argv)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py”, line 394, in run_from_argv’, ’ self.execute(*args, **cmd_options)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py”, line 445, in execute’, ’ output = self.handle(*args, **options)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/commands/migrate.py”, line 222, in handle’, ’ executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/executor.py”, line 110, in migrate’, ’ self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/executor.py”, line 148, in apply_migration’, ’ state = migration.apply(state, schema_editor)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/migration.py”, line 115, in apply’, ’ operation.database_forwards(self.app_label, schema_editor, old_state, project_state)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/operations/fields.py”, line 62, in database_forwards’, ’ field,’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/schema.py”, line 18, in add_field’, ’ super(DatabaseSchemaEditor, self).add_field(model, field)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/schema.py”, line 398, in add_field’, ’ self.execute(sql, params)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/schema.py”, line 111, in execute’, ’ cursor.execute(sql, params)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py”, line 64, in execute’, ’ return self.cursor.execute(sql, params)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py”, line 98, in exit’, ’ six.reraise(dj_exc_type, dj_exc_value, traceback)’, ’ File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py”, line 64, in execute’, ’ return self.cursor.execute(sql, params)’, ‘django.db.utils.ProgrammingError: column “pqgroup_eusage” of relation “storageadmin_share” already exists’, ‘’, ‘’]
[root@rocky rockons-metastore]#

looks to me like it tries to make sure that its own postgres database schema is up to date, and then has problems applying changes (aka add a field to a relation that already exists).

What should I do?

@seethap Hello again.

Thanks for this report:

Yes looks that way to me too.

@suman and @Flyer this looks related to:

But it seems to be failing on the idempotent front. Ie the db migration command is rc = 1 in the reported scenario: which appears to simply be a repeat call of the db update stuff ie

... column "pqgroup_eusage" of relation "storageadmin_share" already exists', '', '']

I’ll tighten the migration logic even more, but this is the legacy of the original bug. What’s happening here is that the migration was applied to the DB but not registered properly by django. We need to fake apply it once and it will be happy. What’s the output of /opt/rockstor/bin/django showmigrations?

admin
[ ] 0001_initial
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
django_ztask
(no migrations)
oauth2_provider
[X] 0001_initial
[ ] 0002_08_updates
sessions
[ ] 0001_initial
sites
[ ] 0001_initial
smart_manager
[ ] 0001_initial
[ ] 0002_auto_20170216_1212
storageadmin
[X] 0001_initial
[X] 0002_auto_20161125_0051
[ ] 0003_auto_20170114_1332

Ok, but can you tell me how to “apply a fake apply once” so I can get my server up and running agin? Please?

Really sorry for my delayed response. Could you try these commands and capture the output?

/opt/rockstor/bin/django migrate --noinput --database=default storageadmin 0001_initial
/opt/rockstor/bin/django showmigrations --list
/opt/rockstor/bin/django migrate --noinput --database=default storageadmin
/opt/rockstor/bin/django showmigrations --list

This will fake apply the first one(of storageadmin app), unapply the other two and then apply them again.

Not sure that that worked. Here the output

[root@rocky ~]# /opt/rockstor/bin/django migrate --noinput --database=default storageadmin 0001_initial
Operations to perform:
Target specific migration: 0001_initial, from storageadmin
Running migrations:
Rendering model states… DONE
Unapplying storageadmin.0002_auto_20161125_0051… OK
[root@rocky ~]# /opt/rockstor/bin/django showmigrations --list
admin
[ ] 0001_initial
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
django_ztask
(no migrations)
oauth2_provider
[X] 0001_initial
[ ] 0002_08_updates
sessions
[ ] 0001_initial
sites
[ ] 0001_initial
smart_manager
[ ] 0001_initial
[ ] 0002_auto_20170216_1212
storageadmin
[X] 0001_initial
[ ] 0002_auto_20161125_0051
[ ] 0003_auto_20170114_1332
[root@rocky ~]# /opt/rockstor/bin/django migrate --noinput --database=default storageadmin
Operations to perform:
Apply all migrations: storageadmin
Running migrations:
Rendering model states… DONE
Applying storageadmin.0002_auto_20161125_0051… OK
Applying storageadmin.0003_auto_20170114_1332…Traceback (most recent call last):
File “/opt/rockstor/bin/django”, line 44, in
sys.exit(djangorecipe.manage.main(‘rockstor.settings’))
File “/opt/rockstor/eggs/djangorecipe-1.9-py2.7.egg/djangorecipe/manage.py”, line 9, in main
management.execute_from_command_line(sys.argv)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/init.py”, line 354, in execute_from_command_line
utility.execute()
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/init.py”, line 346, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py”, line 394, in run_from_argv
self.execute(*args, **cmd_options)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py”, line 445, in execute
output = self.handle(*args, **options)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/commands/migrate.py”, line 222, in handle
executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/executor.py”, line 110, in migrate
self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/executor.py”, line 148, in apply_migration
state = migration.apply(state, schema_editor)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/migration.py”, line 115, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/operations/fields.py”, line 62, in database_forwards
field,
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/schema.py”, line 18, in add_field
super(DatabaseSchemaEditor, self).add_field(model, field)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/schema.py”, line 398, in add_field
self.execute(sql, params)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/schema.py”, line 111, in execute
cursor.execute(sql, params)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py”, line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column “pqgroup_eusage” of relation “storageadmin_share” already exists

[root@rocky ~]# /opt/rockstor/bin/django showmigrations --list
admin
[ ] 0001_initial
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
django_ztask
(no migrations)
oauth2_provider
[X] 0001_initial
[ ] 0002_08_updates
sessions
[ ] 0001_initial
sites
[ ] 0001_initial
smart_manager
[ ] 0001_initial
[ ] 0002_auto_20170216_1212
storageadmin
[X] 0001_initial
[X] 0002_auto_20161125_0051
[ ] 0003_auto_20170114_1332

OK, I see what’s going on clearly now. If you “fake” apply the 3rd migration, it should be all set

/opt/rockstor/bin/django migrate --noinput --fake --database=default storageadmin 0003_auto

After this, to make sure it doesn’t fail again you can restart the rockstor-pre service a few times and it shouldn’t fail. systemctl restart rockstor-pre; systemctl status -l rockstor-pre

1 Like

Thanks, that did the job :slight_smile: