Replication tasks not starting

danjcutler · December 28, 2016, 12:25am

Hi all,

New Rockstor user here, so apologies if I’m missing something dead obvious. I’ve had a look through the forums and can’t find anything relevant.

I’ve just installed Rockstor to two machines, currently connected over a local network, but will be via VPN once moved to a remote location. I’ve successfully added each machine to the other’s appliances list using a newly generated secret key on each.

I’ve created a number of replication tasks, setting a range of fairly aggressive schedules to test (1min up to hourly), however none of my replication tasks ever start, and as a result none of them ever have any entry in the Last Backup column to click on. The tasks don’t show up in the other machines’ receive replication list either. I’ve tried both ways, and have tried removing all replication tasks, and appliances, and setting up again from scratch.

Any ideas?

Thanks in advance

Dan

Flyer · December 28, 2016, 10:53am

Hi @danjcutler and welcome to Rockstor
Can you please provide some logs from both machines? You can collect them with Rockstor logs manager - rockstor logs, actually your issue doesn’t seem related to system services like samba, ntp, nginx etc
BR
Mirko

danjcutler · December 29, 2016, 10:35am

Thanks Mirko,

I had a look through the logs initially but didn’t see anything that struck me as relevant - hopefully you’ll have more luck:

Sender nas:

    ConnectionError: HTTPSConnectionPool(host='GARDNERNAS', port=443): Max retries exceeded with url: /o/token/ (Caused by : [Errno -5] No address associated with hostname)
[27/Dec/2016 10:43:40] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:44:33] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 64, in set_token
    raise Exception(msg)
Exception: Exception while setting access_token for url(https://10.0.0.133:443). Make sure credentials are correct: 'access_token'
[27/Dec/2016 10:44:33] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:45:45] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 64, in set_token
    raise Exception(msg)
Exception: Exception while setting access_token for url(https://10.0.0.133:443). Make sure credentials are correct: 'access_token'
[27/Dec/2016 10:45:45] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:47:13] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 53, in set_token
    verify=False)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 87, in post
    return request('post', url, data=data, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='gardnernas', port=443): Max retries exceeded with url: /o/token/ (Caused by : [Errno -2] Name or service not known)
[27/Dec/2016 10:47:13] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:47:57] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 53, in set_token
    verify=False)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 87, in post
    return request('post', url, data=data, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='gardnernas', port=443): Max retries exceeded with url: /o/token/ (Caused by : [Errno -5] No address associated with hostname)
[27/Dec/2016 10:47:57] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:53:20] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 53, in set_token
    verify=False)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 87, in post
    return request('post', url, data=data, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='gardnernas', port=443): Max retries exceeded with url: /o/token/ (Caused by : [Errno -2] Name or service not known)
[27/Dec/2016 10:53:20] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:53:47] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 53, in set_token
    verify=False)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 87, in post
    return request('post', url, data=data, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='gardnernas', port=443): Max retries exceeded with url: /o/token/ (Caused by : [Errno -2] Name or service not known)
[27/Dec/2016 10:53:47] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:56:11] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 53, in set_token
    verify=False)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 87, in post
    return request('post', url, data=data, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='gardnernas', port=443): Max retries exceeded with url: /o/token/ (Caused by : [Errno -2] Name or service not known)
[27/Dec/2016 10:56:11] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:56:45] ERROR [storageadmin.util:47] exception: Failed to authenticate on remote appliance. Verify port number, id and secret are correct and try again.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/appliances.py", line 63, in _get_remote_appliance
    url=base_url)
  File "/opt/rockstor/src/rockstor/cli/rest_util.py", line 53, in set_token
    verify=False)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 87, in post
    return request('post', url, data=data, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='gardnernas', port=443): Max retries exceeded with url: /o/token/ (Caused by : [Errno -2] Name or service not known)
[27/Dec/2016 10:56:45] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 10:59:44] ERROR [storageadmin.views.network:85] Unknown ctype: bridge config: {}
[27/Dec/2016 11:15:03] DEBUG [smart_manager.data_collector:804] Python-socketio listening on port http://127.0.0.1:8001
[27/Dec/2016 11:15:12] DEBUG [storageadmin.views.command:152] Bootstrap operations completed
[27/Dec/2016 11:21:38] ERROR [storageadmin.util:47] exception: Replica(1) is enabled. If you are sure, disable it first and then delete.
Traceback (most recent call last):
  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run
    client, addr = self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable
[27/Dec/2016 11:21:38] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 11:41:30] ERROR [storageadmin.views.network:85] Unknown ctype: bridge config: {}
[27/Dec/2016 11:53:01] DEBUG [storageadmin.views.rockon_helpers:70] Attempted to remove a container(bittorrent-btsync). out: [''] err: ['Error response from daemon: no such id: bittorrent-btsync', 'Error: failed to remove containers: [bittorrent-btsync]', ''] rc: 1.
[27/Dec/2016 11:53:06] DEBUG [storageadmin.views.rockon:84] Rockon(BTSync) is in pending state
[27/Dec/2016 11:53:06] DEBUG [storageadmin.views.rockon:84] Rockon(BTSync) is in pending state
[27/Dec/2016 11:53:21] DEBUG [storageadmin.views.rockon_helpers:127] exception while installing the Rockon(20)
[27/Dec/2016 11:53:21] ERROR [storageadmin.views.rockon_helpers:128] Error running a command. cmd = ['/usr/bin/docker', 'run', '-d', '--restart=unless-stopped', '--name', 'bittorrent-btsync', '-v', '/mnt2/Music:/mnt/sync', '-v', '/etc/localtime:/etc/localtime:ro', '-p', '55555:55555/tcp', '-p', '55555:55555/udp', '-p', '8888:8888/tcp', 'bittorrent/sync']. rc = 1. stdout = ['']. stderr = ["Unable to find image 'bittorrent/sync:latest' locally", 'latest: Pulling from bittorrent/sync', '2fa9c5fc7cbe: Pulling fs layer', 'ac3135b453bf: Pulling fs layer', '266acc0807ea: Pulling fs layer', 'e589529aaccb: Pulling fs layer', 'f588d23d4fa0: Pulling fs layer', 'fe114ccd76fd: Pulling fs layer', 'cebe95436090: Pulling fs layer', 'f852029df4da: Pulling fs layer', '54642a45f186: Pulling fs layer', '06286ffd6f26: Pulling fs layer', 'd77427867c1e: Pulling fs layer', 'c5af9811e743: Pulling fs layer', 'c99582c31eb9: Pulling fs layer', 'e056f0c084f2: Pulling fs layer', 'b8f8d24e3596: Pulling fs layer', 'fe114ccd76fd: Verifying Checksum', 'fe114ccd76fd: Download complete', 'f588d23d4fa0: Verifying Checksum', 'f588d23d4fa0: Download complete', 'c5af9811e743: Verifying Checksum', 'c5af9811e743: Download complete', 'd77427867c1e: Verifying Checksum', 'd77427867c1e: Download complete', 'c99582c31eb9: Verifying Checksum', 'c99582c31eb9: Download complete', 'e056f0c084f2: Verifying Checksum', 'e056f0c084f2: Download complete', 'e589529aaccb: Verifying Checksum', 'e589529aaccb: Download complete', '266acc0807ea: Verifying Checksum', '266acc0807ea: Download complete', '06286ffd6f26: Verifying Checksum', '06286ffd6f26: Download complete', 'ac3135b453bf: Verifying Checksum', 'ac3135b453bf: Download complete', '54642a45f186: Verifying Checksum', '54642a45f186: Download complete', 'f852029df4da: Verifying Checksum', 'f852029df4da: Download complete', 'Pulling repository docker.io/bittorrent/sync', 'Tag latest not found in repository docker.io/bittorrent/sync', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 125, in install
    globals().get('%s_install' % rockon.name.lower(), generic_install)(rockon)
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 216, in generic_install
    run_command(cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/usr/bin/docker', 'run', '-d', '--restart=unless-stopped', '--name', 'bittorrent-btsync', '-v', '/mnt2/Music:/mnt/sync', '-v', '/etc/localtime:/etc/localtime:ro', '-p', '55555:55555/tcp', '-p', '55555:55555/udp', '-p', '8888:8888/tcp', 'bittorrent/sync']. rc = 1. stdout = ['']. stderr = ["Unable to find image 'bittorrent/sync:latest' locally", 'latest: Pulling from bittorrent/sync', '2fa9c5fc7cbe: Pulling fs layer', 'ac3135b453bf: Pulling fs layer', '266acc0807ea: Pulling fs layer', 'e589529aaccb: Pulling fs layer', 'f588d23d4fa0: Pulling fs layer', 'fe114ccd76fd: Pulling fs layer', 'cebe95436090: Pulling fs layer', 'f852029df4da: Pulling fs layer', '54642a45f186: Pulling fs layer', '06286ffd6f26: Pulling fs layer', 'd77427867c1e: Pulling fs layer', 'c5af9811e743: Pulling fs layer', 'c99582c31eb9: Pulling fs layer', 'e056f0c084f2: Pulling fs layer', 'b8f8d24e3596: Pulling fs layer', 'fe114ccd76fd: Verifying Checksum', 'fe114ccd76fd: Download complete', 'f588d23d4fa0: Verifying Checksum', 'f588d23d4fa0: Download complete', 'c5af9811e743: Verifying Checksum', 'c5af9811e743: Download complete', 'd77427867c1e: Verifying Checksum', 'd77427867c1e: Download complete', 'c99582c31eb9: Verifying Checksum', 'c99582c31eb9: Download complete', 'e056f0c084f2: Verifying Checksum', 'e056f0c084f2: Download complete', 'e589529aaccb: Verifying Checksum', 'e589529aaccb: Download complete', '266acc0807ea: Verifying Checksum', '266acc0807ea: Download complete', '06286ffd6f26: Verifying Checksum', '06286ffd6f26: Download complete', 'ac3135b453bf: Verifying Checksum', 'ac3135b453bf: Download complete', '54642a45f186: Verifying Checksum', '54642a45f186: Download complete', 'f852029df4da: Verifying Checksum', 'f852029df4da: Download complete', 'Pulling repository docker.io/bittorrent/sync', 'Tag latest not found in repository docker.io/bittorrent/sync', '']
[27/Dec/2016 11:53:21] DEBUG [storageadmin.views.rockon:84] Rockon(BTSync) is in pending state
[27/Dec/2016 11:53:59] DEBUG [storageadmin.views.rockon_helpers:70] Attempted to remove a container(bittorrent-btsync). out: [''] err: ['Error response from daemon: no such id: bittorrent-btsync', 'Error: failed to remove containers: [bittorrent-btsync]', ''] rc: 1.
[27/Dec/2016 11:54:02] DEBUG [storageadmin.views.rockon:84] Rockon(BTSync) is in pending state
[27/Dec/2016 11:54:03] DEBUG [storageadmin.views.rockon:84] Rockon(BTSync) is in pending state
[27/Dec/2016 11:54:17] DEBUG [storageadmin.views.rockon:84] Rockon(BTSync) is in pending state
[27/Dec/2016 11:54:17] DEBUG [storageadmin.views.rockon_helpers:127] exception while installing the Rockon(20)
[27/Dec/2016 11:54:17] ERROR [storageadmin.views.rockon_helpers:128] Error running a command. cmd = ['/usr/bin/docker', 'run', '-d', '--restart=unless-stopped', '--name', 'bittorrent-btsync', '-v', '/mnt2/Music:/mnt/sync', '-v', '/etc/localtime:/etc/localtime:ro', '-p', '55555:55555/tcp', '-p', '55555:55555/udp', '-p', '8888:8888/tcp', 'bittorrent/sync']. rc = 1. stdout = ['']. stderr = ["Unable to find image 'bittorrent/sync:latest' locally", 'Pulling repository docker.io/bittorrent/sync', 'Tag latest not found in repository docker.io/bittorrent/sync', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 125, in install
    globals().get('%s_install' % rockon.name.lower(), generic_install)(rockon)
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 216, in generic_install
    run_command(cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 109, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = ['/usr/bin/docker', 'run', '-d', '--restart=unless-stopped', '--name', 'bittorrent-btsync', '-v', '/mnt2/Music:/mnt/sync', '-v', '/etc/localtime:/etc/localtime:ro', '-p', '55555:55555/tcp', '-p', '55555:55555/udp', '-p', '8888:8888/tcp', 'bittorrent/sync']. rc = 1. stdout = ['']. stderr = ["Unable to find image 'bittorrent/sync:latest' locally", 'Pulling repository docker.io/bittorrent/sync', 'Tag latest not found in repository docker.io/bittorrent/sync', '']
[27/Dec/2016 12:11:00] DEBUG [fs.btrfs:713] Quota inconsistency while assigning 0/3054. Rescan scheduled.
[27/Dec/2016 12:11:22] ERROR [storageadmin.util:47] exception: Replica(3) is enabled. If you are sure, disable it first and then delete.
Traceback (most recent call last):
  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run
    client, addr = self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable
[27/Dec/2016 12:11:22] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[27/Dec/2016 12:16:54] ERROR [storageadmin.views.network:85] Unknown ctype: bridge config: {}
[27/Dec/2016 12:31:52] ERROR [storageadmin.views.network:85] Unknown ctype: bridge config: {}
[27/Dec/2016 12:32:10] ERROR [storageadmin.views.network:85] Unknown ctype: bridge config: {}
[27/Dec/2016 14:32:46] DEBUG [smart_manager.data_collector:804] Python-socketio listening on port http://127.0.0.1:8001
[27/Dec/2016 14:32:53] DEBUG [storageadmin.views.command:152] Bootstrap operations completed
[28/Dec/2016 11:34:03] ERROR [storageadmin.util:47] exception: save() prohibited to prevent data loss due to unsaved related object 'group'.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/user.py", line 165, in post
    suser.save()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 685, in save
    "unsaved related object '%s'." % field.name
ValueError: save() prohibited to prevent data loss due to unsaved related object 'group'.
[28/Dec/2016 11:34:03] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[28/Dec/2016 11:34:34] ERROR [storageadmin.util:47] exception: User(cutler) already exists. Please choose a different username
Traceback (most recent call last):
  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run
    client, addr = self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable
[28/Dec/2016 11:34:34] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.15-14
[29/Dec/2016 20:55:47] DEBUG [smart_manager.data_collector:804] Python-socketio listening on port http://127.0.0.1:8001
[29/Dec/2016 20:55:56] DEBUG [storageadmin.views.command:152] Bootstrap operations completed

danjcutler · December 29, 2016, 10:36am

Receiver NAS:

    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
IntegrityError: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (5, pbkdf2_sha256$20000$3WlYghpPinx8$GMqd6c5bLByRT68SkaTn7b47lUXQDGb..., null, f, Sam, , , , f, t, 2016-12-28 01:45:58.031537+00).

[28/Dec/2016 12:45:58] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.16-2
[28/Dec/2016 12:46:36] ERROR [storageadmin.util:47] exception: password reset is required to enable admin access. please provide a new password
Traceback (most recent call last):
  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run
    client, addr = self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable
[28/Dec/2016 12:46:36] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.16-2
[28/Dec/2016 12:46:56] ERROR [storageadmin.util:47] exception: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (6, pbkdf2_sha256$20000$dLal7Y6885eC$Z/Nc6j1c+PWHg1LfwWoZ63V6UymEZwf..., null, f, Sam, , , , f, t, 2016-12-28 01:46:56.434038+00).
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/user.py", line 201, in put
    None, new_pw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 187, in create_user
    **extra_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 182, in _create_user
    user.save(using=self._db)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 846, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 885, in _do_insert
    using=using, raw=raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 920, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 974, in execute_sql
    cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
IntegrityError: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (6, pbkdf2_sha256$20000$dLal7Y6885eC$Z/Nc6j1c+PWHg1LfwWoZ63V6UymEZwf..., null, f, Sam, , , , f, t, 2016-12-28 01:46:56.434038+00).

[28/Dec/2016 12:46:56] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.16-2
[28/Dec/2016 12:47:06] ERROR [storageadmin.util:47] exception: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (7, pbkdf2_sha256$20000$tn6M89PLB9aE$yIypP22iHWzEnfXVVCnCb1vF5mFSpqK..., null, f, Sam, , , , f, t, 2016-12-28 01:47:06.680788+00).
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/user.py", line 201, in put
    None, new_pw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 187, in create_user
    **extra_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 182, in _create_user
    user.save(using=self._db)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 846, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 885, in _do_insert
    using=using, raw=raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 920, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 974, in execute_sql
    cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
IntegrityError: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (7, pbkdf2_sha256$20000$tn6M89PLB9aE$yIypP22iHWzEnfXVVCnCb1vF5mFSpqK..., null, f, Sam, , , , f, t, 2016-12-28 01:47:06.680788+00).

[28/Dec/2016 12:47:06] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.16-2
[28/Dec/2016 12:54:16] ERROR [storageadmin.views.network:151] NetworkConnection matching query does not exist.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 149, in update_connection
    dconfig['connection'] = NetworkConnection.objects.get(name=dconfig['connection'])
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get
    self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.
[28/Dec/2016 13:14:30] ERROR [storageadmin.util:47] exception: password reset is required to enable admin access. please provide a new password
Traceback (most recent call last):
  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run
    client, addr = self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable
[28/Dec/2016 13:14:30] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.16-2
[28/Dec/2016 13:14:42] ERROR [storageadmin.util:47] exception: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (8, pbkdf2_sha256$20000$4KwE67RSgebl$8XFHmvTFkXzdKyyyYMKOz44gU+A5zbS..., null, f, andrew, , , , f, t, 2016-12-28 02:14:42.744708+00).
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/user.py", line 201, in put
    None, new_pw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 187, in create_user
    **extra_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 182, in _create_user
    user.save(using=self._db)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 846, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 885, in _do_insert
    using=using, raw=raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 920, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 974, in execute_sql
    cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
IntegrityError: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (8, pbkdf2_sha256$20000$4KwE67RSgebl$8XFHmvTFkXzdKyyyYMKOz44gU+A5zbS..., null, f, andrew, , , , f, t, 2016-12-28 02:14:42.744708+00).

[28/Dec/2016 13:14:42] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.16-2
[29/Dec/2016 08:39:29] ERROR [storageadmin.views.network:151] NetworkConnection matching query does not exist.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 149, in update_connection
    dconfig['connection'] = NetworkConnection.objects.get(name=dconfig['connection'])
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get
    self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.
[29/Dec/2016 20:39:57] ERROR [storageadmin.views.network:151] NetworkConnection matching query does not exist.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 149, in update_connection
    dconfig['connection'] = NetworkConnection.objects.get(name=dconfig['connection'])
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get
    self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.
[29/Dec/2016 20:55:43] DEBUG [storageadmin.views.command:152] Bootstrap operations completed
[29/Dec/2016 20:58:28] ERROR [storageadmin.views.network:151] NetworkConnection matching query does not exist.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 149, in update_connection
    dconfig['connection'] = NetworkConnection.objects.get(name=dconfig['connection'])
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get
    self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.
[29/Dec/2016 21:01:45] ERROR [storageadmin.util:47] exception: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (9, pbkdf2_sha256$20000$R1EtYLjn17xM$SbeiOEwG8fy3Hdh5OA7wR+PqbyLV93m..., null, f, andrew, , , , f, t, 2016-12-29 10:01:45.068486+00).
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/user.py", line 201, in put
    None, new_pw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 187, in create_user
    **extra_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 182, in _create_user
    user.save(using=self._db)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 846, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 885, in _do_insert
    using=using, raw=raw)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 920, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 974, in execute_sql
    cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
IntegrityError: null value in column "last_login" violates not-null constraint
DETAIL:  Failing row contains (9, pbkdf2_sha256$20000$R1EtYLjn17xM$SbeiOEwG8fy3Hdh5OA7wR+PqbyLV93m..., null, f, andrew, , , , f, t, 2016-12-29 10:01:45.068486+00).

[29/Dec/2016 21:01:45] DEBUG [storageadmin.util:48] Current Rockstor version: 3.8.16-2

Flyer · December 29, 2016, 11:53am

Hi @danjcutler, reading your logs from a smartphone - so maybe i’m missing something - first think i noticed was an authentication error: have you paired your machines and do they see each other?

danjcutler · January 2, 2017, 4:04am

Hi Mirko,

Sorry for the delay - went away for NYE.

Yep - I did have errors initially when trying to connect the appliances, however when I removed the entries, rebooted and tried from scratch they successfully paired.

Looking under appliances in each NAS I can confirm that they see each other successfully. I also tried removing the entries and re-entering to fix the replication issue, with no luck.

Unfortunately even after the four day break the replication still hasn’t done anything.

Dan

rhenriks_99 · January 11, 2017, 10:09pm

I just installed rockstor on 2 servers and attempted to configure replication following the tutorials. When the replication send task executes, I get an email shortly thereafter which says:

Failed to start a new Sender for Replication Task(2). Exception: Models aren’t loaded yet.

Has anyone seen that before? I like rockstor and may purchase it but only if I can get this feature to work as backups very important…

Thanks

Flyer · January 11, 2017, 11:21pm

Better call @suman (Rockstor replication dad )

M.

lang.chr86 · February 7, 2017, 8:31am

Hi all

I’m evaluating rockstor as my new NAS OS and am trying to use the replication functionality to
backup my primary NAS to a second one. Sadly I was not able to get it running until now and did also
not found much useful information in the documentation or the community forum. Sadly, this thread
that covers my problem did not provide any helpfull information.

Setup

Two VMs in VirtualBox with Network-Bridge and various HDDs.

Installed 3.8.16-1 with ISO
Updated to 3.8.16-8 via testing updates.

Steps

I tried the following steps in my first attempt to setup replication on the
original 3.8.16-1 version. After this did not work, I updated and tried again. After this I
did a fresh install and tried again but without any luck.

Setup Rockstor
Setup Replication Service with correct Ethernet connection
Pair both appliances with each other with the default generated key-pair (but also tried with other key-pair)
Create pool and share on primary NAS
Activate SMB service and add export for share
Add some data to share
Create backup pool on secondary NAS
Add replication task on sender with 5 minutes interval

After this step I would expect that at some point in the future (approx. 5 minutes) the replication
task would start and I would notice some traffic. But sadly this was never the case. I also
would expect that the replication task is visible in the receive view on the backup NAS.

After some testing I also tried to prepare some read-only snapshots on the primary NAS. But this
did not help either.

I never got any error message or other notification that I was doing something wrong. Also the
rockstor log did not show anything about replication.

I would be very happy if you could point out the thing that I am missing.
Thank you for your work at rockstor.

lang.chr86 · February 9, 2017, 7:26pm

I tried with Rockstor 3.8.14 and with the same setup as before. With this version the replication starts and also the backup machine sees the received share/snapshot. Although the share or its snapshots cannot correctly be exported via SMB to access the stored data.
Same behaviour with 3.8.15-0.