Error when updating Rock-ons list

Manuel_J_Ortega · August 17, 2019, 5:31pm

Brief description of the problem

Trying to update Rock-ons list.

Detailed step by step instructions to reproduce the problem

Pressing update in All tab in Rock-ons.

Web-UI screenshot

Error Traceback provided on the Web-UI


            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 395, in _get_available
    response = requests.get(remote_root, timeout=10)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 215, in send
    raise Timeout(e)
Timeout: HTTPConnectionPool(host='rockstor.com', port=80): Request timed out. (timeout=10)

phillxnet · August 17, 2019, 7:47pm

@Manuel_J_Ortega Welcome to the Rockstor community.

I’ve just checked the rock-ons server and it looks to be fine. Earlier today I was changing some config on that server and needed to reload the service a few times. It may be that you were just unlucky and caught the time when the web server was re-configuring.

Give it another go and let us know if it succeeds.

Another possibility is that your rockstor system is unable to reach the rock-ons web page; ie gateway/router firewall or the like. To test this you could execute the following in a terminal on the Rockstor machine:

wget rockstor.com/rockons/root.json

the terminal / console output should indicate if the web get (wget) was successful.

Hope that helps.

Manuel_J_Ortega · August 18, 2019, 11:43pm

Hi Philip,

I’ve tried with wget (after installing wget package, it seem not to be installed by default), and it can reach the server and works properly, but the webui is still giving the same failure.

vesper1978 · August 19, 2019, 11:08am

Hi @Manuel_J_Ortega,

Sounds like it’s timing out when the GUI tries to reach it. Can you run

time wget rockstor.com/rockons/root.json

This will return how long it took for the wget to complete. You want to look at “real”, as this is the full amount of time. I suspect it will take longer than 10 seconds, which is how long the GUI waits for a response before timing out.

Manuel_J_Ortega · August 19, 2019, 4:35pm

The response very quick, as you can see…

And now it works form the web-ui !!!
I don’t know why, I’ve only changed the local IP adress of the server in my DCHP router…

phillxnet · August 19, 2019, 5:25pm

@Manuel_J_Ortega Hello again and glad it’s now working as intended.

Yes me neither. But we are in the process of moving our background services over to new servers so maybe your network had cached the older ip. Bit of a long shot though as for the rock-ons that was a few days ago.

Oh well, at least it’s working for you now. Sorry to have not pinned this down further though. But do please report again if you see this again.

@vesper1978 Thanks for you improved test on this one. It prompted a thought that what we could do with is a shell script that more closely mimics what Rockstor actually does in total which is to get the root.json file and then retrieve each of the rock-on json definition files mentioned in that, but it processes (slightly) each in turn. So we would need a tiny delay between each subsequent get of each individual json after the first root.json get. The latest code that does this can be found here:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon.py#L414-L425


def _get_available(self):
    if Service.objects.get(name='docker').config is None:
        # don't fetch if service is not configured.
        return {}


    url_root = settings.ROCKONS.get('remote_metastore')
    remote_root = ('%s/%s' %
                   (url_root, settings.ROCKONS.get('remote_root')))
    msg = ('Error while processing remote metastore '
           'at ({}).').format(remote_root)
    with self._handle_exception(self.request, msg=msg):
        response = requests.get(remote_root, timeout=10)

and then in turn:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon.py#L430-L439


meta_cfg = {}
for k, v in root.items():
    cur_meta_url = '%s/%s' % (url_root, v)
    msg = ('Error while processing Rock-on profile '
           'at ({}).').format(cur_meta_url)
    with self._handle_exception(self.request, msg=msg):
        cur_res = requests.get(cur_meta_url, timeout=10)
        if (cur_res.status_code != 200):
            cur_res.raise_for_status()
        meta_cfg.update(cur_res.json())

So only really a quick contents tally between each json get. But at the server end I do see a bit of a range between how fast all of the files are retrieved in order and I’ve put this down to client machine speed or internet link. Hence my only other thought here was a really slow machine that just couldn’t work it’s way through all the requests in turn before timing out. But the timeout is on each json retrieval in turn so seems pretty unlikely.

Anyway, food for thought and such a script may help in the future with more properly testing this web retrieval in a way completely independant of the Rockstor code which may help with narrowing down where the problem lies in the future. We may for example have to add some kind of delay between each request, or build in a backing off mechanism if a ‘too many connections’ type thing arises. Different than what we see here but related with regard to tuning our services.

Hope this helps and anyone interested is welcome to chip in with robustness measures for the future as it’s a bad user experience when this arises and can be a challenge to exactly duplicate what Rockstor actually does. I’m also considering enabling compression at the web server end for these files so that the request call can unzip them on the fly. All in good time hopefully.

vesper1978 · August 20, 2019, 8:20am

@phillxnet,

I work for a company that has to factor in worst case connection scenarios. I’d like to make some suggestions.

Would it be possible to get the “timeout” value increased from “10” to something like “30” ? Or perhaps looking at a CDN such as CloudFlare (There’s a free version that would work) to to cache rockstor.com and the Rock-On registry so that anyone making a request should be pulling from an Edge location close to them instead of directly from the rockstor.com web server 99% of the time? Or even both?

Implementing increasing the timeout in the code would probably suffice. Implementing CloudFlare would give rockstor.com resiliency against having too many requests coming in as the CDN would do all the heavy lifting.