Rockstor 3 SSL certificate error when refreshing Rock-ons: a temporary workaround

Flox · October 19, 2021, 11:19am

As has been reported by a few users, our now legacy Rockstor 3 is now facing issues when trying to refresh the list of available Rock-ons.

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

After a bit of investigation, the issue appears related to older certificates used by the python requests package, which comes bundled with its own. Due to the age of these certificates in our legacy Rockstor 3 version, these certificates are no-longer valid, resulting in the error reported in the other forum thread linked above. Now that we know that, we can try to get around it.

Note that the following is a “hack” that should not be considered to be a solution. If successful, it will only allow to fetch the list of available Rock-ons but it will not resolve potential similar issues elsewhere that might arise. This issue is related to the older software used in Rockstor 3, an issue addressed in our Rockstor v4 effort along many other issues. As a result, the workaround detailed below should be considered as such and is provided as a temporary hack to help those who want to take the risk of a hack.

For those willing to give that a try, the call in question that is failing in this case is the following:

github.com

rockstor/rockstor-core/blob/1b861bdb916dce11012379234e8b3608c38e21e0/src/rockstor/storageadmin/views/rockon.py#L480


      
          
          
def _get_available(self):
              if Service.objects.get(name="docker").config is None:
                  # don't fetch if service is not configured.
                  return {}
          
          
    url_root = settings.ROCKONS.get("remote_metastore")
              remote_root = "{}/{}".format(url_root, settings.ROCKONS.get("remote_root"))
              msg = "Error while processing remote metastore at ({}).".format(remote_root)
              with self._handle_exception(self.request, msg=msg):
                  response = requests.get(remote_root, timeout=10)
                  if response.status_code != 200:
                      response.raise_for_status()
                  root = response.json()
          
          
    meta_cfg = {}
              for k, v in root.items():
                  cur_meta_url = "{}/{}".format(url_root, v)
                  msg = "Error while processing Rock-on profile at ({}).".format(cur_meta_url)
                  with self._handle_exception(self.request, msg=msg):
                      cur_res = requests.get(cur_meta_url, timeout=10)

This is thus where we need to do some changes. As mentioned above, the problem here is that the python requests package uses its own CA bundle when making this request, which seems to have expired and is thus rejected. We can, however, force it to use the system’s CA bundle, which should then work fine. Note that there’s a second similar call that would need to be adjusted as well:

github.com

rockstor/rockstor-core/blob/1b861bdb916dce11012379234e8b3608c38e21e0/src/rockstor/storageadmin/views/rockon.py#L490


      
              response = requests.get(remote_root, timeout=10)
              if response.status_code != 200:
                  response.raise_for_status()
              root = response.json()
          
          
meta_cfg = {}
          for k, v in root.items():
              cur_meta_url = "{}/{}".format(url_root, v)
              msg = "Error while processing Rock-on profile at ({}).".format(cur_meta_url)
              with self._handle_exception(self.request, msg=msg):
                  cur_res = requests.get(cur_meta_url, timeout=10)
                  if cur_res.status_code != 200:
                      cur_res.raise_for_status()
                  meta_cfg.update(cur_res.json())
          
          
local_root = settings.ROCKONS.get("local_metastore")
          if os.path.isdir(local_root):
              for f in os.listdir(local_root):
                  fp = "{}/{}".format(local_root, f)
                  msg = "Error while processing Rock-on profile at ({}).".format(fp)
                  with self._handle_exception(self.request, msg=msg):

Now, let’s actually work around that:

run yum update and wait for all updates to be applied. You might even consider a reboot at this point if a lot of packages related to the core system were included. This is to ensure we have an up-to-date system, especially anything related to ssl certificates.
Locate your system certificates. By default, they should be located at /etc/ssl/certs/. Make sure the following does list the certificates.

ls -la /etc/ssl/certs/*

Now, we can tell python requests to use one of this certificates instead of its own. As mentioned above, we need to do this in two different lines of the same file:

sed -i "s:requests.get(remote_root, timeout=10):requests.get(remote_root, timeout=10, verify='/etc/ssl/certs/ca-bundle.crt'):g" /opt/rockstor/src/rockstor/storageadmin/views/rockon.py

sed -i "s:requests.get(cur_meta_url, timeout=10):requests.get(cur_meta_url, timeout=10, verify='/etc/ssl/certs/ca-bundle.crt'):g" /opt/rockstor/src/rockstor/storageadmin/views/rockon.py

for a quick info, the sed -i "s:pattern:replacement:g" file command simply replaces inplace (-i option) the characters that match a specific pattern (pattern) with a new string of characters (replacement).

Inspect the file to make sure have indeed added , verify='/etc/ssl/certs/ca-bundle.crt' to the requests. This will ensure the system’s certificates are used instead of requests’s default ones.

Finally, restart the rockstor service for the changes to take effect:

systemctl restart rockstor

You should now be able to go to Rockstor’s webUI, Rock-ons page, and click the “Update” button to get the list of available Rock-ons.

scrosler · October 21, 2021, 1:37am

Not so lucky. Did I miss a step?

Error:

  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 395, in _get_available
    response = requests.get(remote_root, timeout=10, verify='/etc/ssl/certs/ca-bundle.crt')
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 289, in request
    history = [r for r in gen] if allow_redirects else []

This is a fresh install with all updates. Rebooted one time to make sure I did it correctly.

Thanks!

Flox · October 21, 2021, 11:37am

Curious… if a step was missed, it was on my end.

Just to be sure, the error you see is still the same? Could you paste the full output when you get the chance to see if it differs from before?

I did try a lot of things before getting to that hack on the requests calls; none of these were successful by themselves but maybe they were required for the requests alterations to be effective. I’ll need to setup another test instance from scratch to properly test this out, though, so I unfortunately won’t be able to try it that quickly…

phillxnet · October 21, 2021, 11:47am

@Flox Could this be down to a difference between our CentOS Stable (much newer) and Testing releases. It may be that there were some additional upstream patches, i.e. “… if allow_redirects …”.

@scrosler do you have a stable subscription in order to test this ?

Flox · October 21, 2021, 11:56am

The tests I’ve done were on a Testing channel install with Rockstor-3.9.1-16, but it’s a good point: Stable channel is much more recent than the lastest Testing in CentOS so this SSL certs issue might not even be one there… I have not tested it yet.

The tests I originally did were centered around update-ca-trust extract after placing a rockstor.pem created from our own certs in the correct location… I removed them and the requests hack described above still worked, hence my thought they weren’t doing anything but maybe they did help after all…

Hooverdan · October 21, 2021, 7:48pm

Not sure that this is a relevant data point for you, but upon checking I had the same SSL_CERT issue on my Rockstor version: 3.9.2-57

After the above: verifying it listed certificates, the sed commands, file validation and the rockstor restart using systemctl, it worked again

Flox · October 21, 2021, 8:11pm

That is very relevant and extremely helpful! Thank you so much for testing that.

I’m glad that worked for you! Now I’m really puzzled as to why it didn’t seem to work for @scrosler, though. The sed command seems to have worked for them as it seems right from the logs we can see… hopefully the additional information requested will shed some light into this.

Thanks again for testing that, @Hooverdan, that was really really helpful!

scrosler · October 21, 2021, 11:03pm

I’ll try again right now. I tried before update and after update missing the update portion the first time. I will do a quick re-install and post back…

scrosler · October 22, 2021, 12:17am

I tried again. The result is the same as my first few attempts. The commands are pure copy and paste. The install is 100% fresh. I’ll add but more than likely not relevant, that this happens on VM and physical machine.

The edits are being made because the traceback shows that in the message so we know that part is good.

I did not deviate one character from the instructions.

Also this is NOT the same error that I was getting before the fix was attempted. I have that posted in another thread if you would like to contrast the two outputs that I am getting. I will

    Traceback (most recent call last):

File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon.py”, line 395, in _get_available
response = requests.get(remote_root, timeout=10, verify=’/etc/ssl/certs/ca-bundle.crt’)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 55, in get
return request(‘get’, url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 44, in request
return session.request(method=method, url=url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 289, in request
history = [r for r in gen] if allow_redirects else []
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 133, in resolve_redirects
proxies=proxies
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 279, in request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 374, in send
r = adapter.send(request, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py”, line 213, in send
raise SSLError(e)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

Flox · October 22, 2021, 11:56am

Curious… The one thing I can think of at the moment is related to the following:

Did you activate an update channel? I’m not sure on what version of Rockstor you are for the moment and I just want to make sure as the iso you are using installs 3.9.1-0, and activating the Testing channel, for instance, will bring you to 3.9.1-16. It’s still a very old version but that’s the version I’ve tested this workaround so it’s still worth making sure you’re at least on that version. Running yum info rockstor will give you that information reliably.
Speaking of Rockstor versions, I believe you were going to try building your Rockstor 4 installer; let us know how it went and if we can help with it. Rockstor 4 is the version to use at the moment, really, especially if we seem to have a hard time getting Rockstor 3 to work well for you.

I though it was the same, indeed… I was comparing the one above to the traceback in that post. I seem to be forgetting a post, unfortunately…

Let us know if updating to Rockstor 3.9.1-16 makes a difference for you, it’s far fetched but it’s a simple thing to check so worth the try. If that still does not help, then we’ll try something else.

Cheers, and sorry I can’t seem to really grasp why this hack does not work here.

scrosler · October 23, 2021, 3:38am

There are no worries here my friend. I’ll go ahead and rebuild one more time. I know that in my haste I probably made errors. Brb…

scrosler · October 23, 2021, 3:39am

And by “not the same error” I mean the verbiage changed indicating that the edits provided did indeed stick. I may have worded it more confusing that it needed to be.

ArmyHill01 · October 24, 2021, 5:37pm

I can confirm using the last Rockstor 3 iso w/ all latest updates applied and following the original workarounds steps do not seem to be working. I continue to get:

Houston, we've had a problem.
Error while processing remote metastore at http://rockstor.com/rockons/root.json. Lower level exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) 


            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 395, in _get_available
    response = requests.get(remote_root, timeout=10, verify='/etc/ssl/certs/ca-bundle.crt')
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 289, in request
    history = [r for r in gen] if allow_redirects else []
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 133, in resolve_redirects
    proxies=proxies
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 213, in send
    raise SSLError(e)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

Flox · October 24, 2021, 7:20pm

Hi @ArmyHill01,

Thanks a lot for the report… it’s curious it works for some but not others.
I’d like to make sure of the same as for @scrosler:

Could you thus confirm the Rockstor version and try to activate the Testing channel and try again? I have little hope that it would help that particular issue, but it’s really worth a try and thus I’d like to rule this out.
I’ll try to find some time to test further when I can.

scrosler · October 24, 2021, 10:11pm

Yes! I sure will. Had PC problems this weekend but I am back at it

scrosler · October 24, 2021, 10:13pm

@Flox, Ahhh, will try with testing channel and report back!

Sorry, Main PC was down most all weekend

Flox · October 24, 2021, 10:36pm

No worries, @scrosler, I hope your problems with your main PC are now resolved!

ArmyHill01 · October 25, 2021, 12:03am

Sorry, forgot to mention, yes I’m on testing and updated to 3.9.1-16

scrosler · October 25, 2021, 5:49pm

This is a fresh install, update to testing 3.9.1-16. Yum update confirms no packages when I drop down to command line. Edited the lines of code as noted and still receive the error about the certificate.

If you have access to Hyper-V I can zip the hard drive up for you. Or, I have this machine in a DMZ if you want to access it directly for experimentation / confirmation. Otherwise if you have any other suggestions let me know. I would happy to try them!

        Traceback (most recent call last):

File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon.py”, line 395, in _get_available
response = requests.get(remote_root, timeout=10, verify=’/etc/ssl/certs/ca-bundle.crt’)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 55, in get
return request(‘get’, url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 44, in request
return session.request(method=method, url=url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 289, in request
history = [r for r in gen] if allow_redirects else []
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 133, in resolve_redirects
proxies=proxies
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 279, in request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 374, in send
r = adapter.send(request, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py”, line 213, in send
raise SSLError(e)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

scrosler · October 25, 2021, 5:52pm

My exact steps…
Log in / create user
Activate testing channel and select update
SSH in and run “yum update” after GUI completes
Edit python script pointing to the local certificate.
Create Rock-On data share
Activate Rock-On
Press Update
Fails