Rockstor 3 SSL certificate error when refreshing Rock-ons: a temporary workaround

How-to
3

Curious… if a step was missed, it was on my end.

Just to be sure, the error you see is still the same? Could you paste the full output when you get the chance to see if it differs from before?

I did try a lot of things before getting to that hack on the requests calls; none of these were successful by themselves but maybe they were required for the requests alterations to be effective. I’ll need to setup another test instance from scratch to properly test this out, though, so I unfortunately won’t be able to try it that quickly…

1 Like
4

@Flox Could this be down to a difference between our CentOS Stable (much newer) and Testing releases. It may be that there were some additional upstream patches, i.e. “… if allow_redirects …”.

@scrosler do you have a stable subscription in order to test this ?

5

The tests I’ve done were on a Testing channel install with Rockstor-3.9.1-16, but it’s a good point: Stable channel is much more recent than the lastest Testing in CentOS so this SSL certs issue might not even be one there… I have not tested it yet.

The tests I originally did were centered around update-ca-trust extract after placing a rockstor.pem created from our own certs in the correct location… I removed them and the requests hack described above still worked, hence my thought they weren’t doing anything but maybe they did help after all…

2 Likes
6

Not sure that this is a relevant data point for you, but upon checking I had the same SSL_CERT issue on my Rockstor version: 3.9.2-57

After the above: verifying it listed certificates, the sed commands, file validation and the rockstor restart using systemctl, it worked again

2 Likes
7

That is very relevant and extremely helpful! Thank you so much for testing that.

I’m glad that worked for you! Now I’m really puzzled as to why it didn’t seem to work for @scrosler, though. The sed command seems to have worked for them as it seems right from the logs we can see… hopefully the additional information requested will shed some light into this.

Thanks again for testing that, @Hooverdan, that was really really helpful!

2 Likes
8

I’ll try again right now. I tried before update and after update missing the update portion the first time. I will do a quick re-install and post back…

9

I tried again. The result is the same as my first few attempts. The commands are pure copy and paste. The install is 100% fresh. I’ll add but more than likely not relevant, that this happens on VM and physical machine.

The edits are being made because the traceback shows that in the message so we know that part is good.

I did not deviate one character from the instructions.

Also this is NOT the same error that I was getting before the fix was attempted. I have that posted in another thread if you would like to contrast the two outputs that I am getting. I will

    Traceback (most recent call last):

File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon.py”, line 395, in _get_available
response = requests.get(remote_root, timeout=10, verify=’/etc/ssl/certs/ca-bundle.crt’)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 55, in get
return request(‘get’, url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 44, in request
return session.request(method=method, url=url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 289, in request
history = [r for r in gen] if allow_redirects else []
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 133, in resolve_redirects
proxies=proxies
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 279, in request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 374, in send
r = adapter.send(request, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py”, line 213, in send
raise SSLError(e)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

2 Likes
10

Curious… The one thing I can think of at the moment is related to the following:

Did you activate an update channel? I’m not sure on what version of Rockstor you are for the moment and I just want to make sure as the iso you are using installs 3.9.1-0, and activating the Testing channel, for instance, will bring you to 3.9.1-16. It’s still a very old version but that’s the version I’ve tested this workaround so it’s still worth making sure you’re at least on that version. Running yum info rockstor will give you that information reliably.
Speaking of Rockstor versions, I believe you were going to try building your Rockstor 4 installer; let us know how it went and if we can help with it. Rockstor 4 is the version to use at the moment, really, especially if we seem to have a hard time getting Rockstor 3 to work well for you.

I though it was the same, indeed… I was comparing the one above to the traceback in that post. I seem to be forgetting a post, unfortunately… :thinking:

Let us know if updating to Rockstor 3.9.1-16 makes a difference for you, it’s far fetched but it’s a simple thing to check so worth the try. If that still does not help, then we’ll try something else.

Cheers, and sorry I can’t seem to really grasp why this hack does not work here.

1 Like
11

There are no worries here my friend. I’ll go ahead and rebuild one more time. I know that in my haste I probably made errors. Brb…

12

And by “not the same error” I mean the verbiage changed indicating that the edits provided did indeed stick. I may have worded it more confusing that it needed to be.

1 Like
13

I can confirm using the last Rockstor 3 iso w/ all latest updates applied and following the original workarounds steps do not seem to be working. I continue to get:

Houston, we've had a problem.
Error while processing remote metastore at http://rockstor.com/rockons/root.json. Lower level exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) 


            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 395, in _get_available
    response = requests.get(remote_root, timeout=10, verify='/etc/ssl/certs/ca-bundle.crt')
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 289, in request
    history = [r for r in gen] if allow_redirects else []
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 133, in resolve_redirects
    proxies=proxies
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 213, in send
    raise SSLError(e)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
1 Like
14

Hi @ArmyHill01,

Thanks a lot for the report… it’s curious it works for some but not others.
I’d like to make sure of the same as for @scrosler:

Could you thus confirm the Rockstor version and try to activate the Testing channel and try again? I have little hope that it would help that particular issue, but it’s really worth a try and thus I’d like to rule this out.
I’ll try to find some time to test further when I can.

1 Like
15

Yes! I sure will. Had PC problems this weekend but I am back at it :wink:

16

@Flox, Ahhh, will try with testing channel and report back!

Sorry, Main PC was down most all weekend :frowning:

17

No worries, @scrosler, I hope your problems with your main PC are now resolved!

18

Sorry, forgot to mention, yes I’m on testing and updated to 3.9.1-16

image
image619×537 43.1 KB

1 Like
19

This is a fresh install, update to testing 3.9.1-16. Yum update confirms no packages when I drop down to command line. Edited the lines of code as noted and still receive the error about the certificate.

If you have access to Hyper-V I can zip the hard drive up for you. Or, I have this machine in a DMZ if you want to access it directly for experimentation / confirmation. Otherwise if you have any other suggestions let me know. I would happy to try them!

        Traceback (most recent call last):

File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon.py”, line 395, in _get_available
response = requests.get(remote_root, timeout=10, verify=’/etc/ssl/certs/ca-bundle.crt’)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 55, in get
return request(‘get’, url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py”, line 44, in request
return session.request(method=method, url=url, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 289, in request
history = [r for r in gen] if allow_redirects else []
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 133, in resolve_redirects
proxies=proxies
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 279, in request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py”, line 374, in send
r = adapter.send(request, **kwargs)
File “/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py”, line 213, in send
raise SSLError(e)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

20

My exact steps…
Log in / create user
Activate testing channel and select update
SSH in and run “yum update” after GUI completes
Edit python script pointing to the local certificate.
Create Rock-On data share
Activate Rock-On
Press Update
Fails

2 Likes
21

Had some time to review this today. Pulled rockon.py locally and started debugging. Code checked out fine so I knew it was a host issue.
After a few snapshots and restores later, just running update-ca-trust was enough to solve this for me AFTER using the sed replacements.

Hope this helps someone else!

4 Likes
22

Nice!
Thanks a lot, @ArmyHill01, for working that out and sharing the solution here, it’s really appreciated!

2 Likes