I recently added two new drives to my server a couple of weeks ago and had to re-generate my nginx configuration file for some reason.
Now when I try to update rock-ons I get an error after a minute and a half or so
Houston, we've had a problem.
Unknown internal error doing a POST to /api/rockons/update
If I look in the network log in my browser it’s getting a 502 bad gateway response.
@sorokya Welcome to the Rockstor community.
Yes this is a strange one and we have seen it before. It is likely that your issues with:
are related to this ‘re-generate’. It may be that you are missing something the Rockstor depends upon.
Take a look at the following issue where there are instructions on re-generating our specific nginx configuration using the template included within the /opt/rockstor directory:
https://github.com/rockstor/rockstor-core/issues/1833
The second comment there has a one line command string that, assuming you have an empty nginx, will re-create it from the template.
@petermc @kysdaddy and @Zonk in the following forum thread has confirmed this to work (repeated in forum thread):
Hope that helps. You didn’t mention re-generating your nginx via this route so I though it best to start with that as a possible cause.
Hey, yeah I should have mentioned I did use that method to regenerate it. Here is my /opt/rockstor/etc/nginx/nginx.conf file
the only change I made to try to fix it was changing the timeouts to 3000 
daemon off;
worker_processes 2;
events {
worker_connections 1024;
use epoll;
}
http {
include /opt/rockstor/etc/nginx/mime.types;
default_type application/octet-stream;
log_format main
'$remote_addr - $remote_user [$time_local] '
'"$request" $status $bytes_sent '
'"$http_referer" "$http_user_agent" '
'"$gzip_ratio"';
client_header_timeout 10m;
client_body_timeout 10m;
send_timeout 10m;
connection_pool_size 256;
client_header_buffer_size 1k;
large_client_header_buffers 4 8k;
request_pool_size 4k;
gzip on;
gzip_min_length 1100;
gzip_buffers 4 8k;
gzip_types text/plain;
output_buffers 1 32k;
postpone_output 1460;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 75 20;
ignore_invalid_headers on;
index index.html;
server {
listen 443 default_server;
server_name "~^(?<myhost>.+)$";
ssl on;
ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
ssl_certificate /opt/rockstor/certs/rockstor.cert;
ssl_certificate_key /opt/rockstor/certs/rockstor.key;
location /site_media {
root /media/; # Notice this is the /media folder that we create above
}
location ~* ^.+\.(zip|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|mov) {
access_log off;
expires 30d;
}
location /static {
root /opt/rockstor/;
}
location /logs {
root /opt/rockstor/src/rockstor/;
}
location / {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 3000;
proxy_read_timeout 3000;
proxy_pass http://127.0.0.1:8000/;
}
location /socket.io {
proxy_pass http://127.0.0.1:8001;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
location /shell/ {
valid_referers server_names;
if ($invalid_referer) { return 404; }
proxy_pass http://127.0.0.1:4200;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}
@sorokya
OK, thanks for the update, then you have a different problem. So at least we can rule that one out. I assume you have rebooted since then?
I’ve rebooted since regenerating my config. But only restarted the rockstor service since changing the timeouts.
I’ll do a reboot and post an update
Alright, I tried another update after rebooting the entire server and still getting the same response/error
Screenshot of error
Request/Response from firefox
{
"log": {
"version": "1.1",
"creator": {
"name": "Firefox",
"version": "65.0.2"
},
"browser": {
"name": "Firefox",
"version": "65.0.2"
},
"pages": [
{
"startedDateTime": "2019-03-09T07:53:47.968-05:00",
"id": "page_1",
"title": "Rockstor",
"pageTimings": {
"onContentLoad": -203230,
"onLoad": -203190
}
}
],
"entries": [
{
"pageref": "page_1",
"startedDateTime": "2019-03-09T07:53:47.968-05:00",
"request": {
"bodySize": 0,
"method": "POST",
"url": "https://192.168.11.10/api/rockons/update",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Host",
"value": "192.168.11.10"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
},
{
"name": "Accept",
"value": "application/json, text/javascript, */*; q=0.01"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.5"
},
{
"name": "Accept-Encoding",
"value": "gzip, deflate, br"
},
{
"name": "Referer",
"value": "https://192.168.11.10/home"
},
{
"name": "X-CSRFToken",
"value": "KQOC0sQbuEkYTDcZ6obd5sdDEjMvTCoJ"
},
{
"name": "X-Requested-With",
"value": "XMLHttpRequest"
},
{
"name": "DNT",
"value": "1"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Cookie",
"value": "csrftoken=KQOC0sQbuEkYTDcZ6obd5sdDEjMvTCoJ; sessionid=gv7jjbw7zvs82129knl9dhoq2xztbpnx"
},
{
"name": "Pragma",
"value": "no-cache"
},
{
"name": "Cache-Control",
"value": "no-cache"
},
{
"name": "Content-Length",
"value": "0"
}
],
"cookies": [
{
"name": "csrftoken",
"value": "KQOC0sQbuEkYTDcZ6obd5sdDEjMvTCoJ"
},
{
"name": "sessionid",
"value": "gv7jjbw7zvs82129knl9dhoq2xztbpnx"
}
],
"queryString": [],
"headersSize": 573
},
"response": {
"status": 502,
"statusText": "Bad Gateway",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Server",
"value": "nginx/1.12.2"
},
{
"name": "Date",
"value": "Sat, 09 Mar 2019 12:55:49 GMT"
},
{
"name": "Content-Type",
"value": "text/html"
},
{
"name": "Content-Length",
"value": "173"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Keep-Alive",
"value": "timeout=20"
}
],
"cookies": [],
"content": {
"mimeType": "text/html",
"size": 173,
"text": "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.12.2</center>\r\n</body>\r\n</html>\r\n"
},
"redirectURL": "",
"headersSize": 181,
"bodySize": 354
},
"cache": {},
"timings": {
"blocked": 0,
"dns": 0,
"connect": 0,
"ssl": 0,
"send": 0,
"wait": 121392,
"receive": 0
},
"time": 121392,
"_securityState": "secure",
"serverIPAddress": "192.168.11.10",
"connection": "443"
}
]
}
}
I also see these errors in the logs after rebooting about not being able to connect to rockstor.com. I tried connecting to the site directly on the server over SSH using lynx and it connects fine.
[09/Mar/2019 07:51:01] ERROR [storageadmin.util:40] Error while processing Rock-on profile at http://rockstor.com/rockons/bitcoind.json. Lower level exception: HTTPConnectionPool(host='rockstor.com', port=80): Max retries exceeded with url: /rockons/bitcoind.json (Caused by : [Errno -2] Name or service not known)
[09/Mar/2019 07:51:01] ERROR [storageadmin.util:44] exception: HTTPConnectionPool(host='rockstor.com', port=80): Max retries exceeded with url: /rockons/bitcoind.json (Caused by : [Errno -2] Name or service not known)
Traceback (most recent call last):
File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
yield
File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 406, in _get_available
cur_res = requests.get(cur_meta_url, timeout=10)
File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 279, in request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/sessions.py", line 374, in send
r = adapter.send(request, **kwargs)
File "/opt/rockstor/eggs/requests-1.1.0-py2.7.egg/requests/adapters.py", line 209, in send
raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='rockstor.com', port=80): Max retries exceeded with url: /rockons/bitcoind.json (Caused by : [Errno -2] Name or service not known)
@sorokya So this looks an issue with you getting to the Rock-ons host webpage, ie:
From a machine on the same network as your Rockstor instance, can you get, via a web browser (ie port 80) to the following site:
http://rockstor.com/rockons/root.json
It’s what Rockstor is initially requesting to get a list of available Rock-on definition files, it then, in turn, requests each of the files listed in that root.json. I.e. from your log error it’s failing on the following json:
http://rockstor.com/rockons/bitcoind.json
Both work fine for me from the UK, ie I see the required contents of the requested files in the browser.
Let us know if those urls work ok on the same network as your Rockstor machine. Just trying to narrow this down. We have had issues of ‘overload’ on that server and we are in the process of refactoring our backend stuff, I’m working on that now as it goes, but it will be a little while longer before we are able to do the planned migration of our backend services. It may be that we have some kind of rate limit that is kicking in and spoiling your connection.
See how that goes first. You could also double check via a wget of the root and other files on the Rockstor machine itself:
wget http://rockstor.com/rockons/gogs.json
--2019-03-09 13:30:33-- http://rockstor.com/rockons/gogs.json
Resolving rockstor.com (rockstor.com)... 209.204.150.37
Connecting to rockstor.com (rockstor.com)|209.204.150.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2353 (2.3K) [text/plain]
Saving to: ‘gogs.json’
100%[=========================================================================================================================>] 2,353 --.-K/s in 0.001s
2019-03-09 13:30:34 (4.46 MB/s) - ‘gogs.json’ saved [2353/2353]
wget http://rockstor.com/rockons/bitcoind.json
--2019-03-09 13:32:34-- http://rockstor.com/rockons/bitcoind.json
Resolving rockstor.com (rockstor.com)... 209.204.150.37
Connecting to rockstor.com (rockstor.com)|209.204.150.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1288 (1.3K) [text/plain]
Saving to: ‘bitcoind.json’
100%[=========================================================================================================================>] 1,288 --.-K/s in 0s
2019-03-09 13:32:34 (298 MB/s) - ‘bitcoind.json’ saved [1288/1288]
Good ideas.
Here’s the root.json file open over port 80 on my desktop on the same network
Here’s the wget tests from the rockstor machine itself
[root@rockstor test]# wget http://rockstor.com/rockons/root.json
--2019-03-09 08:39:05-- http://rockstor.com/rockons/root.json
Resolving rockstor.com (rockstor.com)... 209.204.150.37
Connecting to rockstor.com (rockstor.com)|209.204.150.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2042 (2.0K) [text/plain]
Saving to: ‘root.json’
100%[==================================================================================================================================================>] 2,042 --.-K/s in 0s
2019-03-09 08:39:05 (186 MB/s) - ‘root.json’ saved [2042/2042]
[root@rockstor test]# wget http://rockstor.com/rockons/gogs.json
--2019-03-09 08:39:10-- http://rockstor.com/rockons/gogs.json
Resolving rockstor.com (rockstor.com)... 209.204.150.37
Connecting to rockstor.com (rockstor.com)|209.204.150.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2353 (2.3K) [text/plain]
Saving to: ‘gogs.json’
100%[==================================================================================================================================================>] 2,353 --.-K/s in 0s
2019-03-09 08:39:10 (220 MB/s) - ‘gogs.json’ saved [2353/2353]
Okay, I think I just fixed it!
I looked in my /etc/resolve.conf file to see what nameserver the rockstor server was trying to use and it was 192.168.11.1 (my router)
I run a pi-hole DNS on the network at 192.168.11.16 and I tried pointing it directly to that and it now works!
I was having this same issue with Steam on another computer on my network. So it must be some weird DNS thing going on.
@sorokya Well that all looks dandy. Strange.
Sorted. Well done and thanks for reporting back. I was a little out of ideas actually.
Glad your up and running again. Nice to have a record of this in the forum now in case others run into the same ‘issue’. We also have the following pi-hole improvements lined up in our Rock-on offering of the same: