Built on openSUSE testing channel live (early-adopters/developers only)

Announcements
56

Couple of new observations / issues

  1. Cannot download logs - tried both ways (Download Logs Archive & Through the Logs Reader) - Error message is “Couldn’t download - No file”
  2. The Rockstor log is full and is filled with this message "
[14/Jan/2021 09:20:52] ERROR [system.osi:196] non-zero code(1) returned by command: ['/usr/sbin/btrfs', 'qgroup', 'show', '/mnt2/Data/PlexBackUp']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']

Not sure if something needs to be set-up. If these are issues - I can move this to the issue tracker - but just wanted to check before doing that.
Thanks

1 Like
57

Hi again, @dvgeek, and thanks a lot for the feedback :wink:

I tried to replicate it on Rockstor-4.0.5 and both ways worked for me… I’m unfortunately not familiar with this area of Rocsktor so I’ll have to dig a bit to have some idea of what could be happening. Do you see the link created when clicking on “Download Logs Archive”? If so, does it look like https://<rockstor-ip>/logs/requested_logs.tgz?

That is the normal and expected behavior when quotas are disabled. We actually had a discussion about improving this as it can be misleading, indeed. The forum thread is below:

… with the corresponding Github issue for reference:
https://github.com/rockstor/rockstor-core/issues/2236#issue-769611226

Sorry I can’t be of more help with the logs issue at the moment…

1 Like
58

Yup - I can see the link - and the browser tries to download the file - but errors out with the message “Couldn’t download - No file”. I could copy and paste the download link - and then I get a 404 Not found.
Tried with a couple of different browsers too - didn’t work.

Too bad about the logs - any way it can turned off or change level of detail ?

59

@dvgeek Hello there.
Re:

We currently only have a command line script to enable or disable debug level of logging and by default that’s disabled. As per the referenced discussion we are due to move the quotas disabled to a debug level rather than it’s current error rating.

The code for where this is defined is indicated in the ‘error’ message you receive so in this case the following:

ERROR [system.osi:196] …

refers to system/osi.py but that is a general process wrapper for all run_command() operations:

As can be seen there is a “log” conditional. So you could for the time being find the ‘problem’ run_command entry and adjust it’s log parameter. There are 3 possible candidates in

You could just change the:

log=True

to

log=False

and that ‘condition’ will no longer be logged.

Alternatively you could just enable quotas on the indicated “Data” pool.

Hope that helps.

2 Likes
60

Took the easy way out - enabled quotas on the pool.
Thanks.

Still not able to download logs though…

61

@dvgeek Thanks for the updated, yes enabling quotas is definitely the easier way around this. And the cost, performance wise, is reducing all the time so that’s good.

Given this is a persistent problem for you, could you start a new forum thread detailing the reproducer for this issue. @Flox has indicated that it works for him in 4.0.5-0 so maybe there is something about your setup or some config change you’ve made that is triggering this. If in a new thread we can focus on just this and once a reproducer is at hand we can open a new GitHub issue with a means to prove any fix proposed. It is a strange one as we haven’t had any other reports to date in that area so would be good to get to the cause of this. Remember to detail the browser you are experiencing this issue with and of course the version of Rockstor you are using.

Hope that helps.

3 Likes
62

Hello!

Starting to get my feet wet with OpenSUSE. I followed all of the instructions for a tumbleweed install. I was able to start the install of Rockstor, but I got one error:

Retrieving: rockstor-4.0.1-0.x86_64.rpm ..............................................................[done (4.5 MiB/s)]
rockstor-4.0.1-0.x86_64.rpm:
    Header V4 RSA/SHA256 Signature, key ID 5f043187: NOKEY
    V4 RSA/SHA256 Signature, key ID 5f043187: NOKEY

Looking for gpg key ID 5F043187 in cache /var/cache/zypp/pubkeys.
Repository Rockstor-Testing does not define additional 'gpgkey=' URLs.
rockstor-4.0.1-0.x86_64 (Rockstor-Testing): Signature verification failed [4-Signatures public key is not available]
Abort, retry, ignore? [a/r/i] (a):

I did an Ignore, and it seems to finish up just fine. After the install, Rockstor failed to start (the service failed). Looking, it was complaining about a missing libzmq.so. I did a quick install of that:

zypper in libzmq5

After doing that, the service started up just fine. I then rebooted, but after the reboot SSH failed to start. In the log I saw:

Feb 08 22:27:02 rocknas sshd[2083]: /etc/ssh/sshd_config line 131: Subsystem 'sftp' already defined.

I opened the config file, and sure enough sftp was defined twice:

# override default of no subsystems
Subsystem      sftp    /usr/libexec/ssh/sftp-server

# This enables accepting locale enviroment variables LC_* LANG, see sshd_config(5).
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server
###BEGIN: Rockstor SFTP CONFIG. DO NOT EDIT BELOW THIS LINE###
Subsystem       sftp    internal-sftp
AllowUsers root

So it appears that when Rockstor did isn’t install, it added an SFTP line to the sshd_config file, but didn’t first check if such a line was already there. When I commented out the first instance of sftp, it started up just fine.

I’ll keep poking around, and let you know if I come across anything else. Is this thread the best place to report?

1 Like
63

Hi @kupan787,

Thanks a lot for your excellent report, and nice finding! I believe we can be sure of what’s happening here thanks to that.

What you found is indeed related to how we detect and ensure proper config of sshd and apparently the default config shipped in Tumbleweed has now changed.

You are absolutely correct…

Not quite… we actually do check for it, but we currently check for the default config line shipped in Leap (and Tumbleweed at the time of that particular work). As this has now changed, we fail to properly detect it and inactivate it. You can see more details in the corresponding pull request linked below:

As you can see there, the default config in Leap is…

Subsystem   sftp    /usr/lib/ssh/sftp-server

… but thanks to your findings, we now know that it will likely soon ship as:

Subsystem      sftp    /usr/libexec/ssh/sftp-server

This small change in path is enough for us not to detect it and thus failing to inactivate it. We should thus make this part more robust and account for what is seemingly to come in Leap in future releases. Would you mind opening an issue in our Github repo?

Yep, that’s what I would have done to fix it as well :wink:

Thanks a lot for your findings and sharing them, that’s exactly the kind of things we need to keep improving the project and keeping up with upstream changes!

2 Likes
64

@kupan787 Glad to see you experimenting with the ‘Built on openSUSE’ side here.
Re:

Unfortunately we’ve had to stop the Tumbleweed support as some of the required libraries, namely some Python2 ones, are no longer available. Which puts us between a rock and a hard place. Both me and @Flox were quite exited for for Rockstor to be working on Tumbleweed for a while but their dropping of the required libraries that serve our technical dept position was an show stopper really. Hence no rpm’s available past the one your found there. We just couldn’t build it any more. There is however a light at the end of the tunnel on this one. But it’s along tunnel. We have to complete our migration to newer Django, then newer Python. The again back to update our Django. The following 2 issues cover the first 2 parts of this require endeavour before we can again work on Tumbleweed:

and

Both have received some attention and I’m currently working on the latter. But no promises just yet unfortunately as I’m still at the viability assessment phase.

Hence us concentrating only on the Leap 15* variant now. This is a shame as the Tumbleweed builds helped with seeing what was comming down the line on occasions, but all in good time and in the end I hope to have our dependencies in a healthy enough state to server both the stable and edge releases of openSUSE. But with the missing Python 2 libs and the effort to track both OS variants we had to drop the Tumbleweed.

I see @Flox has now addressed your more specific issues re sftp but you are likely to find many more as we are just not testing on Tumbleweed any more. But as stated I really hope to change that once we have all our dependencies in a shiny new state. Then it should again be an all round positive effort to at least try and track the super fast moving Tumbleweed openSUSE variant. And as @Flox points out in the above reply, it’s a useful endeavour if we could only get our dependencies lined up and new enough.

I’m a little surprised actually that you were able to install this rpm. Is this a prior existing Tumbleweed install, as last time I tried I could only build/install our own rpms on Tumbleweeds that had the required, now removed upstream, dependencies already installed. That was the main reason why I stopped building the rpms in fact. Plus without those same rpms available and maintained upstream we can’t offer a Tumbleweed target for he installer either.

Hope that helps. And also note that the Leap variants get pretty aggressive back-ports on the kernel and btrfs so there’s that. Also I hope to start building Leap 15.3 rpms soon, but again my plate is a little full currently so no promises on that either.

1 Like
65

So here is what I did, right or wrong:

# Get latest Tumbleweed ISO
# Install openSUSE Tumbleweed, with following options:
   # Activate Online Repos
   # Select Server (non transactional)
   # Default partitioning
   # Select Timezone
   # Skip user creation
   # Update GRUB to disable IPv6
	      ipv6.disable=1
   # Change Software
	   - Enable "Base Development"
	   - Enable "Python 3 Development"
	   - Disable AppArmor
   # Switch to/using NetworkManager

   # Let installer complete, and reboot after install

# After reboot, log in via SSH as root

# Apply all upstream updates
zypper dup --no-recommends

# Add ‘shells’ OBS repo
zypper --non-interactive addrepo --refresh -p105 http://download.opensuse.org/repositories/shells/openSUSE_Tumbleweed/ shells
zypper --non-interactive --gpg-auto-import-keys refresh

# Stop and Disable AppArmor
systemctl stop apparmor
systemctl disable apparmor

reboot

# Add missing packages

zypper in python-devel

# Grab RPMs that aren't on Tumbleweed
mkdir src
cd src
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/noarch/python2-distro-1.2.0-1.18.noarch.rpm
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/noarch/python2-chardet-3.0.4-3.23.noarch.rpm
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/noarch/python2-certifi-2018.1.18-1.18.noarch.rpm
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/noarch/python2-idna-2.6-1.20.noarch.rpm
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/noarch/python2-py-1.8.1-5.3.5.noarch.rpm
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/noarch/python2-six-1.14.0-1.19.noarch.rpm
wget https://download.opensuse.org/repositories/openSUSE:/Leap:/15.2:/Update/standard/noarch/python-ipaddress-1.0.18-lp152.4.3.1.noarch.rpm
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.3/repo/oss/noarch/python2-urllib3-1.24-9.10.1.noarch.rpm
wget https://rpmfind.net/linux/opensuse/distribution/leap/15.2/repo/oss/noarch/python2-requests-2.20.1-lp152.2.1.noarch.rpm

rpm -i python2-distro-1.2.0-1.18.noarch.rpm
rpm -i python2-chardet-3.0.4-3.23.noarch.rpm
rpm -i python2-certifi-2018.1.18-1.18.noarch.rpm 
rpm -i python2-idna-2.6-1.20.noarch.rpm 
rpm -i python2-py-1.8.1-5.3.5.noarch.rpm 
rpm -i python2-six-1.14.0-1.19.noarch.rpm
rpm -i python-ipaddress-1.0.18-lp152.4.3.1.noarch.rpm
rpm -i python2-urllib3-1.24-9.10.1.noarch.rpm 
rpm -i python2-requests-2.20.1-lp152.2.1.noarch.rpm

# Add this package that rockstor needs, but doesn't install
zypper in libzmq5

# Add Rockstor Repo

zypper addrepo -f http://updates.rockstor.com:8999/rockstor-testing/tumbleweed/ Rockstor-Testing
zypper refresh
zypper in --no-recommends rockstor

# Comment out SFTP line in /etc/ssh/sshd_config

reboot
1 Like
66

Here you go!

3 Likes
67

@kupan787 Thanks for sharing your work-around there.
Re:

That’s the bit I was wondering about. And frankly the bit we can’t do/recommend. Nice you found them though. Well done. I’ve not invested the time myself as it seemed better to use the time to address the core problem of us depending on these non standard dependencies in the first place. Also the ones you are using are build for Leap 15.3/15.2 and not Tumbleweed. And I believe their base compiler versions are now different. So there may be hidden surprises to come on that front. These will likely only differ further as time goes on. But at least bodes well for our 15.3 builds anyway. And I’m thinking the support life-cycle of 15.3 may be the only time we have left to move over to what comes standard in Tumbleweed anyway.

The issue you first posted regarding the signature for the rockstor repo:

can be avoided I believe by the instructions here:

Before you add the Rockstor repo as you have indicated. That way your systems rpm db already has a copy of the required key.

It’s good to see it’s at least possible still for us to run on Tumbleweed. Even if the missing-from-distro rpms required are from the Leap variants. And given these extra distro requirements I’m definitely leaning to still having tumbleweed unsupported and working towards Leap 15.3 instead. And in the mid term working on getting our old dependencies sorted.

In my current spin-off issue for example:
https://github.com/rockstor/rockstor-core/issues/2276
we are currently using a library that hasn’t development for 9 years. This has to be our focus, as otherwise we may not even work on Leap !!.

So in short we are likely in the near term to still only release rpms for Leap and they are currently at version 4.0.5

with 4.0.6 around the corner hopefully.

But your efforts here have definitely born fruit so I’m eager to re-gain Tumbleweed compatibility. But all in good time.

Hope that helps.

3 Likes
68

Totally understand, and makes sense.

If I continue poking around in tumbleweed, does it help for me to report findings anywhere? Some might be because I’ve kind of hacked together a solution, and so might not be “real” issues. I don’t want to clog up things with goose chases.

1 Like
69

@kupan787 Re:

Definitely, and already has with the presumaly upstream pending ssh change you’ve reported already.

That’s the worry really. Plus, since we arn’t releasing the rpms for Tumblweed, you have to build from source to test the result.

But on the other hand, you could try removing the old Tumbleweed repo and adding the Leap 15.2 repo, I can let you know here once we have a Leap 15.3 repo. That way you at least are testing our latest code as 4.0.1 was actually release on the 2nd August 2020:

So that would definitely make your reports more relevant. It doesn’t adhere to the ideal of running rpms built on the actual OS they target, which all our release rpms are, i.e. all Leap 15.2 rpms are build on 15.2 itself. But it does mean you are running current code which will be better all around, plus you are using Leap 15.2/15.3 packages that you shoe horned in anyway. If you go this way you should end up with 4.0.5 as of now and it will be interesting to hear if this works. Plus if/when we fix anything you report you can see if it’s successfully instantiated in a ‘real’ rpm. Otherwise you are stuck with source builds which don’t update themselves so no end-to-end testing of what we will actually release.

Thanks a tone for your continued involved in this way. It’s an invaluable and central part of how we got to where we are now and much appreciated.

See how you get on with running the Leap 15.2 repo instead. Though you would be advised to uninstall the existing rpm, then remove the old Tumbleweed repo and the /opt/rockstor dir then add the new repo and take it from there. If all goes well you should get 4.0.5. Apologies for not thinking of this before but given your approach in shoe horning the 15.2 lib into Tumbleweed anyway it may make for an interesting experiment. We just have to keep an eye on the ‘setup’ not being standard as it were. And hopefully we can tell, such as @Flox has done already, the actual cause anyway. There are also likely kernel issues, but again we can hopefully narrow down situations there also. These will mainly be down to how the btrfs commands change in their output. But again it could be interesting to see these coming down the line.

Thanks again and do keep us posted on this experiment. And if all falls to pieces then you can always stand up a Leap 15.2 base instead. Or better still build the actual installer as the resulting system is likely very much smaller than what you likely have there anyway which is always nice.

2 Likes
70

Hi, Sorry don’t know if I should be putting this under this post or a new one at the “root”. Since updating to 4.0.6 I’ve lost the Disk Activity chart. I’m running fully updated openSUSE Leap under Vmware ESXi 6.0.

Thanks
Chris

71

Since updating to 4.0.6 I’ve lost the Disk Activity chart

Hmm don’t have that problem here. Did you maybe accidentally remove it at some point?
There is a list of checkboxes on the left to disable and enable certain stats:

image

Is disk activity activated there?

1 Like
72

Hi Marenz, It’s enabled and the “block” is in the dashboard there’s just no graphic.

image
image694×471 23 KB

Thanks
Chris

1 Like
73

In that case, can you do right-click on the webpage and click “inspect”, change to the network tab, reload the dashboard and look for anything that times out (usually shown red). That might give some hint as to what part is not reporting what it should?

1 Like
74

Hi Marenz, Below is the one obvious bad request from the network inspection.

Thanks
Chris

   {

  {
    "pageref": "page_1",
    "startedDateTime": "2021-03-22T13:24:45.654+13:00",
    "request": {
      "bodySize": 0,
      "method": "GET",
      "url": "https://10.24.x.x/socket.io/?EIO=3&transport=polling&t=NXNKTjK&sid=d7c88176be7848ed8ae0132a76b1a45d",
      "httpVersion": "HTTP/1.1",
      "headers": [
        {
          "name": "Host",
          "value": "10.24.x.x"
        },
        {
          "name": "User-Agent",
          "value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0"
        },
        {
          "name": "Accept",
          "value": "*/*"
        },
        {
          "name": "Accept-Language",
          "value": "en-GB,en;q=0.5"
        },
        {
          "name": "Accept-Encoding",
          "value": "gzip, deflate, br"
        },
        {
          "name": "Connection",
          "value": "keep-alive"
        },
        {
          "name": "Referer",
          "value": "https://10.24.x.x/home?cb-table_length=15"
        },
        {
          "name": "Cookie",
          "value": "io=78f09cba5f7b403781628dc85c60b3a2; io=6aff95eba56c4ee5855dd97e0b8c6741; csrftoken=P0qnDAIjYLb5Kx7GQYFljNkjrcYNJCCP; redirect=1; testing=1; sid=daf4de89994bd2469a3085a920c9bcd0; sessionid=6i9nn0xog9bvdpu2ekthkgv3nrgww455"
        }
      ],
      "cookies": [
        {
          "name": "io",
          "value": "78f09cba5f7b403781628dc85c60b3a2"
        },
        {
          "name": "io",
          "value": "6aff95eba56c4ee5855dd97e0b8c6741"
        },
        {
          "name": "csrftoken",
          "value": "P0qnDAIjYLb5Kx7GQYFljNkjrcYNJCCP"
        },
        {
          "name": "redirect",
          "value": "1"
        },
        {
          "name": "testing",
          "value": "1"
        },
        {
          "name": "sid",
          "value": "daf4de89994bd2469a3085a920c9bcd0"
        },
        {
          "name": "sessionid",
          "value": "6i9nn0xog9bvdpu2ekthkgv3nrgww455"
        }
      ],
      "queryString": [
        {
          "name": "EIO",
          "value": "3"
        },
        {
          "name": "transport",
          "value": "polling"
        },
        {
          "name": "t",
          "value": "NXNKTjK"
        },
        {
          "name": "sid",
          "value": "d7c88176be7848ed8ae0132a76b1a45d"
        }
      ],
      "headersSize": 603
    },
    "response": {
      "status": 400,
      "statusText": "BAD REQUEST",
      "httpVersion": "HTTP/1.1",
      "headers": [
        {
          "name": "Server",
          "value": "nginx/1.16.1"
        },
        {
          "name": "Date",
          "value": "Mon, 22 Mar 2021 00:25:46 GMT"
        },
        {
          "name": "Content-Type",
          "value": "text/plain"
        },
        {
          "name": "Content-Length",
          "value": "11"
        },
        {
          "name": "Connection",
          "value": "keep-alive"
        },
        {
          "name": "Keep-Alive",
          "value": "timeout=20"
        },
        {
          "name": "Access-Control-Allow-Origin",
          "value": "*"
        },
        {
          "name": "Access-Control-Allow-Credentials",
          "value": "true"
        }
      ],
      "cookies": [],
      "content": {
        "mimeType": "text/plain",
        "size": 11,
        "text": "Bad Request"
      },
      "redirectURL": "",
      "headersSize": 253,
      "bodySize": 264
    },
    "cache": {},
    "timings": {
      "blocked": 0,
      "dns": 0,
      "connect": 0,
      "ssl": 0,
      "send": 0,
      "wait": 60005,
      "receive": 0
    },
    "time": 60005,
    "_securityState": "secure",
    "serverIPAddress": "10.24.x.x",
    "connection": "443"
  }
]

}
}

75

Hi, I’ve updated and closed my issue on GitHub. The problem seemed to be caused by snappy/snapd and missing or incorrect Python2 pacakages.

Thanks
Chris

3 Likes