Can not create an SFTP share on openSUSE

freaktechnik · April 26, 2020, 2:47pm

When trying to create an SFTP share on openSUSE I got the following error:

[Errno 2] No such file or directory: '/lib64/libpopt.so.0'

            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/sftp.py", line 80, in post
    rsync_for_sftp(chroot_loc)
  File "/opt/rockstor/src/rockstor/system/ssh.py", line 127, in rsync_for_sftp
    ('%s/lib64' % chroot_loc))
  File "/usr/lib64/python2.7/shutil.py", line 133, in copy
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 96, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/lib64/libpopt.so.0'

zypper says libpopt0 is already installed. I can find it at /usr/lib64/libpopt.so.0, so the code is possibly looking for it in the wrong place?

phillxnet · April 26, 2020, 3:16pm

@freaktechnik Nice find.

Let us know if you make an progress in tracking this down any further.

I’d look to this bit first:

I.e. chroot_loc may be set wrong. Haven’t been there for ages. If you take a look at a few recent [openSUSE] related commits you can see how we do stuff contextually using distro. That may be all we need here. But again, I haven’t looked into this yet but if you fancy doing so then great.

Thanks again for a nice report and do post here any progress. Once we have the root cause you could create a GitHub issue to get this one sorted if you fancy:

Developers subsection of Contributing to Rockstor:
http://rockstor.com/docs/contribute.html#developers

Thanks again for the report and if you do get to creating a GitHub issue for this, once it’s clarified a tad, remember to link back to this forum post.

Flox · April 26, 2020, 3:47pm

@freaktechnik, @phillxnet, I was waiting for a computation to run so I had a few minutes to kill.

I thus looked and indeed, we are looking in the wrong place, it seems as we have the libs location hardcoded (for centOS, I assume) here:

github.com

rockstor/rockstor-core/blob/dc98a1cc59587cd9b5225da7cad0556cb84781e1/src/rockstor/system/ssh.py#L123-L127


libs = ('ld-linux-x86-64.so.2', 'libacl.so.1', 'libattr.so.1',
        'libc.so.6', 'libdl.so.2', 'libpopt.so.0', 'libtinfo.so.5',)
for l in libs:
    shutil.copy('/lib64/%s' % l,
                ('%s/lib64' % chroot_loc))

As per @freaktechnik findings, it seems like we should use /usr/lib64 instead of /lib64/ in line 126 (assuming all the other libs have the same location as well in openSUSE.

freaktechnik · April 26, 2020, 7:53pm

It appears only libpopt is in /usr/lib64 on openSUSE, the other ones are still native system libs.

Reported on GitHub as

github.com/rockstor/rockstor-core

libpopt0 not in /lib64 on openSUSE

opened 07:53PM - 26 Apr 20 UTC

closed 07:34PM - 18 May 20 UTC

freaktechnik

As reported in https://forum.rockstor.com/t/can-not-create-an-sftp-share-on-open…suse/7026 It appears libpopt0 is not in /lib64 and instead in /usr/lib64 on openSUSE. This leads to a failure when creating SFTP shares. The offending code is at https://github.com/rockstor/rockstor-core/blob/dc98a1cc59587cd9b5225da7cad0556cb84781e1/src/rockstor/system/ssh.py#L123-L127 When I checked on my system _only_ libpopt0 is in /usr/lib64, all the other libs are still in /lib64, as such a special case for libpopt should likely be introduced.

phillxnet · May 22, 2020, 1:38pm

@freaktechnik Hello again.

On a quick related developer note, how is @Flox’s fix for your reported:

github.com/rockstor/rockstor-core

libpopt0 not in /lib64 on openSUSE

opened 07:53PM - 26 Apr 20 UTC

closed 07:34PM - 18 May 20 UTC

freaktechnik

As reported in https://forum.rockstor.com/t/can-not-create-an-sftp-share-on-open…suse/7026 It appears libpopt0 is not in /lib64 and instead in /usr/lib64 on openSUSE. This leads to a failure when creating SFTP shares. The offending code is at https://github.com/rockstor/rockstor-core/blob/dc98a1cc59587cd9b5225da7cad0556cb84781e1/src/rockstor/system/ssh.py#L123-L127 When I checked on my system _only_ libpopt0 is in /usr/lib64, all the other libs are still in /lib64, as such a special case for libpopt should likely be introduced.

working out in 3.9.2-58, I’m assuming it’s working for you as tested in pr:

github.com/rockstor/rockstor-core

Update libraries needed for Rsync over SFTP

rockstor:master ← FroggyFlox:Issue2155_Rsync_SFTP_libs

opened 04:37PM - 10 May 20 UTC

FroggyFlox

+107 -56

Fixes #2155 @phillxnet, ready for review. As described by @freaktechnik in …#2155, we currently cannot export a share via SFTP in Rockstor built on openSUSE due to out-of-date shared libraries. This pull request thus proposes to update the shared libraries required by `/bin/bash` and `/usr/bin/rsync` in both Leap15.2 beta and Tumbleweed. The libraries were determined as described in https://github.com/rockstor/rockstor-core/issues/2155#issuecomment-626228322. ## Logic We retain the same logic as before: 1. the shared libraries required in each system are hardcoded in `rsync_for_sftp()` 2. we copy these over to the chroot directory defined in `django.settings` The only difference is that we now store these required libraries in a dictionary keyed by distribution id. We can thus simply refer to the `OS_DISTRO_ID` defined at build time to loop into the correct list of libs and copy each one over to the chroot. ## Caveats Especially true in Tumbleweed, this approach is limited by the fact that we have the libraries hard-coded (this is actually the reason why Issue #2155 occurred). We could dynamically fetch those by writing some sort of wrapper around `ldd` for instance, but that would also add another system call each time we need to refresh our chroot dir setup. @phillxnet , do you think that would be worth the cost? Parsing all these libs and their respective paths might end up cost a little in terms of resources, and thus I'm not sure it would be worth it. Thanks for letting me know if you think otherwise and I'll try to work on that. ## Functional test ### Rockstor built on Leap 15.2 In the webUI, export the share `sftp_test` via SFTP and verify the correct directory structure was created: *logs* ``` [10/May/2020 10:42:12] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/bin [10/May/2020 10:42:12] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/usr/bin [10/May/2020 10:42:12] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/lib64 [10/May/2020 10:42:12] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/usr/lib64 [10/May/2020 10:42:12] DEBUG [system.osi:157] Running command: /usr/sbin/usermod -s /bin/bash test [10/May/2020 10:42:12] DEBUG [system.osi:157] Running command: /usr/bin/systemctl reload sshd ``` and structure: ``` rockdev:~ # tree /mnt3/ /mnt3/ └── test ├── bin │ └── bash ├── lib64 │ ├── ld-linux-x86-64.so.2 │ ├── libacl.so.1 │ ├── libattr.so.1 │ ├── libc.so.6 │ ├── libdl.so.2 │ ├── libpthread.so.0 │ ├── libreadline.so.7 │ ├── libtinfo.so.6 │ └── libz.so.1 ├── sftp_test └── usr ├── bin │ └── rsync └── lib64 ├── libcrypto.so.1.1 ├── libpopt.so.0 └── libslp.so.1 7 directories, 14 files ``` ### Rockstor built on Tumbleweed (20200501) In the webUI, export the share `sftp_test` via SFTP and verify the correct directory structure was created: *logs* ``` [10/May/2020 10:58:19] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/bin [10/May/2020 10:58:19] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/usr/bin [10/May/2020 10:58:19] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/lib64 [10/May/2020 10:58:19] DEBUG [system.osi:157] Running command: /usr/bin/mkdir -p /mnt3/test/usr/lib64 [10/May/2020 10:58:19] DEBUG [system.osi:157] Running command: /usr/sbin/usermod -s /bin/bash test [10/May/2020 10:58:19] DEBUG [system.osi:157] Running command: /usr/bin/systemctl reload sshd ``` and structure: ``` rockdev:~ # tree /mnt3/ /mnt3/ └── test ├── bin │ └── bash ├── lib64 │ ├── ld-linux-x86-64.so.2 │ ├── libc.so.6 │ ├── libdl.so.2 │ ├── libpthread.so.0 │ ├── libreadline.so.8 │ ├── libtinfo.so.6 │ └── libz.so.1 ├── sftp_test └── usr ├── bin │ └── rsync └── lib64 ├── libacl.so.1 ├── libcrypto.so.1.1 ├── libpopt.so.0 └── libslp.so.1 7 directories, 13 files ``` ## Unit tests Unit tests outputs are identical to current master (not shown).

Cheers.

freaktechnik · May 22, 2020, 1:41pm

SFTP shares work fine now, thanks.

phillxnet · May 22, 2020, 2:03pm

@freaktechnik Thanks for the confirmation.

I can pop this in the beta release notes. Always good to have additional confirmation, just in case the developer and reviewer missed the same thing/things: @Flox and me respectively in this case.

Thanks for actively helping with Rockstor’s development by the way. I’m quite looking forward to having this whole CentOS to ‘Built on openSUSE’ move over and done with and we are very nearly there now.

Cheers.