Samba Service broken after update -11

Bugs
5

Thanks again

[root@medienserver ~]# journalctl -u nmb.service
-- Logs begin at Sun 2017-09-17 10:27:17 CEST, end at Sun 2017-09-17 11:01:01 CEST. --
Sep 17 10:27:26 medienserver systemd[1]: Starting Samba NMB Daemon...
Sep 17 10:27:26 medienserver nmbd[2553]: /usr/sbin/nmbd: error while loading shared libraries: libwbclient.so.0: cannot open shared object file: No such file or directory
Sep 17 10:27:26 medienserver systemd[1]: nmb.service: main process exited, code=exited, status=127/n/a
Sep 17 10:27:26 medienserver systemd[1]: Failed to start Samba NMB Daemon.
Sep 17 10:27:26 medienserver systemd[1]: Unit nmb.service entered failed state.
Sep 17 10:27:26 medienserver systemd[1]: nmb.service failed.

[root@medienserver ~]# cat /etc/samba/smb.conf
[global]
    log level = 3
    map to guest = Bad User
    cups options = raw
    log file = /var/log/samba/log.%m
    printcap name = /dev/null
    load printers = no

####BEGIN: Rockstor SAMBA GLOBAL CUSTOM####
    workgroup = KOMPANIE
####END: Rockstor SAMBA GLOBAL CUSTOM####

####BEGIN: Rockstor SAMBA CONFIG####
[Bilder]
    root preexec = "/opt/rockstor/bin/mnt-share Bilder"
    root preexec close = yes
    comment = Samba-Export
    path = /mnt2/Bilder
    browseable = yes
    read only = no
    guest ok = yes
[Videos]
    root preexec = "/opt/rockstor/bin/mnt-share Videos"
    root preexec close = yes
    comment = Samba-Export
    path = /mnt2/Videos
    browseable = yes
    read only = no
    guest ok = yes
------snip------
####END: Rockstor SAMBA CONFIG####
6

Hi @Bert,

Looks like you’re missing libwbclient, although I can’t explain why.

Assuming your yum repos are sane, the following should resolve the issue:

yum install libwbclient

If that fails, post results of the following commands:

ls -la /usr/lib64/libwbclient.so.0
yum search libwbclient

Let us know how you get on.

7

Hi @Haioken,

[root@medienserver ~]# yum install libwbclient
Loaded plugins: changelog, fastestmirror
Rockstor-Testing                                                                                                                                                                                                      | 2.9 kB  00:00:00
rockstor                                                                                                                                                                                                              | 2.9 kB  00:00:00
Loading mirror speeds from cached hostfile
 * base: centos.mirror.net-d-sign.de
 * epel: mirror.1000mbps.com
 * extras: mirror.ratiokontakt.de
 * updates: ftp.hosteurope.de
Package libwbclient-4.6.2-10.el7_4.x86_64 already installed and latest version
Nothing to do

then

[root@medienserver ~]# ls -la /usr/lib64/libwbclient.so.0
ls: cannot access /usr/lib64/libwbclient.so.0: No such file or directory

strange?

[root@medienserver ~]# yum search libwbclient
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.net-d-sign.de
 * epel: mirror.1000mbps.com
 * extras: mirror.ratiokontakt.de
 * updates: ftp.hosteurope.de
========================================================================================================= N/S matched: libwbclient ==========================================================================================================
sssd-libwbclient.x86_64 : The SSSD libwbclient implementation
sssd-libwbclient-devel.i686 : Development libraries for the SSSD libwbclient implementation
sssd-libwbclient-devel.x86_64 : Development libraries for the SSSD libwbclient implementation
libwbclient.i686 : The winbind client library
libwbclient.x86_64 : The winbind client library
libwbclient-devel.i686 : Developer tools for the winbind library
libwbclient-devel.x86_64 : Developer tools for the winbind library

  Name and summary matches only, use "search all" for everything.
8

Strange to say the least - possibly mildly scary.
On mine, the file I looked for appears to be a symbolic link to the same directory, but with a minor version on the end:

libwbclient.so.0.13

This link should be in the package:

[ root@rockout (pass 0s) /usr/lib64 ]# rpm -ql libwbclient
/usr/lib64/samba/libwinbind-client-samba4.so
/usr/lib64/samba/wbclient/libwbclient.so.0
/usr/lib64/samba/wbclient/libwbclient.so.0.13

So, first things first, try reinstalling the package, then (if successful) restart nmb,service

yum reinstall libwbclient -y && systemctl restart nmb.service

If that fails, please check for any possible versions:

ls -la /usr/lib64/libwbclient.so*

If you find one, (which is theoretically libwbclient.so.0.13), you may be able to successfully re-create the link:

ln -s libwbclient.so.0.13 libwbclient.so.0

Substituing libwbclient.so.0.13 with whatever you find.

Hope this helps!

2 Likes
9

Hi @Haioken,

[root@medienserver ~]# yum reinstall libwbclient -y
Loaded plugins: changelog, fastestmirror
Rockstor-Testing                                         | 2.9 kB     00:00
rockstor                                                 | 2.9 kB     00:00
-----snip-----
  Installing : libwbclient-4.6.2-10.el7_4.x86_64                            1/1
  Verifying  : libwbclient-4.6.2-10.el7_4.x86_64                            1/1

Installed:
  libwbclient.x86_64 0:4.6.2-10.el7_4

Complete!

testing…

[root@medienserver ~]# ls -la /usr/lib64/libwbclient.so.0
lrwxrwxrwx 1 root root 19 Sep 17 14:29 /usr/lib64/libwbclient.so.0 -> libwbclient.so.0.13

Well that did it, Samba is back again!
Thank you very much for your patience and support!

Bert

10

Fantastic. Where the symlink went is a mystery to me, but great to see it back up and running for you!

11

I appear to run into the same issue, but unfortunately re-installing libwbclient doesn’t seem to work:

[root@rockvm ~]# yum reinstall libwbclient -y
Loaded plugins: changelog, fastestmirror
rockstor                                                                                                                                                               | 2.9 kB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirror.colocenter.nl
 * epel: mirror.speedpartner.de
 * extras: mirror.colocenter.nl
 * updates: mirror.colocenter.nl
Resolving Dependencies
--> Running transaction check
---> Package libwbclient.x86_64 0:4.4.4-14.el7_3 will be updated
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-common-tools-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-winbind-clients-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-winbind-modules-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-common-libs-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-libs-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-client-libs-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwbclient = 4.4.4-14.el7_3 for package: samba-winbind-krb5-locator-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwinbind-client-samba4.so(SAMBA_4.4.4)(64bit) for package: samba-winbind-modules-4.4.4-14.el7_3.x86_64
--> Processing Dependency: libwinbind-client-samba4.so(SAMBA_4.4.4)(64bit) for package: samba-winbind-clients-4.4.4-14.el7_3.x86_64
---> Package libwbclient.x86_64 0:4.6.2-10.el7_4 will be an update
---> Package libwbclient.x86_64 0:4.6.2-10.el7_4 will be erased
--> Finished Dependency Resolution
Error: Package: samba-winbind-clients-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
Error: Package: samba-winbind-modules-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
Error: Package: samba-winbind-krb5-locator-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
Error: Package: samba-common-libs-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
Error: Package: samba-common-tools-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
Error: Package: samba-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
Error: Package: samba-libs-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
Error: Package: samba-winbind-modules-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwinbind-client-samba4.so(SAMBA_4.4.4)(64bit)
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwinbind-client-samba4.so(SAMBA_4.4.4)(64bit)
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
              ~libwinbind-client-samba4.so(SAMBA_4.6.2)(64bit)
Error: Package: samba-winbind-clients-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwinbind-client-samba4.so(SAMBA_4.4.4)(64bit)
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwinbind-client-samba4.so(SAMBA_4.4.4)(64bit)
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
              ~libwinbind-client-samba4.so(SAMBA_4.6.2)(64bit)
Error: Package: samba-client-libs-4.4.4-14.el7_3.x86_64 (@anaconda/3)
           Requires: libwbclient = 4.4.4-14.el7_3
           Removing: libwbclient-4.4.4-14.el7_3.x86_64 (@anaconda/3)
               libwbclient = 4.4.4-14.el7_3
           Updated By: libwbclient-4.6.2-10.el7_4.x86_64 (updates)
               libwbclient = 4.6.2-10.el7_4
           Available: libwbclient-4.6.2-8.el7.i686 (base)
               libwbclient = 4.6.2-8.el7
 You could try using --skip-broken to work around the problem
** Found 272 pre-existing rpmdb problem(s), 'yum check' output follows:
<snip tons of packages being duplicates>
yum-plugin-fastestmirror-1.1.31-42.el7.noarch is a duplicate with yum-plugin-fastestmirror-1.1.31-40.el7.noarch

The file and symlink are both present, though:

[root@rockvm ~]# ls -la /usr/lib64/libwbclient.so*
lrwxrwxrwx. 1 root root 19 18 sep 23:50 /usr/lib64/libwbclient.so.0 -> libwbclient.so.0.12
lrwxrwxrwx. 1 root root 40 18 sep 23:50 /usr/lib64/libwbclient.so.0.12 -> /etc/alternatives/libwbclient.so.0.12-64

I tried a yum history undo, but that didn’t seem to work. Neither did a yum clean all and yum update.
Any further ideas?

12

@Bert, @Haioken, and @doenietzomoeilijk

The update in question that seems to have upset things here would appear to be the system wide shift from upstream that moved our base from CentOS 7.3 to 7.4 around the time that Rockstor 3.9.1.11 was released.
I.e, having just updated a system from Rockstor 3.9.1-10 to 3.9.1-13, samba and co were updated from 4.4.4 to 4.6.2 and initially that caused a failure of the newly upgraded samba to start due to a nmbd failure:

dell1 systemd[1]: Stopping Samba NMB Daemon...
dell1 systemd[1]: Starting Samba NMB Daemon...
dell1 nmbd[3828]: /usr/sbin/nmbd: /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.4.4' not found (required by /lib64
/libwbclient.so.0)
dell1 nmbd[3828]: /usr/sbin/nmbd: /usr/lib64/samba/libwinbind-client-samba4.so: version `SAMBA_4.4.4' not found (required by
 /lib64/libwbclient.so.0)
dell1 systemd[1]: nmb.service: main process exited, code=exited, status=1/FAILURE
dell1 systemd[1]: Failed to start Samba NMB Daemon.
dell1 systemd[1]: Unit nmb.service entered failed state.
dell1 systemd[1]: nmb.service failed.

However in the above case an:

systemctl restart nmb.service

resulted in nmbd starting successfully and consequently samba also. A reboot would have done the same and a subsequent reboot seems to be functioning as expected, ie prior working samba share still working.

So ‘so far’ I’ve only seen a momentary samba issue related to this system wide update.

@doenietzomoeilijk your log appears to indicate a failure of your system to update, ie you don’t appear to be on the new samba version:

yum list installed samba
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirroring.pulsant.co.uk
 * epel: www.mirrorservice.org
 * extras: centos.mirroring.pulsant.co.uk
 * updates: centos.mirroring.pulsant.co.uk
Installed Packages
samba.x86_64                                                     4.6.2-10.el7_4                                                     @updates

the failure of the update is also indicated by the lower version in your links:

ls -la /usr/lib64/libwbclient.so*
lrwxrwxrwx 1 root root 19 Sep 20 14:06 /usr/lib64/libwbclient.so.0 -> libwbclient.so.0.13
lrwxrwxrwx 1 root root 40 Sep 20 14:06 /usr/lib64/libwbclient.so.0.13 -> /etc/alternatives/libwbclient.so.0.13-64

ie this successfully updated machine shows the 0.13 version like @Haioken’s example and @Bert’s successful re-install example.

Which updates release are you subscribed to: testing or stable? I suggest you try and run a system wide update and see what error messages, if any, you see in the logs.

@Haioken Thanks for doing a bunch of the diagnosis here. And yes this is “Strange”. I’m trying a few more systems here that are old enough to be affected and see if anything else sheds light on this.

13

The full output of the re-install command was a bit long to paste in here (32k text limit), I’ve chucked it in a pastebin here.

I’m on stable. The update was done from within the web UI (which notified me of the update in the first place). What exactly do you mean with a ssytem wide update? A yum upgrade tells me it doesn’t need to do anything. I’m open to suggestions as to how to get this boat floating again (and do some googling on my own as well, since this seems to be more of a CentOS / yum issue than a Rockstor issue).

14

A little update to my situation, and maybe some background: I’m still in the “trying out if Rockstor is something I want to run with”-phase, and despite the hiccups I’ve run into so far, I’m actually preferring it to the contenders. The reasonably low-overhead, BTRFS-based setup strikes me as a more tenable solution than the tricked-into-working software from a major NAS vendor that shall remain unnamed. Anyway…

The testing is taking place inside a VM on my laptop with a couple of virtual “disk” to horse around with. This means I can easily make snapshots, and since my previous setup decided to throw in the towel yesterday (complained during boot, didn’t show the “data” volumes, after a reboot I could see the volumes but Rockstor didn’t start, etc), I’ve decided to start with a clean slate: I’ve reinstalled on the rocktstor_root (and was subsequently able to re-add my existing data volumes as if nothing had happened, that was a nice surprise). I have not run any updates yet, and I’ve made a snapshot.

So, in order to maybe assist in troubleshooting this issue: what now? I can run the update, I can do all kinds of checks before I do so and afterwards. @phillxnet: What would you have me do?

15

@doenietzomoeilijk

Thanks for your efforts on this one. Essentially if you could subscribe to the testing channel and then do a Web-UI initiated update and watch what happens as with my tests here I wasn’t able to reproduce the persistent samba issue, ie it would fail to restart directly after the 7.4 related updates but just switching it back on via the services page or rebooting had everything back in place again. However my tests were on updates from Rockstor version 3.9.1-7 and 3.9.1-10 going up to 3.9.1-13. See if your experience is the same as mine this time around, as to address an issue we really need a reliable way to reproduce it.

Thanks again for your testing.

Whenever a new Rockstor package version is detected the update that is run is a system wide one, ie all packages are updated along with the rockstor one. We also have the flashing icon in the top right corner in front of the kernel version in the header that allows for all packages except the rockstor package to be updated. A fancy featured added by @Flyer.

See how you get on as it would be great to narrow down what’s causing these samba restart failures of late. But on the Rockstor code side nothing has really changed on the samba front, hence the suspicion on the upstream updates.

See how it goes.

In my tests I initially setup a pool / share / smb export then did the update to see the effect on the ‘live’ samba.

Yes the pool import seems to work quite nicely, although there are, as always, improvements to be had. Have you seen the related Configuration Backup and Restore feature. That has received some bug fixes of late but also needs more testing / attention.

16

Essentially if you could subscribe to the testing channel and then do a Web-UI initiated update and watch what happens as with my tests here I wasn’t able to reproduce the persistent samba issue, ie it would fail to restart directly after the 7.4 related updates but just switching it back on via the services page or rebooting had everything back in place again. However my tests were on updates from Rockstor version 3.9.1-7 and 3.9.1-10 going up to 3.9.1-13. See if your experience is the same as mine this time around, as to address an issue we really need a reliable way to reproduce it.

You’re sure about the Testing channel? I’m fairly sure I was on the regular one before - wouldn’t attempting to update on the same channel be a more trustworthy test of being reproducible? (Currently I’m on 3.9.1-0, btw).

Of course I could try both stable and testing, rolling back to an earlier snapshot in between.

We also have the flashing icon in the top right corner in front of the kernel version in the header that allows for all packages except the rockstor package to be updated.

Ah yes, that’s how I ran the update in the first place. Hit the little blinky icon and OK’d the following dialog. I could try both web-update and regular yum upgrade.

I’ll see if I can grab some time later tonight.

17

Update: I’ve run the update on the stable channel using the blinky thingy, and it worked properly. Samba was switched off post-install (and during the installation I kept a tail on log messages, and I did see that dreaded SAMBA-4.4.4 not found message whizzing past, but eventually the update was done, and I was able to switch on Samba again.

Things I’ve done differently from last time: I’ve actually sat and waited for the thing to do it’s, er, thing. There’s no real visual feedback when an update is in progress, and I may have interrupted a previous one (not sure if at some point I told the thing to reboot or not, that time - I may have). Maybe there should be some visual clue telling the hapless user that an upgrade is in progress and would they please refrain from mucking with the system, thank you very much. I’m starting to think this may have been a user error, at least on my end.

I have a dump of things yum wanted to go and upgrade, and I have the logs of during the upgrade, if anyone’s interested in them. If need be, I could roll back the snapshot and try an update on the testing channel, for science.

18

I don’t know whether that’s already on the list of improvements, but I would second this suggestion. I find myself preferring doing a “manual” yum update from SSH rather than using the webUI as this allows me to see what is going on and most importantly when the update is complete.

19

I’d assume there’s a lock file somewhere while yum is working its magic - in the very least it shows up in the process list a couple of times. Would it be possible for the system to monitor that? Kind of like the current blinky thingy supposedly works? I’m not familiar with how things work under the hood in Rockstor - I’m far too new to it :slight_smile: Also while I know my way around basic Linux things (my work laptop runs Fedora, and I’ve been tinkering with all kinds of *nix boxen for years), I’m not super familiar with how yum or yum-cron works, assuming it’s yum-cron that is used for the updates.

20

Tiny update: I got a second update after the first one (this time just some Samba stuff). During the update, there was a /var/run/yum.pid present, so that’s definitely something to test for. Samba was still On after the update, everything continues to work splendidly.

21

@doenietzomoeilijk

Great, glad to hear it; and thanks for the confirmation / testing.

And:

Not as far as I’m aware, at least in the issue record anyway.

@doenietzomoeilijk Would you care to make a GitHub issue with this feature as a request and indicate @Flox as a second suggester. Might be nice to link back to this forum thread also as then the implementer can see the genesis of the request as it were. Not entirely sure how we would monitor such a thing but as mentioned we could reflect a yum db lock status as a warning that package maintenance is ‘currently active’ or the like with a suitable warning against ill advised actions or the like.

Nice work people (@Bert, @Haioken, @doenietzomoeilijk, @Flox)

22

No problem, I’ll cook up an issue tomorrow morning - it’s getting kinda late here, at the moment. :wink:

23

There’s an issue up - @Flox, feel free to +1 it.

24

I was having the same issue after installing 3.9.1 on a new machine and I had tried installing all the updates including the most recent test channel updates and it sill wasn’t working until I found this thread. I tried each recommendation one at a time and nothing worked (including systemctl restart nmb.service at first), and yum wouldn’t let me reinstall libwbclient-4.6.2, so I uninstalled it (only, I left the dependencies since there were many) and then installed it again, and while I was at it I ran yum clean all and then eventually something clicked and systemctl restart nmb.service finally fixed it.

After writing all that I see that I seem to have had the same issue as doenietzomoeilijk, with the SAMBA_4.4.4 not found popping up somewhere along the way. And now that I’ve read through and saw that it may have been caused by an interrupted update cycle, I may have rebooted during an update as well. I was booting off a really slow USB flash drive and it wouldn’t shut down, so I thought it had some issue and just turned the computer off after a few minutes and restarted, so that may have been what caused my issue as well, maybe?

1 Like