Rsync error - "protocol version mismatch -- is your shell clean?"

Troubleshooting
1

I’m on Rockstor 4.1.0-0 and I have some scheduled tasks on another box that rsyncs to a share on the Rockstor. This has all been working very nicely to the extent that I stopped checking the logs regularly and I now find that the jobs have been failing for nearly a year - I know, mea culpa!

The logs now report an rsync error which I’m not familiar with and the references that I’ve found on line have not been particularly helpful.

In the logs I get:
[24423] protocol version mismatch – is your shell clean?
[24423] (see the rsync man page for an explanation)
[24423] rsync error: protocol incompatibility (code 2) at compat.c(178) [sender=3.1.2]

Running the command manually rather than in cron returns the same error.

Rockstor has had all of its patches applied and the sending server, although on CentOS 7 (at the moment) is also patched.

Discussions reference .bashrc but I haven’t edited that at all.

One site I found recommended using the command:
‘ssh @ true | wc -c’ as a test and to expect nothing more than a ‘0’ in response.

I got ‘43’, which is not quite the answer to life, the universe and everything so I’m still stuck.

Any thoughts or suggestions please?

2

@jmangan, I have not really used rsync beyond some experimentation, but when looking at the man page under Diagnostics, they are suggesting a slightly different view of it, I think checking essentially the same thing as above, but allows you to check what the discrepancy/error message is:

https://download.samba.org/pub/rsync/rsync.1#DIAGNOSTICS

rsync occasionally produces error messages that may seem a little cryptic. The one that seems to cause the most confusion is
“protocol version mismatch – is your shell clean?”.

This message is usually caused by your startup scripts or remote shell facility producing unwanted garbage on the stream
that rsync is using for its transport. The way to diagnose this problem is to run your remote shell like this:

   ssh remotehost /bin/true > out.dat

then look at out.dat. If everything is working correctly then out.dat should be a zero length file. If you are getting the above error from rsync then you will probably find that out.dat contains some text or data.

Maybe that will give you a better insight?

3 Likes
3

Thanks. I think it may have given me a deeper insight - into some cockup I’ve made.

I thought I had set this for key authentication but now I can’t find the keys. My password isn’t being accepted and the password safe entry that I referenced in my documentation doesn’t exist.

I think I have some work to do . . .

1 Like
4

I’m not sure what’s going on here.

The remote site (RockStor) is set to only allow SFTP, not shell, access. And the account I’m using to connect to it only exists on the RockStor.

So if I try:
ssh Account@remotehost
I get:
This service allows sftp connections only.
Connection to rockstor.mangan.local closed.

If I try:
sftp Account@remotehost
I get:
Connected to remotehost.
sftp>

All good.

So now I try:
rsync Account@remotehost:RemoteDir
protocol version mismatch – is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(178) [Receiver=3.1.2]

If I try:
ssh remotehost /bin/true > out.dat
I get prompted for a password. I’m haven’t specified a remote account so I’ve assumed ‘root’? If I put the remote ‘root’ password in I get dropped back to the prompt and ‘out.dat’ is empty.

If I try:
ssh Account@remotehost /bin/true > out.dat
I don’t get prompted for a password, I get dropped back to the prompt.
out.dat contains:
“This service allows sftp connections only.”

I’m not sure what I’ve learned, though.

5

I haven’t been able to find anything explicitly that says this but could this be some incompatibility between OpenSSH 7.4 (on CentOS 7.9) and OpenSSH 8.4 (on RockStor)?

It seems unlikely because I can get enough of an SSH connection to get told off and I assume that SFTP is using SSH as well but I’m not sure what else it could be,

6

@jmangan Hello again.
Re:

Pretty sure we default to this for all users but we do allow root to ssh I think. Take a look at the following file on the Rockstor front:

/etc/ssh/sshd_config

It will typically have for example:

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server
###BEGIN: Rockstor SFTP CONFIG. DO NOT EDIT BELOW THIS LINE###
Subsystem       sftp    internal-sftp
AllowUsers root

And the explanation of the approach we have take re sftp etc is excellently explained by @Flox in their pr which was the last major change in this area:

Notably an upstream reference here:

In addition, this PR also takes the opportunity to add per-user customization settings to be in accordance with the settings recommended on hte openSUSE wiki:
SDB:SFTP server with Chroot - openSUSE Wiki

Quite possibly actually. I think there were changes made more recently in ssh regarding defaults. You may well be able to force the client CentOS ssh to do the right thing as it may already have the newer mode/protocol or what-ever in it but just not use it as default.

Yes, however it is a sftp subsystem: take a look at that excellent pull request as @Flox took great care in selecting our defaults on that one. We had to negotiate our CentOS to openSUSE move there I believe. I.e. trying to continue our prior users expectations (root ssh login) etc.

If I try to ssh into a Rockstor instance with a normal user created via the Web-UI I get a silent failure on the client where the password is simply re-requested:

client

ssh sshtest@buildvm
Password: 
Password:

server

Rockstor Leap 15.4 installer instance:

Feb 06 12:27:10 buildvm sshd[14898]: User sshtest from 192.168.2.138 not allowed because not listed in AllowUsers
Feb 06 12:27:10 buildvm sshd[14898]: Postponed keyboard-interactive for invalid user sshtest from 192.168.2.138 port 49352 ssh2 [preauth]
Feb 06 12:27:13 buildvm sshd[14900]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.2.138  user=sshtest
Feb 06 12:27:15 buildvm sshd[14898]: error: PAM: Authentication failure for illegal user sshtest from 192.168.2.138

But if I add this user, as the log suggests to the AllowUsers entry (all-be-it in the Rockstor auto area):

AllowUsers root sshtest

and:

systemctl reload sshd.service

I am then able to ssh in via this regular user:

ssh sshtest@buildvm
Password:
Last failed login: Mon Feb 6 12:30:15 WET 2023 from 192.168.2.138 on ssh:notty
There were 3 failed login attempts since the last successful login.

Welcome to Rockstor built on openSUSE.
To update your system from here do:

    zypper refresh
    zypper up --no-recommends

Or use Rockstor’s Web User Interface (Web-UI): recommended method.

Type ‘myip’ then Enter key to get my address for your Browser.
Enjoy …
sshtest@buildvm:~>

Hope that helps.

2 Likes
7

@phillxnet, thanks for that. I’m quite happy with the default of not letting ‘ordinary’ users log on with SSH. I just wanted to be sure @Hooverdan was following what I had tried and why I wasn’t getting the response he, perhaps, had expected.

I’ll have a look at the PR you referenced in case I can modify the settings on the CentOS side but I’m also thinking it may just be time to bite the bullet and migrate to Alma.

Thanks for the comprehensive response.

3 Likes
8

@phillxnet - it looks like I have my answer.
Under ‘Caveats and Limitations’ in the link you supplied:
Configure SFTP server at buildtime and update customization settings. Fixes #2168 by FroggyFlox · Pull Request #2173 · rockstor/rockstor-core · GitHub

the first paragraph states that:
" As discussed in #2168 (comment), the rsync over ssh functionality is disabled by our enforcing of ForceCommand internal-sftp for all exports. I did add a TODO and comment in the code to help with future work to re-enable this feature as custom option surfaced in the webUI:"

I haven’t seen a custom option in the Web UI so, you know what’s coming next, do you have any kind of timescale for this or a workaround in the meantime?

I would also like to say I don’t have many reasons to ask questions on here but, when I do, I find the level of responsiveness and helpfulness to be quite excellent!

Thanks again.

2 Likes
9

@jmangan Hello again, and glad you’ve made some progress on this one.
Re:

Not currently, but you can go ahead and make whatever changes your setup reqruies directly in the previsouly referenced ssh deamon config file. We only re-write it’s contents when-ever a change is requested that affects that file so it should be stable in the mean time.

Thanks re the kind words re forum responses. We have a nice thing going here on the forum and it is made possible by the diligence and generosity of the contributors, both in code and here on the forum. We can’t of course guarantee any timely response but there are just about enough folks here to generally manage. But more are of course always welcome. So if folks know of stuff relevant to various questions the do feel free to chip-in. I’m more of a generalist myself and am often pulled in many directions (as are we all I suspect). So it’s down to a certain amount of multiple chip-in really.

Glad you got this sorted, or at least narrowed down, and re the specifics I’m not entirely sure I can authoritatively answer beyond what I’ve done to date here. But our setup is very close to upstream and as stated any changes to the config done ‘by-hand’ will persist outside of related Web-UI changes anyway.

Hopefully someone more knowledgable on this front can chip in with the exact config you need here. I’m not that up on the details of ssh as it goes, but as stated we carry a near default openSUSE implementation.

Do go ahead and create a feature request issue in rockstor-core, issues for the specific setting you would like to see, once you have the exact parlance for it, and ideally the exact config you would like to be able to enable. That will help massively for getting the feature in as it is often the majority of the work required. And once well defined can more easily/quickly be added.

Hope that helps. And as always let us know here on the forum you solution once found, and keep that issue creation in mind also as you never know, you may get a ‘taker’ that picks it up and runs with it.

2 Likes
10

Hi @jmangan,

@phillxnet already laid out pretty much everything so I tested his workaround and it seems to work for me. To test, I did the following:

  1. in Rockstor’s webUI: create a new user named testuser.
  2. in Rockstor’s webUI, create a share named media and change the access control to make its owner be the testuser user, and its group be the users group (this is the group to which the testuser belongs).
  3. Edit /etc/ssh/sshd_config to have the AllowUsers read:
AllowUsers root testuser
  1. Reload the sshd service: systemctl reload sshd.service

From my client, I tested without using the ssh protocol explicitly:

~/Documents $ rsync -avz testfile.txt testuser@buildvm:/mnt2/media
(testuser@buildvm) Password: 
sending incremental file list
testfile.txt

sent 106 bytes  received 35 bytes  31.33 bytes/sec
total size is 0  speedup is 0.00

I also tested using the ssh protocol explicitly:

~/Documents $ rsync -avze ssh testfile.txt testuser@buildvm:/mnt2/media
(testuser@buildvm) Password: 
sending incremental file list
testfile.txt

sent 106 bytes  received 35 bytes  31.33 bytes/sec
total size is 0  speedup is 0.00

Could you try this and let us know if it works for you as well? Once we have that confirmed to work, we can then see to get what you need exactly.
My apologies if I misunderstood and this isn’t what you’re trying to achieve…

3 Likes
11

@Flox I don’t know what I’m missing here.

My sshd_config file already has the rsync users in the AllowUsers parameter:

Subsystem sftp internal-sftp
AllowUsers root rsyncuser
Match User rsyncuser
ForceCommand internal-sftp
ChrootDirectory /mnt3/rsyncuser
X11Forwarding no
AllowTcpForwarding no

but I’m still getting the ‘Is your shell clean?’ message.

Could it be because I’m using key-based authentication rather than a password?

Thanks.

12

I’m going to leave this until the weekend and build an Alma box in case it is an OpenSSH version mis-match.

I’ll update once I have something to report.

13

Well it’s not the OpenSSL versions then. I get the same issue with a freshly-built Alma box running 8.7

I’ve tried the two variations @Flox tried with the same result as previously.

One thing I wasn’t clear on was my rsync comand (which used to work) just uses the ‘share_name’ for the destination rather than ‘/mnt2/share_name’. I tried using that as well but it made no difference. (I’ve also got a couple of shjares on /mnt3/ which I thought were related to rsync usage?).

I checked the group membership and I had, apparently, created a group specifically for my rsync-ing activities. So I added my rsyncuser to ‘users’ but that also made no difference. Is the ‘users’ group significant? Should it be the primary/only group?

Do I also need to change the group ownership of the destination folders to ‘users’?

I’m baffled at the moment.

Thanks,

14

on the /mnt3/ shares: usually when you create assign a non-root owned share and expose it for sftp, it will create a directory (equaling the sftp share owner) under the /mnt3 folder.

1 Like