rsync still broken

rsync still broken

[Frank Fesevur]

Hi Jari,

It looks like rsync is still broken. When you released 3.1.0 it had an
upstream bug that made it extremely slow on cygwin. Other then that it
also had another problem that I reported here:
https://www.cygwin.com/ml/cygwin/2014-08/msg00440.html

Yaakov made 3.1.0 test and we all automatically reverted to 3.0.9.

Recently 3.1.1 was released leaving the broken test version 3.1.0 as
previous. I didn't update my servers yet, The day before yesterday I
did, so I went from 3.0.9 to 3.1.1.

And yesterday I saw that my backup *completely* failed because of these errors:
@ERROR: setgid failed
rsync error: error starting client-server protocol (code 5) at
main.c(1653) [Receiver=3.1.0]
ERROR: /usr/bin/rsync returned 5 while processing
rsync://192.168.200.208/backup-d/DataShares/
@ERROR: setgid failed
rsync error: error starting client-server protocol (code 5) at
main.c(1653) [Receiver=3.1.0]
ERROR: /usr/bin/rsync returned 5 while processing
rsync://192.168.200.208/backup-d/DataGit/
@ERROR: setgid failed

Very similar error as I reported for 3.1.0.

FYI, I run the cygwin based rsync daemon on my Windows servers and
have a Ubuntu server that backups the data with rsnapshot (which
utilizes rsync to do the transfers). The errors above are shown on the
Ubuntu server.

Luckily I still had a 3.0.9 executable on one machine and I've put
that exe on all the updated servers to make sure the backup worked
again last night. So those systems new think they have 3.1.1 but the
actually have the exe of good old properly working 3.0.9

Note that 3.0.9 is not available on the mirrors anymore. I hope that
it can be restored on the mirrors and made current again for the time
being and that this you have time to look into this problem. But the
bottom line is that 3.1.x breaks my backup.

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync still broken

[Linda Walsh]

Frank Fesevur wrote:

> And yesterday I saw that my backup *completely* failed because of these errors:
> @ERROR: setgid failed
> rsync error: error starting client-server protocol (code 5) at
> main.c(1653) [Receiver=3.1.0]
> ERROR: /usr/bin/rsync returned 5 while processing
> rsync://192.168.200.208/backup-d/DataShares/
> @ERROR: setgid failed
> rsync error: error starting client-server protocol (code 5) at
> main.c(1653) [Receiver=3.1.0]
> ERROR: /usr/bin/rsync returned 5 while processing
> rsync://192.168.200.208/backup-d/DataGit/
> @ERROR: setgid failed
====
It sounds like the group you are in on cygwin doesn't exist 
or you are not in it on your target machine.

what group are you in on the windows machine?
if you type 'id', the 2nd number should be your primary gid.

uid=1234(Bliss\law) gid=123(lawgroup) groups=123(lawgroup)...

Then the question is, does your groupname
exist on the server you are transferring it to?  
(or if you are using '--numeric-ids, is your
group# (gid) the same on the server you are 
transferring files to?

If not, are you using the --usermap and/or
--groupmap options to map your Windows ID's to
your server's ID's?

Maybe you have already verified this, but usually
when I get errors in a transfer, it's because the UID's
or user/groupnames on my windows machine don't always match
what is on my server -- they mostly do, but I do see
errors occasionally in it trying to set things.

You can also try the --fake-super option -- that
might fake the id's enough for it to work...

Good luck!


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync still broken

[Frank Fesevur]

2015-03-12 21:34 GMT+01:00 Linda Walsh:
> It sounds like the group you are in on cygwin doesn't exist or you are not
> in it on your target machine.
>
> what group are you in on the windows machine?
> if you type 'id', the 2nd number should be your primary gid.
>
> uid=1234(Bliss\law) gid=123(lawgroup) groups=123(lawgroup)...

The first gid of the user running the rsyncd service is 512, but...

> Then the question is, does your groupname
> exist on the server you are transferring it to?  (or if you are using
> '--numeric-ids, is your
> group# (gid) the same on the server you are transferring files to?

... I use --numeric-ids and I have these two lines in the rsyncd.conf
uid = 0
gid = 0

If I understand it correctly now rsync sends all files with uid=0 and
gid=0. And obviously those uid and gid exist on the Linux machine.

> If not, are you using the --usermap and/or
> --groupmap options to map your Windows ID's to
> your server's ID's?

No, I don't use that.

> Maybe you have already verified this, but usually
> when I get errors in a transfer, it's because the UID's
> or user/groupnames on my windows machine don't always match
> what is on my server -- they mostly do, but I do see
> errors occasionally in it trying to set things.

I thought uid=0 and gid=0 would solve that.

> You can also try the --fake-super option -- that
> might fake the id's enough for it to work...

There is a "fake super = yes" in the rsyncd.conf and the --fake-super
option is added on the Linux server.

But the thing that surpises me is that in 3.0.9 is just worked.

Regards,
Frank

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rsync still broken

[Linda Walsh]

Frank Fesevur wrote:
> ... I use --numeric-ids and I have these two lines in the rsyncd.conf
> uid = 0
> gid = 0
---
	How is your local rsync talking to the server?

I.e. using the 'rsyncd' daemon running on the server?

For me, I don't have the rsyncd daemon running full time on the
server, but use the "remote shell" option with my RSYNC_RSH environment
var set to 'ssh'.  Just reading on the man page that this is the
default on most systems these days.

If you use the "rsyncd" daemon, you should read the rsyncd.conf manpage:
Especially this section:

       uid    This  parameter  specifies  the  user  name or user ID ...
              The default
              when  run  by a super-user is to switch to the system’s "nobody"
              user.  The default for a non-super-user is to not try to  change
              the user.  See also the "gid" parameter.
---
It doesn't look like uid and gid set to 0 will "just work".

If you use the 'ssh' protocol for transfers, you would need to be
able to login from your local system to the server as 'root' with
no password:

i.e.:

   ssh root@server

If that doesn't work -- rsync probably won't work correctly either.

> But the thing that surpises me is that in 3.0.9 is just worked.
---
I'm still at 3.0.9 and don't want to break things, but you are right:
---
law.Bliss> whoami
Bliss\law
law.Bliss> mkdir cygwin
law.Bliss> mkdir /h/cygwin			## /h=homedir on server
law.Bliss> llg -a /h/cygwin
total 0
drwxrwxr-x+ 1 Bliss\law lawgroup 0 Mar 19 17:07 ./
drwxr-xr-x+ 1 Bliss\law lawgroup 0 Mar 17 20:36 ../

law.Bliss> echo "test" >cygwin/test.txt
law.Bliss> rsync -a cygwin/. ishtar:cygwin/.
law.Bliss> llg -a /h/cygwin
total 4
drwxrwxr-x+ 1 Bliss\law lawgroup 0 Mar 19 16:42 ./
drwxr-xr-x+ 1 Bliss\law lawgroup 0 Mar 17 20:36 ../
-rw-rw-r--  1 Bliss\law lawgroup 5 Mar 19 16:42 test.txt

law.Bliss> rsync -v
rsync  version 3.0.9  protocol version 30
----
Yup -- seems to work in 3.0.9.



BTW -- rsync is pretty slow for some transfers -- created
a 1G local file and timed transfers with rsync, cp and dd.
with 'dd' I used "direct" which should be synchronous/unbuffered,
but I don't think rsync or cp have options to turn that off.
Still 'dd' was fastest, followed by 'cp' and rsync, well:



law.Bliss> dd if=/dev/zero of=1G bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.96472 s, 547 MB/s	#local file

law.Bliss> time rsync 1G ishtar:cygwin/
18.76sec 3.43usr 2.40sys (31.07% cpu)			#using rsync proto

law.Bliss> rm /h/cygwin/1G             
law.Bliss> time cp 1G /h/cygwin
8.53sec 0.00usr 0.77sys (9.13% cpu)			#cp using SMB/CIFS

law.Bliss> rm /h/cygwin/1G             
law.Bliss> time dd if=1G of=/h/cygwin/1G bs=1M count=1024 oflag=direct       
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 6.91213 s, 155 MB/s	# dd using SMB/CIFS
6.95sec 0.01usr 0.67sys (9.85% cpu)

law.Bliss> rm /h/cygwin/1G				# rsync using SMB/CIFS
law.Bliss> time rsync 1G /h/cygwin/       
29.18sec 3.52usr 3.08sys (22.65% cpu)



You might try reinstalling 3.0.9 and see if that still works --
would help narrow it down to whether or not it is in cygwin or in 
rsync.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple