Cygwin: 2.6.0, rsync 3.1.2, "file has vanished"

Cygwin: 2.6.0, rsync 3.1.2, "file has vanished"

[Shaun Maher]

Hi.

I'm trying to use rsync on Windows Server 2012R2 to sync files to a *nix system (Ubuntu 14.04 or FreeNAS 9.10).  The file set I'm trying to sync contains files with unusual characters in the names.  Most files sync without issue but those files with unusual characters do not sync and rsync logs a message like the following:

  file has vanished: "BackUps(old)/Backup Pre-Rolled Payroll 2015"16-MBTT2016.zip" (in VSSG)
  file has vanished: "Corrupted files/Payment Summaries/Payment Summaries 2015"2016" (in VSSG)
  file has vanished: "Restore/MBTT2017"2016-07-12 MBTT2016.zip" (in VSSG)

The " character mid file name is where the unusual character is.  The character in question in all my test cases is (on Windows) a dot vertically centered.  I've only a minimal understanding of character encoding but it seems to me that it is a 0x2D followed by a 0x31 which makes it a type of dash I guess.

I've given this a good Googling but haven't been able to work it out.  I've tried every combination of --iconv and "charset = " that has been suggested.

I'm a bit stuck on how to move forward from here.  Any help or suggestions would be greatly appreciated.  I can provide an archive with some example files is it will help.

Cheers.
Shaun.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin: 2.6.0, rsync 3.1.2, "file has vanished"

[Brian Inglis]

On 2016-11-28 00:07, Shaun Maher wrote:
> I'm trying to use rsync on Windows Server 2012R2 to sync files to a
> *nix system (Ubuntu 14.04 or FreeNAS 9.10). The file set I'm trying
> to sync contains files with unusual characters in the names. Most
> files sync without issue but those files with unusual characters do
> not sync and rsync logs a message like the following:
>   file has vanished: "BackUps(old)/Backup Pre-Rolled Payroll
> 2015"16-MBTT2016.zip" (in VSSG)
>   file has vanished: "Corrupted files/Payment Summaries/Payment
> Summaries 2015"2016" (in VSSG)
>   file has vanished: "Restore/MBTT2017"2016-07-12 MBTT2016.zip" (in
> VSSG)
> The " character mid file name is where the unusual character is. The
> character in question in all my test cases is (on Windows) a dot
> vertically centered. I've only a minimal understanding of character
> encoding but it seems to me that it is a 0x2D followed by a 0x31
> which makes it a type of dash I guess.
> I've given this a good Googling but haven't been able to work it out.
> I've tried every combination of --iconv and "charset = " that has
> been suggested.
> I'm a bit stuck on how to move forward from here. Any help or
> suggestions would be greatly appreciated. I can provide an archive
> with some example files is it will help.

utf-16 0x2d31 would be "â´±" Tifinagh letter Yab utf-8 0xe2b4b1 which 
seems unlikely 
"⸱" word separator middle dot would be utf-16 0x2e31 utf-8 0xe2b8b1
you can type this in any console window as 
	<alt...><num-+><2><e><3><1><...alt>
meaning you hold alt down while typing the other characters.

Your file names should look like: 
"BackUps(old)/Backup Pre-Rolled Payroll 2015⸱16-MBTT2016.zip"
"Corrupted files/Payment Summaries/Payment Summaries 2015⸱2016"
"Restore/MBTT2017⸱2016-07-12 MBTT2016.zip"

If you're not using a utf-8 character set, Cygwin may not be encoding 
this utf-16 character, or using the correct utf-8 equivalent for Linux, 
so it can be reversibly decoded.

I don't know if rsync --iconv utf-16,utf-8 can bypass Cygwin to get the 
original names, or if it will internally encode, then rsync decode and 
encode, and produce a mess.

You might want to get the creator to change the punctuation to just 
a normal minus and not get too creative, or skip those directories.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin: 2.6.0, rsync 3.1.2, "file has vanished"

[Wayne Davison]

On Sun, Nov 27, 2016 at 11:07 PM, Shaun Maher <shaun@mach.com.au> wrote:
>   file has vanished: "BackUps(old)/Backup Pre-Rolled Payroll 2015"16-MBTT2016.zip" (in VSSG)

Those file-has-vanished errors shouldn't be affected by the --iconv
option because that is a sender-side error, and the rsync sender
directly reads file names via readdir() and then tries to stat() those
names (it is the receiving side that might get converted names via the
--iconv translations).

So, the question is: why is rsync getting a filename via readdir that
it can't stat?  Can you try out this C program:
http://opencoder.net/list-and-stat.c

You compile it with gcc ("gcc -o list-and-stat list-and-stat.c") and
then run it with a directory name where one of your problem files
exists. It will output things like this:

74 65 73 74 e2 b8 b1 66 69 6c 65 | test⸱file

If you see a "FAILED to stat" message, send the byte sequence for that
file to the list and mention what filesystem type it is mounted on.

If it doesn't generate an error, what rsync version are you using? Is
it the cygwin version or something else?

The error indicates that an rsync daemon module is involved (the "in
VSSG" bit), so you might try a manual rsync copy of a problem file and
see if the issue is the daemon context or rsync in general.

..wayne..

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin: 2.6.0, rsync 3.1.2, "file has vanished"

[Shaun Maher]

Hi Wayne.

Thanks for the quick response.  I genuinely appreciate it.

>Those file-has-vanished errors shouldn't be affected by the --iconv...
This was also my understanding but I wanted to be through.

>Can you try out this C program: http://opencoder.net/list-and-stat.c
I have done this.  The result for one of the problematic files is:

    4d 42 54 54 32 30 31 37 22 32 30 31 36 2d 30 37 2d 31 32 20 4d 42 54 54 32 30 31 36 2e 7a 69 70 | MBTT2017"2016-07-12 MBTT2016.zip

So, it can see the file in the directory AND stat it?

>If it doesn't generate an error, what rsync version are you using?
I am using the rsync 3.1.2 provided by cygwin.

>The error indicates that an rsync daemon module is involved...
Wow.  OK.  When using rsync to copy locally, without daemon mode, it works.

Any other suggestions?

Cheers.
Shaun.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple