Re: patch command incorrectly capitalizes filenames that live on external USB flash drives

[Brian Inglis <Brian.Inglis@SystematicSw.ab.ca>]

On 2020-05-13 23:07, Jason Gross wrote:
> On Wed, May 13, 2020 at 10:32 PM Jason Gross wrote:
>> On Wed, May 13, 2020 at 8:31 PM Jason Gross wrote:
>>> Brian Inglis wrote:
>>>> Marco Atzeri wrote:
>>>>> Thomas Wolff wrote:
>>>>>> On Tue, Apr 28, 2020 at 3:27 PM Jason Gross wrote:

[Please insert comments inline or add at the bottom on this mailing list]

>>>>>> Consider the following script in foo.sh:
>>>>>> #!/usr/bin/env bash
>>>>>> set -ex
>>>>>> cd "$1"
>>>>>> rm -rf foo
>>>>>> mkdir foo
>>>>>> cd foo
>>>>>> cat > Makefile <<EOF
>>>>>> a
>>>>>> b
>>>>>> c
>>>>>> d
>>>>>> e
>>>>>> EOF
>>>>>> cat > diff <<EOF
>>>>>> diff --git a/Makefile b/Makefile
>>>>>> index 9405325..86d2f8c 100644
>>>>>> --- a/Makefile
>>>>>> +++ b/Makefile
>>>>>> @@ -1,5 +1,5 @@
>>>>>>  a
>>>>>>  b
>>>>>> -c
>>>>>> +ccc
>>>>>>  d
>>>>>>  e
>>>>>> EOF
>>>>>> patch -p1 -i ./diff
>>>>>> ls
>>>>>> If I run `./foo.sh /cygdrive/c/`, I get, as expected,
>>>>>> + cd /cygdrive/c/
>>>>>> + rm -rf foo
>>>>>> + mkdir foo
>>>>>> + cd foo
>>>>>> + cat
>>>>>> + cat
>>>>>> + patch -p1 -i ./diff
>>>>>> patching file Makefile
>>>>>> + ls
>>>>>> diff  Makefile

>>> Sorry for the late reply; I can see replies to my messages at 
>>> https://cygwin.com/pipermail/cygwin/2020-April/244660.html, but somehow 
>>> I'm not receiving them in Gmail. I've tried (re?)subscribing to the 
>>> cygwin mailing list, hopefully this fixes the problem.
>>>> You are throwing a puzzle into the mailing list and if you are lucky, 
>>>> someone may like to solve it.>>>> But perhaps: can you try to minimize your test case, please.>>>> Something
like: touch Makefile; ls (if that's it).
>>> I think there's some sort of misconception here. touch and cat create 
>>> correctly capitalized files, and sed -i doesn't change capitalization, 
>>> even on my FAT32 drive. patch is the only command I've found so far
>>> which capitalizes filenames when modifying files. I can try to dig into
>>> the source code of patch and figure out a minimal C program that breaks 
>>> casing on files, but, come on, the fact that patch seems to capitalize 
>>> the file name of every file it modifies, and no other utility does this, 
>>> that seems like a pretty minimal test-case to me. And anyway, the cygwin 
>>> patch sources (version 2.7.4) are impossible to compile, because safe.c 
>>> can't find sys/resource.h and passing -I/usr/include via CFLAGS hits an 
>>> internal bug in patch's configure script (search.h: present but cannot
>>> be compiled; sys/timeb.h: present but cannot be compiled; fcntl.h:
>>> present but cannot be compiled). (I've emailed bug-patch@gnu.org as
>>> requested by the configure script, but so far 
>>> https://lists.gnu.org/archive/html/bug-patch/ isn't showing anything 
>>> newer than January.)
>>>>>> If I instead run `./foo.sh /cygdrive/h/`, I get
>>>>>> ```
>>>>>> + cd /cygdrive/h/
>>>>>> + rm -rf foo
>>>>>> + mkdir foo
>>>>>> + cd foo
>>>>>> + cat
>>>>>> + cat
>>>>>> + patch -p1 -i ./diff
>>>>>> patching file Makefile
>>>>>> + ls
>>>>>> diff  MAKEFILE
>>>>>> ```
>>>>>>
>>>>>> My C drive is an internal SSD (NTFS), my H drive is an external flash
>>>>>> drive (FAT32).  I installed cygwin with the commands:
>>>>>> ```
>>>>>> powershell -Command "(New-Object
>>>>>> Net.WebClient).DownloadFile('http://www.cygwin.com/setup-x86_64.exe',
>>>>>> 'setup-x86_64.exe')"
>>>>>> SET CYGMIRROR=http://mirror.easyname.at/cygwin
>>>>>> SET CYGROOT=H:\cygwin64
>>>>>> SET CYGCACHE=%CYGROOT%\var\cache\setup
>>>>>> setup-x86_64.exe -qnNdO -R %CYGROOT% -l %CYGCACHE% -s %CYGMIRROR% -P
>>>>>> rsync -P patch -P diffutils -P make -P unzip -P m4 -P findutils -P
>>>>>> time -P wget -P curl -P git -P
>>>>>> mingw64-x86_64-binutils,mingw64-x86_64-gcc-core,mingw64-x86_64-gcc-g++,mingw64-x86_64-pkg-config,mingw64-x86_64-windows_default_manifest
>>>>>> -P mingw64-x86_64-headers,mingw64-x86_64-runtime,mingw64-x86_64-pthreads,mingw64-x86_64-zlib
>>>>>> -P python3
>>>>>> ```
>>>>>>
>>>>>> Running `patch -v` says `GNU patch 2.7.4`.  Note that this happens
>>>>>> regardless of whether I install cygwin itself on my external flash
>>>>>> drive or on my internal HD.
>>>>>>
>>>>>> This came up when trying to run `opam install findlib` (which fails
>>>>>> when the home directory is on an external USB drive).

>>>> That might be expected with FAT32, which is normally the default format
>>>> for flash drives, for maximum compatibility with microcontrollers,
>>>> which may not create VFAT Long File Names when file names are <= 8.3,
>>>> so they appear as upper case.
>>> This does not explain why `ls` displays "Makefile" as "Makefile" before I
>>> run `patch`, but displays the filename as "MAKEFILE" after I run `patch`.
>>> Nor does it explain why this happens to patch-modified files, but not to
>>> files modified via sed -i.
>>>> use a flash driver with NTFS and check the difference
>>> Indeed, I can confirm that this issue occurs when it's FAT or FAT32, and 
>>> does not occur under NTFS nor exFAT.
>>>> I doubt it is a patch issue
>>> Do you have another utility that you suggest I try that you think will 
>>> display the same problem as patch?  So far, `| tee -a`, `sed -i`, 
>>> `touch`, `>`, and `>>` all do not display this issue, while `patch` 
>>> does.
>> By the way, when I run the same script on the same flash drive from WSL, it
>> works fine, and does not capitalize the filename.  So this does seem to be
>> a cygwin-patch specific issue...
> Also btw, the bug I reported to patch is at 
> https://lists.gnu.org/archive/html/bug-patch/2020-05/msg00000.html . It's not
> clear to me if this is a cygwin issue or a patch issue.
It may be neither - it may be a Mingw/Msys2 issue - that could explain why you
can not reproduce the issue with other utilities.
You appear to have installed and be running x86_64-w64-mingw32-gcc, binutils,
and libraries, and not Cygwin gcc, binutils, and libraries.
That will not build Cygwin sources: examine your bug report carefully, check
your *PATH*, especially for a ".:" entry, and show the output from:

	$ pwd; which -a cat gcc ls patch tee touch sed

To re-/build Cygwin utilities, install cygport, which will install most of the
required packages and run it.

If you are using a FAT32 file system and create or rename a file with a
naturally short name <= 8.3, you will likely end up with an all uppercase name,
unless the program uses a case sensitive long filename API to ensure output case.
If you use Cygwin rather than Mingw/Msys2 utilities, it may work, or you may
have to force use of a long name > 8.3 e.g. Makefile9.temp, then rename to keep
the desired case.
It's been a while since I've done anything on any FAT file system.

You may want to try enabling long paths to see if this solves your problem:

https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#enable-long-paths-in-windows-10-version-1607-and-later

$ regtool get -pv /proc/registry/HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet\
/Control/FileSystem/LongPathsEnabled
0

$ regtool set -iv /proc/registry/HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet\
/Control/FileSystem/LongPathsEnabled 1

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]