Strange paths in NTFS reparse points created by Cygwin Setup for e.g. TTF fonts

Strange paths in NTFS reparse points created by Cygwin Setup for e.g. TTF fonts

[Michael Soegtrop]

Dear Cyhwin Users and Team,

since a while I have issues removing cygwin installations, especially 
the symlinks to true type fonts in /usr/share/fonts/microsoft.

Looking at these links with Windows tools I get:

C:\bin\cygwin\usr\share\fonts\microsoft>fsutil reparsepoint query 
wingding.ttf
Reparse Tag Value : 0xa000001d
Tag value: Microsoft
Tag value: Name Surrogate

Reparse Data Length: 0x00000025
Reparse Data:
0000:  02 00 00 00 2f 6d 6e 74  2f 63 2f 57 69 6e 64 6f  ..../mnt/c/Windo
0010:  77 73 2f 46 6f 6e 74 73  2f 77 69 6e 67 64 69 6e  ws/Fonts/wingdin
0020:  67 2e 74 74 66                                    g.ttf

I wonder if the path "/mnt/c/Windows/Fonts/wingding.ttf" is something 
which should be written into a NTFS reparse point by cygwin setup. 
Probably not - it looks like a cygwin path and it is understandable that 
this confuses NTFS.



Btw.: I meanwhile managed to reliably remove cygwin install folders with 
the below command sequence ($1 being the cygwin folder name) - sorry an 
ugly mix of bash and DOS tools:


PATH_WINFMT="$(cygpath -w -a $1)"

rm -rf "$1" && { echo "first removal run successful"; exit; }

TAKEOWN /F "$PATH_WINFMT" /R /D Y

ICACLS "$PATH_WINFMT" /grant <your user name>:F /T

CMD /C "DIR /AL /S /B $PATH_WINFMT" | sed 's|\\|\\\\|g' | tr -d "\r" | 
while read f; do echo removing reparsepoint "$f"; FSUTIL REPARSEPOINT 
DELETE "$f"; done

rm -rfv "$1"


Best regards,

Michael

Re: Strange paths in NTFS reparse points created by Cygwin Setup for e.g. TTF fonts

[Ken Brown]

On 11/5/2020 4:41 PM, Michael Soegtrop via Cygwin wrote:
> Dear Cyhwin Users and Team,
> 
> since a while I have issues removing cygwin installations, especially the 
> symlinks to true type fonts in /usr/share/fonts/microsoft.
> 
> Looking at these links with Windows tools I get:
> 
> C:\bin\cygwin\usr\share\fonts\microsoft>fsutil reparsepoint query wingding.ttf
> Reparse Tag Value : 0xa000001d
> Tag value: Microsoft
> Tag value: Name Surrogate

This is the IO_REPARSE_TAG_LX_SYMLINK tag, which WSL uses for symlinks and which 
Cygwin also uses by default for symlinks on systems that support it.  But you 
can change this if you don't like it, as I said in my reply to your earlier 
message about this:

   https://cygwin.com/pipermail/cygwin/2020-September/246277.html

> Reparse Data Length: 0x00000025
> Reparse Data:
> 0000:  02 00 00 00 2f 6d 6e 74  2f 63 2f 57 69 6e 64 6f  ..../mnt/c/Windo
> 0010:  77 73 2f 46 6f 6e 74 73  2f 77 69 6e 67 64 69 6e  ws/Fonts/wingdin
> 0020:  67 2e 74 74 66                                    g.ttf
> 
> I wonder if the path "/mnt/c/Windows/Fonts/wingding.ttf" is something which 
> should be written into a NTFS reparse point by cygwin setup. Probably not - it 
> looks like a cygwin path and it is understandable that this confuses NTFS.

It's not written by setup.  It's a Cygwin symlink [also a WSL symlink, as I said 
above], created by the postinstall script

   /etc/postinstall/zp_fontconfig_cache_1.sh

The fact that it contains a Cygwin path as its reparse data is not at all 
surprising:

ls -l /usr/share/fonts/microsoft/wingding.ttf
lrwxrwxrwx ... /usr/share/fonts/microsoft/wingding.ttf -> 
/c/Windows/Fonts/wingding.ttf

Ken

Re: Strange paths in NTFS reparse points created by Cygwin Setup for e.g. TTF fonts

[L A Walsh]

On 2020/11/05 13:41, Michael Soegtrop via Cygwin wrote:
> I wonder if the path "/mnt/c/Windows/Fonts/wingding.ttf" is something 
> which should be written into a NTFS reparse point by cygwin setup. 
> Probably not - it looks like a cygwin path and it is understandable that 
> this confuses NTFS.
>   
====
    Uh....except that "/mnt/c/Windows/Fonts/wingding.ttf" is a valid
NT pathname (file and reg).  For that matter, "/mnt/\x00\x01\x04" is a valid
NT pathname -- how, why?: NT uses a 'count' hidden before the string, so any
character is valid.  However, the above is not universally true in 'Windows'
where some system libraries enforce using backslash as the separator.

    I'm guessing you get some permission denied messages when you try
to delete some fonts.  This can happen in 2 cases: 1) if you try to 
delete the
file in cygwin or 2), if some program hardlinked the font to the same in the
windows directory instead of symlinking it. 

    With both, it comes down to your delete command trying to delete
(for example) a font that is linked into the /windows/fonts dir
and some program (or lib) already has it open.

    Generally, you can't delete *open* files in Windows.  In windows,
an open will lock some range of bytes on the disk.  Windows "lock"s
are locks on byte ranges, "on disk" that windows won't let you write
to, move or delete while someone has that file open.




   

Re: Strange paths in NTFS reparse points created by Cygwin Setup for e.g. TTF fonts

[Michael Soegtrop]

Hi Ken, L A Walsh,

 > But you can change this if you don't like it, as I said in my reply 
to your earlier message about this:

I don't know if I like it yet - I am still in the process of 
understanding what is going on.

I maintain the Windows build of a large open source project and we also 
deliver scripts to setup cygwin on our user's machines. I need to 
understand what is going on before I make changes to settings. I am 
afraid that using non standard settings in the end will lead to more 
confusion than good - especially if people have several cygwin 
installations. Also I need to be able to support our users in case they 
have issues with this.

The interesting point is that in our CI we do several 100 cygwin 
installations (sometimes more than 1000) per month. For 95%..99% the 
removal with "rm -rf" from another cygwin installation works well, in 
about 1%..5% of the case about 250 symlinks / reparse points remain. 
This happens all on the same machines with identical scripts.

With my manual installations of cygwin it is about the same ratio. I had 
two cases where removing a cygwin installation from Windows explorer 
with "shift+delete" did not work cause of these about 250 symlinks - I 
would say I did roughly 100 installations in this time frame.

What I don't understand is why it does work sometimes and sometimes not. 
I always use the same scripts to install and remove cygwin on the same 
machines and then do pretty much the same thing with this cygwin (build 
our open source software) before I delete it.

It is unlikely that the issue is that the target files are open as L A 
Welsh suggested because always either all symlinks or none at all 
remain. The number is always the same (with recent versions afair 258).

@ L A Walsh: you wrote "1) if you try to delete the file in cygwin"

Why shouldn't I be able to remove symlinks with rm -rf from within a 
cygwin? As far as I understand the standard behavior of "rm" is to 
remove the symlink and not its target. What I do when I remove a cygwin 
installation in our CI is an "rm -rf" from a different cygwin 
installation. As I said in most cases this works but rarely it doesn't.

In case it doesn't work the symlinks are quite hard to get rid of. 
FSUTIL REPARSEPOINT DELETE is the only method which works I found so 
far. Even after a reboot, resetting the ACLs in various ways, ... no 
usual method to remove these files works.

I understand that the contents / path of the symlink is as expected, so 
I am looking out for other oddities which could explain this behavior.

Best regards,

Michael

Re: Strange paths in NTFS reparse points created by Cygwin Setup for e.g. TTF fonts

[Ken Brown]

On 11/8/2020 9:40 AM, Michael Soegtrop wrote:
> Why shouldn't I be able to remove symlinks with rm -rf from within a cygwin? As 
> far as I understand the standard behavior of "rm" is to remove the symlink and 
> not its target. What I do when I remove a cygwin installation in our CI is an 
> "rm -rf" from a different cygwin installation. As I said in most cases this 
> works but rarely it doesn't.
[...]
> I understand that the contents / path of the symlink is as expected, so I am 
> looking out for other oddities which could explain this behavior.

I've seen this behavior only in the following situation.  Suppose you create a 
symlink in one Cygwin installation that uses WSL symlinks.  Then you try to 
delete it from a different Cygwin installation with version < 3.1.5 (which is 
when WSL symlinks were first supported).  This will fail, because the older 
Cygwin installation will see a reparse point of a type that it doesn't recognize.

 From what you say, it sounds like this doesn't apply to your situation, so I'm 
at a loss to explain it.

Do you get any error message when you're unable to delete a symlink in one 
Cygwin installation from another?  If not, you could try running 'rm <cygwin 
root>/usr/share/fonts/microsoft/wingding.ttf' under strace to see if that give 
you any clues.  You could also try 'file', 'ls -l', 'stat', etc., to see if 
Cygwin is correctly recognizing the file as a symlink.

For example, I have a Cygwin installation in C:\cygwin64test.  From a different 
Cygwin installation in which C: is mounted on /c, I see the following:

$ file /c/cygwin64test/usr/share/fonts/microsoft/wingding.ttf
/c/cygwin64test/usr/share/fonts/microsoft/wingding.ttf: symbolic link to 
/c/Windows/Fonts/wingding.ttf

Now I remove that symlink under strace:

$ strace -o strace.out rm /c/cygwin64test/usr/share/fonts/microsoft/wingding.ttf

In the file strace.out I see

    57   26112 [main] rm 39456 symlink_info::check: 0x0 = NtCreateFile 
(\??\C:\cygwin64test\usr\share\fonts\microsoft\wingding.ttf)
   100   26212 [main] rm 39456 symlink_info::check: 29 = 
symlink.check(C:\cygwin64test\usr\share\fonts\microsoft\wingding.ttf, 
0xFFFFB680) (mount_flags 0x4008, path_flags 0x10)
    26   26238 [main] rm 39456 path_conv::check: 
this->path(C:\cygwin64test\usr\share\fonts\microsoft\wingding.ttf), has_acls(1)
    25   26263 [main] rm 39456 _unlink_nt: Trying to delete 
\??\C:\cygwin64test\usr\share\fonts\microsoft\wingding.ttf, isdir = 0
   218   26481 [main] rm 39456 _unlink_nt: 
\??\C:\cygwin64test\usr\share\fonts\microsoft\wingding.ttf, return status = 0x0
    33   26514 [main] rm 39456 unlink: 0 = 
unlink(/c/cygwin64test/usr/share/fonts/microsoft/wingding.ttf)

In your case I would expect to see some errors reported when you are unable to 
remove the symlink.

Ken

Re: Strange paths in NTFS reparse points created by Cygwin Setup for e.g. TTF fonts

[L A Walsh]

On 2020/11/08 06:40, Michael Soegtrop wrote:
> What I don't understand is why it does work sometimes and sometimes not. 
> I always use the same scripts to install and remove cygwin on the same 
> machines and then do pretty much the same thing with this cygwin (build 
> our open source software) before I delete it.
----
	Likely because the windows settings are different on the
various machines -- especially the various policies.

	But, first, basic, you know that windows symlinks are
turned off for normal users by default when they first get it (i.e. they
can't create them).  Which ways work and who can create them are settings
under fsultil/behavior/(set/query) symlinkEvaluation.  These 4 settings
control directions of symlinking

	There's another setting that is suppose to control where fonts
can be installed, but can't find it off hand -- in the group policy 
editor.  If I remember, it was suppose to allow/disallow fonts being
installed in folders other than /windows/fonts.  I think the default
might be to allow, but am not sure.  I thought I disabled it, and what
I've seen is the same font-file hardlinked between windows/fonts
and the application path.  


> It is unlikely that the issue is that the target files are open as L A 
> Welsh suggested because always either all symlinks or none at all 
> remain. The number is always the same (with recent versions afair 258).
----
	Windows doesn't hold open all fonts -- but some subset -- they
have some font cache services that may be running at times to support
some apps.  So could be if one of those apps has run, it started its
font caching service, which could lock it.  Each font system has a
set of "core fonts" that it will load whether they are called upon
or not -- that could create a toggle situation between a bunch being
used or not.

	Some applications (though I don't think rm does this) will
follow the symlink by default, so when they try to delete a file,
it doesn't delete the symlink but tries to delete the font, which
may be opened by some service.  

	BTW, are you familiar with 'Process Hacker' (google it, its
open source, and hasn't been changed for a while).  It can replace
the task manager and ProcessExplorer that MS helps distribute.
It's more powerful than either.  But specifically if you can't delete
a file, you can find out what process(es) have the file open.

	Finding that out might enable you to find the what apps
might be running and holding those files open.  Symlinks and
their older counterparts mountd+junction have different formats
for directories and volumes.  Some will work with network, some not.

	If they are all Microsoft windows symlinks, you might
look at the fsutil settings -- as well as open files.

	You maybe said, but don't remember -- is there any error
message when you can't delete those files?

> 
> @ L A Walsh: you wrote "1) if you try to delete the file in cygwin"
> 
> Why shouldn't I be able to remove symlinks with rm -rf from within a 
> cygwin? As far as I understand the standard behavior of "rm" is to 
> remove the symlink and not its target. 
---
	With posix/linux type symlinks, yes, but am not equally
sure that windows behaves the same in all circumstances.  One thing
you could try is to use cygwin-only symlinks and not use
native symlinks -- the cygwin-only links should be more reliably
just controlled by cygwin rules.  If they are native links, there
might be some windows settings interaction(s).

> In case it doesn't work the symlinks are quite hard to get rid of. 
> FSUTIL REPARSEPOINT DELETE is the only method which works I found so 
> far. Even after a reboot, resetting the ACLs in various ways, ... no 
> usual method to remove these files works.
---
	How were these links created?  Also, MSsymlinks unlike
unix symlinks require the target's existence when they are setup.
unix symlinks do no checks on the target.  If the target is missing,
some operations on that symlink (if it is a win-symlink) might not
work as expected.

	Also note, that these two entries in my root dir
look like this from unix:
lrwxrwxrwx   1           20 Nov  6  2014 Prog -> /Program Files (x86)/
lrwxrwxrwx   1           13 Apr 21  2013 Prog64 -> Program Files/

Both sorta look like symlinks.  But in windows, they look 
different because they were made differently, and they may
have different effects on 'rm' from cygwin.

2014/11/06  19:45    <JUNCTION>     Prog [C:\Program Files (x86)]
2013/04/21  22:53    <SYMLINKD>     Prog64 [Program Files]


1. Make sure the symlinks you see in cygwin are the same type.
2. you say you have symlinks left over after you try deletion.
Do they point to existing files? or not?  If not, what happens if
you create the target of one of them, and then try deleting the
symlink?  
3. you said you had to use fsutil to remove some symlinks --
have you tried just 'del' in windows? 

Anyway, sorry to write so much, but I'm trying to point to how there
can be many differences, even when you think you are doing the same
thing to delete a cygwin.

-linda