public inbox for cygwin@cygwin.com
 help / color / mirror / Atom feed
* Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
@ 2023-09-01  4:23 Cedric Blancher
  2023-09-01 10:57 ` Corinna Vinschen
  0 siblings, 1 reply; 14+ messages in thread
From: Cedric Blancher @ 2023-09-01  4:23 UTC (permalink / raw)
  To: cygwin

Good morning!

During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
odd when I looked at the network traffic generated by one of our
cluster nodes:
It seems that for each call to a tool (i.e. starting "sed" from
"bash") Cygwin searches for *.lnk files.

Is this correct even when the filesystem in question has native
symlink support (e.g. NFS)?

Excerpt from tshark:
1746 3.800878384 192.168.2.152 → 192.168.2.150 NFS 298 V4 Call lookup
LOOKUP DH: 0x49b89d4e/staden | LOOKUP sed
1747 3.800910202 192.168.2.150 → 192.168.2.152 NFS 318 V4 Reply (Call
In 1746) lookup LOOKUP | LOOKUP Status: NFS4ERR_NOENT
1748 3.801293956 192.168.2.152 → 192.168.2.150 NFS 266 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.exe
1749 3.801319051 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1748) lookup LOOKUP Status: NFS4ERR_NOENT
1750 3.801636441 192.168.2.152 → 192.168.2.150 NFS 266 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.lnk
1751 3.801659474 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1750) lookup LOOKUP Status: NFS4ERR_NOENT
1752 3.801981533 192.168.2.152 → 192.168.2.150 NFS 270 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.exe.lnk
1753 3.802003824 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1752) lookup LOOKUP Status: NFS4ERR_NOENT
1754 3.810270585 192.168.2.152 → 192.168.2.150 NFS 266 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.bat
1755 3.810345438 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1754) lookup LOOKUP Status: NFS4ERR_NOENT
1756 3.810768112 192.168.2.152 → 192.168.2.150 NFS 270 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.bat.exe
1757 3.810795216 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1756) lookup LOOKUP Status: NFS4ERR_NOENT
1758 3.811234114 192.168.2.152 → 192.168.2.150 NFS 270 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.bat.lnk
1759 3.811254882 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1758) lookup LOOKUP Status: NFS4ERR_NOENT
1760 3.811728762 192.168.2.152 → 192.168.2.150 NFS 274 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.bat.exe.lnk
1761 3.811802518 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1760) lookup LOOKUP Status: NFS4ERR_NOENT
1762 3.820028572 192.168.2.152 → 192.168.2.150 NFS 266 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.sh
1763 3.820128384 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1762) lookup LOOKUP Status: NFS4ERR_NOENT
1764 3.820495809 192.168.2.152 → 192.168.2.150 NFS 270 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.sh.exe
1765 3.820560543 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1764) lookup LOOKUP Status: NFS4ERR_NOENT
1766 3.821000559 192.168.2.152 → 192.168.2.150 NFS 270 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.sh.lnk
1767 3.821071034 192.168.2.150 → 192.168.2.152 NFS 162 V4 Reply (Call
In 1766) lookup LOOKUP Status: NFS4ERR_NOENT
1768 3.821519236 192.168.2.152 → 192.168.2.150 NFS 274 V4 Call lookup
LOOKUP DH: 0x2cf62c0d/sed.sh.exe.lnk

Also, searching for sed.sh.exe.lnk looks weird. Bug, or feature?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2023-09-01  4:23 Cygwin generates syscalls for *.lnk files on filesystems with native symlink support? Cedric Blancher
@ 2023-09-01 10:57 ` Corinna Vinschen
  2023-09-26  5:12   ` Cedric Blancher
  2023-12-18 12:04   ` Cedric Blancher
  0 siblings, 2 replies; 14+ messages in thread
From: Corinna Vinschen @ 2023-09-01 10:57 UTC (permalink / raw)
  To: cygwin

On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> Good morning!
> 
> During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> odd when I looked at the network traffic generated by one of our
> cluster nodes:
> It seems that for each call to a tool (i.e. starting "sed" from
> "bash") Cygwin searches for *.lnk files.
> 
> Is this correct even when the filesystem in question has native
> symlink support (e.g. NFS)?

Yes.  During file handling, Cygwin doesn't know what filesystem a
file is on until it could actually open the file and request file
and filesystem info from the open handle.  So if Cygwin couldn't open
"foo" because the NtCreateFile call returned with status
STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
STATUS_NO_SUCH_FILE, or one of the countless other status codes the
kernel (or the driver) might return in case a file doesn't exist,
it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
and try again.

Corinna

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2023-09-01 10:57 ` Corinna Vinschen
@ 2023-09-26  5:12   ` Cedric Blancher
  2023-12-18 12:04   ` Cedric Blancher
  1 sibling, 0 replies; 14+ messages in thread
From: Cedric Blancher @ 2023-09-26  5:12 UTC (permalink / raw)
  To: cygwin

On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > Good morning!
> >
> > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > odd when I looked at the network traffic generated by one of our
> > cluster nodes:
> > It seems that for each call to a tool (i.e. starting "sed" from
> > "bash") Cygwin searches for *.lnk files.
> >
> > Is this correct even when the filesystem in question has native
> > symlink support (e.g. NFS)?
>
> Yes.  During file handling, Cygwin doesn't know what filesystem a
> file is on until it could actually open the file and request file
> and filesystem info from the open handle.  So if Cygwin couldn't open
> "foo" because the NtCreateFile call returned with status
> STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> kernel (or the driver) might return in case a file doesn't exist,
> it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> and try again.

Our concern here is that this produces an absurd amount of extra
syscalls (6x more!!, compared to native POSIX), which hurts CIFS and
NFS performance.

The example with calling sed in a script is especially BAD, as trying
to execute "sed" in a bash shell script leads it try for sed.exe,
sed.sh, sed.bat, multiplied by looking for *.lnk files (i.e.
sed.exe.lnk, sed.sh.lnk, sed.bat.lnk).

And what finally made my brain go BOOM was the (IMO) absurd
combination like "sed.bat.exe.lnk"

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2023-09-01 10:57 ` Corinna Vinschen
  2023-09-26  5:12   ` Cedric Blancher
@ 2023-12-18 12:04   ` Cedric Blancher
  2024-01-08 13:56     ` Corinna Vinschen
  1 sibling, 1 reply; 14+ messages in thread
From: Cedric Blancher @ 2023-12-18 12:04 UTC (permalink / raw)
  To: cygwin

On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > Good morning!
> >
> > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > odd when I looked at the network traffic generated by one of our
> > cluster nodes:
> > It seems that for each call to a tool (i.e. starting "sed" from
> > "bash") Cygwin searches for *.lnk files.
> >
> > Is this correct even when the filesystem in question has native
> > symlink support (e.g. NFS)?
>
> Yes.  During file handling, Cygwin doesn't know what filesystem a
> file is on until it could actually open the file and request file
> and filesystem info from the open handle.

Why? If you have the path name you could lookup the (cached) mount
points, and determine the filesystem type. Same solution applies for
UNC paths, where you can easily lookup the filesystem type, and cache
it per-process or in Cygserver.

> So if Cygwin couldn't open
> "foo" because the NtCreateFile call returned with status
> STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> kernel (or the driver) might return in case a file doesn't exist,
> it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> and try again.

Can this machinery please be turned off via CYGWIN env var option? As
discussed in https://www.mail-archive.com/cygwin@cygwin.com/msg174547.html
this machinery causes very bad filesystem lookup performance, and it
would IMO a good idea to have an option to turn this off, and just
allow and expect native links (for NTFS, ReFS and NFS). Maybe CYGWIN
env var option winsymlinks_expect:native? winsymlinks_expect takes a :
seperated list which symlink types are to be expected.

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2023-12-18 12:04   ` Cedric Blancher
@ 2024-01-08 13:56     ` Corinna Vinschen
  2024-01-08 17:11       ` matthew patton
  2024-06-07 13:22       ` Roland Mainz
  0 siblings, 2 replies; 14+ messages in thread
From: Corinna Vinschen @ 2024-01-08 13:56 UTC (permalink / raw)
  To: cygwin

On Dec 18 13:04, Cedric Blancher via Cygwin wrote:
> On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
> <cygwin@cygwin.com> wrote:
> >
> > On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > > Good morning!
> > >
> > > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > > odd when I looked at the network traffic generated by one of our
> > > cluster nodes:
> > > It seems that for each call to a tool (i.e. starting "sed" from
> > > "bash") Cygwin searches for *.lnk files.
> > >
> > > Is this correct even when the filesystem in question has native
> > > symlink support (e.g. NFS)?
> >
> > Yes.  During file handling, Cygwin doesn't know what filesystem a
> > file is on until it could actually open the file and request file
> > and filesystem info from the open handle.
> 
> Why? If you have the path name you could lookup the (cached) mount
> points, and determine the filesystem type. Same solution applies for
> UNC paths, where you can easily lookup the filesystem type, and cache
> it per-process or in Cygserver.

No can do.

We *do* filesystem info caching, but this requires to be able to
open the file in the first place.  If the file exists, we're done, but
if not, we have to evaluate all symbolic links in the path first.
Simply reducing the path to the mount point and then translate it into
a Windows path doesn't cut it.

After the file (or its parent dir) could be opened, so we know the path
is valid, we can fetch the filesystem info right from the open file
handle.  At this point, we can shortcut the whole thing, reducing
the three necessary calls to kernel functions to only the first one
and skipping all the filesystem evaluation code.

> > So if Cygwin couldn't open
> > "foo" because the NtCreateFile call returned with status
> > STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> > STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> > kernel (or the driver) might return in case a file doesn't exist,
> > it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> > and try again.
> 
> Can this machinery please be turned off via CYGWIN env var option? As
> discussed in https://www.mail-archive.com/cygwin@cygwin.com/msg174547.html
> this machinery causes very bad filesystem lookup performance, and it
> would IMO a good idea to have an option to turn this off, and just
> allow and expect native links (for NTFS, ReFS and NFS). Maybe CYGWIN
> env var option winsymlinks_expect:native? winsymlinks_expect takes a :
> seperated list which symlink types are to be expected.

We won't add any more options to the CYGWIN env variable if it's not
necessary, and there's no proof at all that this is necessary in this
case.

If the file exists as specified, there will be no further check for
.exe or .lnk.  If it doesn't exist, *and* the return code from
the kernel is STATUS_OBJECT_PATH_NOT_FOUND (or any one of a
number of equivalent return values) we're done here and are going
to fall back to checking for symlinks in the leading path components.

If we got over that, the check for .exe is unavoidable.

So we're at the .lnk check. I'm pretty sure, if you try to measure the
performance it's not that bad. After all, at this point, the OS has
already cached the parent dir info and the NtOpenFile call returns
adequately snappy.

We might get away with dropping the .exe.lnk check.  In the past we
had the problem that these file existed, and you can still easily
create them, but theoretically nobody should do this.  Getting rid
of this check may be a minor backward incompatibility.

The additional problem with getting rid of .lnk is this: Even if you
switch to native symlinks exclusively (which may subtly break POSIX
compatibility, which is why Cygwin defaults to
IO_REPARSE_TAG_LX_SYMLINK) you won't switch to native symlinks
exclusively.  The reason is that FIFOs (and other device files but those
don;'t really matter since they were never fully utilized) are
implemented as Windows shortcuts.

Having said that.  I'm not opposed to fixes and improvements to the
path handling code.  It's long, winding and convoluted, and there's
certainly room for improvement.

We won't do that for Cygwin 3.5, though.  If all goes well, it will be
release in the next few weeks.

After that, when we cut the 3.5 branch, the main branch is open for new
stuff for 3.6.

However, before making generic suggestions, which may or may not be
helpful, it would be really great if you could have a look into the code
and try to understand why things are done the way they are done.  I'm
open to questions if you don't understand the code, but at least the
last couple of years, we tried hard to add comments everywhere to
explain things.  If you see a way to improve the code *without breaking
stuff*, I'm all ears, but it needs to be based on real, testable code
changes or at least POC code.

For instance: Getting rid of .lnk files isn't easy with backward
compatibility in mind.  Creating FIFOs differently isn't that much
trouble, but if a new DLL doesn't handle .lnk files it doesn't handle
the old FIFOs and already existing installations would break for no
apparent reason.  We don't want that.  Also, as long as we support MVFS,
we still need shortcut symlinks.  So we need either a transition period
or an incredibly bright idea how to fix it on the spot.


Corinna

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 13:56     ` Corinna Vinschen
@ 2024-01-08 17:11       ` matthew patton
  2024-01-08 18:11         ` Corinna Vinschen
  2024-06-07 13:22       ` Roland Mainz
  1 sibling, 1 reply; 14+ messages in thread
From: matthew patton @ 2024-01-08 17:11 UTC (permalink / raw)
  To: cygwin; +Cc: Corinna Vinschen

[-- Attachment #1: Type: text/plain, Size: 543 bytes --]

> For instance: Getting rid of .lnk files isn't easy with backward compatibility in mind.
screw backward compatability! :)Why carry around bandaids on bandaids for an OS that is 10 years out of support?
Obviously not ripe for 3.6 or maybe even 3.7, but at some point we should just put a stick in the sand and say this .lnk crap is dead and you have to either compile it back in with an IFDEF or a CYGWIN var. And while we're at it use OS-native links instead of Junctions. I'm not sure how that came about but Junctions are unhelpful.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 17:11       ` matthew patton
@ 2024-01-08 18:11         ` Corinna Vinschen
  2024-01-08 18:44           ` matthew patton
  0 siblings, 1 reply; 14+ messages in thread
From: Corinna Vinschen @ 2024-01-08 18:11 UTC (permalink / raw)
  To: cygwin

On Jan  8 17:11, matthew patton via Cygwin wrote:
> > For instance: Getting rid of .lnk files isn't easy with backward compatibility in mind.
> screw backward compatability! :)Why carry around bandaids on bandaids
> for an OS that is 10 years out of support?

We're using .lnk files for FIFOs. That's the compat concern, not some OS
from the past.

> Obviously not ripe for 3.6 or maybe even 3.7, but at some point we
> should just put a stick in the sand and say this .lnk crap is dead and
> you have to either compile it back in with an IFDEF or a CYGWIN var.

No ifdefs, and a Cygwin var setting unfortunately exists.

> And while we're at it use OS-native links instead of Junctions. I'm
> not sure how that came about but Junctions are unhelpful.

Cygwin does not create symlinks as junctions.  No idea where you got
that idea.


Corinna

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 18:11         ` Corinna Vinschen
@ 2024-01-08 18:44           ` matthew patton
  2024-01-08 19:05             ` Rainer Emrich
  2024-01-08 19:17             ` Jeffrey Altman
  0 siblings, 2 replies; 14+ messages in thread
From: matthew patton @ 2024-01-08 18:44 UTC (permalink / raw)
  To: cygwin; +Cc: Corinna Vinschen

[-- Attachment #1: Type: text/plain, Size: 983 bytes --]

> Cygwin does not create symlinks as junctions.  No idea where you got that idea.
$ echo $CYGWINwinsymlinks:nativestrict
$ /usr/bin/ln -s default.GGG6q test1
01/08/2024  01:24 PM    <JUNCTION>     test1 [...]Type=File
$ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
01/08/2024  01:25 PM    <JUNCTION>     test2.nocygwin [...]Type=File


# 'ln' is a wrapper function around 'mklink' since Cygwin does the wrong thing.https://github.com/tb3088/shell-environment/blob/ccf7aa161899c2c4ebe2d9e980e674bc726a3ef3/.functions_os.CYGWIN_NT#L10

$ ln default.GGG6q test3.mklinksymbolic link created for test3.mklink <<===>> default.GGG6q01/08/2024  01:25 PM    <SYMLINK>      test3.mklink [default.GGG6q]Type=.symlink

C:\Users\xxx>type default.GGG6qfoobar
C:\Users\xxx>type test1The file cannot be accessed by the system.
C:\Users\pattonma>type test2.nocygwinThe file cannot be accessed by the system.
C:\Users\xxx>type test3.mklinkfoobar

  

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 18:44           ` matthew patton
@ 2024-01-08 19:05             ` Rainer Emrich
  2024-01-08 19:17             ` Jeffrey Altman
  1 sibling, 0 replies; 14+ messages in thread
From: Rainer Emrich @ 2024-01-08 19:05 UTC (permalink / raw)
  To: matthew patton, cygwin; +Cc: Corinna Vinschen

Am 08.01.2024 um 19:44 schrieb matthew patton via Cygwin:
>> Cygwin does not create symlinks as junctions.  No idea where you got that idea.
> $ echo $CYGWINwinsymlinks:nativestrict
> $ /usr/bin/ln -s default.GGG6q test1
> 01/08/2024  01:24 PM    <JUNCTION>     test1 [...]Type=File
> $ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
> 01/08/2024  01:25 PM    <JUNCTION>     test2.nocygwin [...]Type=File
I have CYGWIN=winsymlinks:native
A JUNCTION is created if you try to ln to nonexisting file or directory.
If you ln to an existing directory you get a SYMLINKD and for an 
existing file you get SYMLINK, as expected!

Rainer


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 18:44           ` matthew patton
  2024-01-08 19:05             ` Rainer Emrich
@ 2024-01-08 19:17             ` Jeffrey Altman
  2024-01-08 19:21               ` Corinna Vinschen
  2024-01-08 19:57               ` matthew patton
  1 sibling, 2 replies; 14+ messages in thread
From: Jeffrey Altman @ 2024-01-08 19:17 UTC (permalink / raw)
  To: matthew patton, cygwin; +Cc: Corinna Vinschen

On 1/8/2024 1:44 PM, matthew patton via Cygwin wrote:
>> Cygwin does not create symlinks as junctions.  No idea where you got that idea.
> $ echo $CYGWINwinsymlinks:nativestrict
> $ /usr/bin/ln -s default.GGG6q test1
> 01/08/2024  01:24 PM    <JUNCTION>     test1 [...]Type=File
> $ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
> 01/08/2024  01:25 PM    <JUNCTION>     test2.nocygwin [...]Type=File

JUNCTIONS are a type of reparse point tag. Many tools report things as 
JUNCTIONS when they don't know what else to call it because JUNCTIONS 
were the only type of reparse tag commonly used.

This is the output of JP Software's Take Command for an AFS volume root 
directory

[\\afs\yfs]dir Directory of \\afs\yfs\* 5/26/2021 11:49 <DIR> . 
11/26/2019 14:25 <JUNCTION> amd64_linux26 
[your-file-system.com#amd64_linux26] 11/24/2019 20:29 <DIR> backups 
10/20/2022 10:43 <DIR> project 12/05/2011 10:14 <JUNCTION> public 
[#root.public] 9/06/2020 9:27 <DIR> service 7/26/2010 20:44 <JUNCTION> 
support [#root.support] 6/15/2011 11:40 <JUNCTION> test [#test.test] 
2/15/2012 8:49 <DIR> u 3/28/2023 12:50 <DIR> user 11/28/2011 17:01 
<SYMLINKD> usr [user] 12/10/2009 0:34 <JUNCTION> www [#root.www] 
12/13/2016 1:29 30 autorun.inf 6/07/2015 21:54 104 Desktop.ini 
11/26/2019 14:42 <SYMLINK> local [@sys\usr\local] 148 bytes in 3 files 
and 12 dirs 0 bytes free

The JUNCTIONS are AFS volume mount points using the Microsoft designated 
reparse point for OpenAFS. The SYMLINKD is a symlink reparse point to a 
directory using the NTFS symlink reparse point. The SYMLINK is a symlink 
reparse point to an object of unknown type (which Windows expects to be 
a file not a directory) using the NTFS symlink reparse point.

Here is the cygwin ls -l output

[\\afs\yfs]c:\tools\cygwin\bin\ls -l . total 38 -rw-r--r-- 1 jaltm 
mkpasswd 104 Jun 7 2015 Desktop.ini drwxr-xr-x 1 jaltm mkpasswd 0 Nov 26 
2019 amd64_linux26 -rw-r--r-- 1 jaltm mkpasswd 30 Dec 13 2016 
autorun.inf drwxr-xr-x 1 jaltm mkpasswd 0 Nov 24 2019 backups lrwxrwxrwx 
1 jaltm mkpasswd 14 Nov 26 2019 local -> @sys/usr/local drwxr-xr-x 1 
jaltm mkpasswd 0 Oct 20 2022 project drwxr-xr-x 1 jaltm mkpasswd 0 May 
21 2023 public drwxr-xr-x 1 jaltm mkpasswd 0 Sep 6 2020 service 
drwxr-xr-x 1 jaltm mkpasswd 0 Nov 17 2015 support drwxr-xr-x 1 jaltm 
mkpasswd 0 May 26 2019 test drwxr-xr-x 1 jaltm mkpasswd 0 Feb 15 2012 u 
drwxr-xr-x 1 jaltm mkpasswd 0 Mar 28 2023 user lrwxrwxrwx 1 jaltm 
mkpasswd 4 Nov 28 2011 usr -> user drwxr-xr-x 1 jaltm mkpasswd 0 Jul 10 
2017 www

As part of this reply I will note that the NTFS symlinks differ from 
POSIX symlinks in significant ways

 1. A pre-existing file system object is required in order to attach a
    reparse tag
 2. The type of the target must be known when the reparse tag is applied
    to a pre-existing file system object
 3. The reparse tag may be removed and replaced any number of times
    without deleting the pre-existing object to which it is attached.  
    Whereas a POSIX symlink inode target cannot be altered once
    created.  The inode must be deleted and replaced.
 4. The Windows file APIs do not behave as many applications expect them
    to when a symlink reparse tag is present.  For example, opening a
    file handle will traverse the reparse tag and open the target but
    the file info api when given the same path will return the
    information belonging to the object on which the reparse tag was
    applied.   This breaks many applications such as the java runtime
    among others.

Jeffrey Altman



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 19:17             ` Jeffrey Altman
@ 2024-01-08 19:21               ` Corinna Vinschen
  2024-01-08 19:57               ` matthew patton
  1 sibling, 0 replies; 14+ messages in thread
From: Corinna Vinschen @ 2024-01-08 19:21 UTC (permalink / raw)
  To: cygwin

On Jan  8 14:17, Jeffrey Altman via Cygwin wrote:
> On 1/8/2024 1:44 PM, matthew patton via Cygwin wrote:
> > > Cygwin does not create symlinks as junctions.  No idea where you got that idea.
> > $ echo $CYGWINwinsymlinks:nativestrict
> > $ /usr/bin/ln -s default.GGG6q test1
> > 01/08/2024  01:24 PM    <JUNCTION>     test1 [...]Type=File
> > $ (unset CYGWIN; /usr/bin/ln -s default.GGG6q test2.nocygwin)
> > 01/08/2024  01:25 PM    <JUNCTION>     test2.nocygwin [...]Type=File
> 
> JUNCTIONS are a type of reparse point tag. Many tools report things as
> JUNCTIONS when they don't know what else to call it because JUNCTIONS were
> the only type of reparse tag commonly used.
> [...]
> As part of this reply I will note that the NTFS symlinks differ from POSIX
> symlinks in significant ways
> 
> 1. A pre-existing file system object is required in order to attach a
>    reparse tag
> 2. The type of the target must be known when the reparse tag is applied
>    to a pre-existing file system object
> 3. The reparse tag may be removed and replaced any number of times
>    without deleting the pre-existing object to which it is attached.  
>    Whereas a POSIX symlink inode target cannot be altered once
>    created.  The inode must be deleted and replaced.
> 4. The Windows file APIs do not behave as many applications expect them
>    to when a symlink reparse tag is present.  For example, opening a
>    file handle will traverse the reparse tag and open the target but
>    the file info api when given the same path will return the
>    information belonging to the object on which the reparse tag was
>    applied.   This breaks many applications such as the java runtime
>    among others.
> 
> Jeffrey Altman

Thanks, Jeffrey.  I couldn't have expressed it so nicely.  Very well
summarized.


Corinna

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 19:17             ` Jeffrey Altman
  2024-01-08 19:21               ` Corinna Vinschen
@ 2024-01-08 19:57               ` matthew patton
  2024-01-08 20:27                 ` Brian Inglis
  1 sibling, 1 reply; 14+ messages in thread
From: matthew patton @ 2024-01-08 19:57 UTC (permalink / raw)
  To: cygwin, Jeffrey Altman; +Cc: Corinna Vinschen

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

> This breaks many applications such as the java runtime among others.
In any event "unreadable files" is a problem all over the place if I use Cygwin's /usr/bin/ln to create links. That's why I was forced to write a wrapper. Even if 'JUNCTION' is false/misleading as to the root cause, plenty of 'DOS' programs, like the AWS CLI and Golang have major problems with the default behavior.
For a "compatibility" environment to spawn a slew of unusable filesystem artifacts is a MAJOR bug, IMO.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 19:57               ` matthew patton
@ 2024-01-08 20:27                 ` Brian Inglis
  0 siblings, 0 replies; 14+ messages in thread
From: Brian Inglis @ 2024-01-08 20:27 UTC (permalink / raw)
  To: cygwin

On 2024-01-08 12:57, matthew patton via Cygwin wrote:
>> This breaks many applications such as the java runtime among others.
> In any event "unreadable files" is a problem all over the place if I use Cygwin's /usr/bin/ln to create links. That's why I was forced to write a wrapper. Even if 'JUNCTION' is false/misleading as to the root cause, plenty of 'DOS' programs, like the AWS CLI and Golang have major problems with the default behavior.
> For a "compatibility" environment to spawn a slew of unusable filesystem artifacts is a MAJOR bug, IMO.

That is why you have env var settings to make things more compatible.
Read the docs and make the best choice for your situation.
Cygwin is compatible with MS Windows but expecting MS Windows to be compatible 
with anything else is expecting too much.
Just avoid using non-POSIX interfaces and you will be much better off.
You can then be confident that anything you do or have can easily be moved to 
another POSIX compatible system and used with only minor differences, unless you 
go crazy and use another proprietary system!

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Cygwin generates syscalls for *.lnk files on filesystems with native symlink support?
  2024-01-08 13:56     ` Corinna Vinschen
  2024-01-08 17:11       ` matthew patton
@ 2024-06-07 13:22       ` Roland Mainz
  1 sibling, 0 replies; 14+ messages in thread
From: Roland Mainz @ 2024-06-07 13:22 UTC (permalink / raw)
  To: cygwin

On Mon, Jan 8, 2024 at 2:56 PM Corinna Vinschen via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Dec 18 13:04, Cedric Blancher via Cygwin wrote:
> > On Fri, 1 Sept 2023 at 13:00, Corinna Vinschen via Cygwin
> > <cygwin@cygwin.com> wrote:
> > >
> > > On Sep  1 06:23, Cedric Blancher via Cygwin wrote:
> > > > Good morning!
> > > >
> > > > During a Cygwin 3.4.8-1.x86_64 debugging session I noticed something
> > > > odd when I looked at the network traffic generated by one of our
> > > > cluster nodes:
> > > > It seems that for each call to a tool (i.e. starting "sed" from
> > > > "bash") Cygwin searches for *.lnk files.
> > > >
> > > > Is this correct even when the filesystem in question has native
> > > > symlink support (e.g. NFS)?
> > >
> > > Yes.  During file handling, Cygwin doesn't know what filesystem a
> > > file is on until it could actually open the file and request file
> > > and filesystem info from the open handle.
> >
> > Why? If you have the path name you could lookup the (cached) mount
> > points, and determine the filesystem type. Same solution applies for
> > UNC paths, where you can easily lookup the filesystem type, and cache
> > it per-process or in Cygserver.
>
> No can do.
>
> We *do* filesystem info caching, but this requires to be able to
> open the file in the first place.  If the file exists, we're done, but
> if not, we have to evaluate all symbolic links in the path first.
> Simply reducing the path to the mount point and then translate it into
> a Windows path doesn't cut it.
>
> After the file (or its parent dir) could be opened, so we know the path
> is valid, we can fetch the filesystem info right from the open file
> handle.  At this point, we can shortcut the whole thing, reducing
> the three necessary calls to kernel functions to only the first one
> and skipping all the filesystem evaluation code.
>
> > > So if Cygwin couldn't open
> > > "foo" because the NtCreateFile call returned with status
> > > STATUS_OBJECT_PATH_NOT_FOUND or STATUS_OBJECT_NAME_NOT_FOUND, or
> > > STATUS_NO_SUCH_FILE, or one of the countless other status codes the
> > > kernel (or the driver) might return in case a file doesn't exist,
> > > it will tack on .lnk and .exe and, for historical reasons, .exe.lnk,
> > > and try again.
> >
> > Can this machinery please be turned off via CYGWIN env var option? As
> > discussed in https://www.mail-archive.com/cygwin@cygwin.com/msg174547.html
> > this machinery causes very bad filesystem lookup performance, and it
> > would IMO a good idea to have an option to turn this off, and just
> > allow and expect native links (for NTFS, ReFS and NFS). Maybe CYGWIN
> > env var option winsymlinks_expect:native? winsymlinks_expect takes a :
> > seperated list which symlink types are to be expected.
>
> We won't add any more options to the CYGWIN env variable if it's not
> necessary, and there's no proof at all that this is necessary in this
> case.
>
> If the file exists as specified, there will be no further check for
> .exe or .lnk.  If it doesn't exist, *and* the return code from
> the kernel is STATUS_OBJECT_PATH_NOT_FOUND (or any one of a
> number of equivalent return values) we're done here and are going
> to fall back to checking for symlinks in the leading path components.
>
> If we got over that, the check for .exe is unavoidable.
[snip]

What about doing lookups in parallel, e.g. use |NtCreateFile()| with
an |IO_STATUS_BLOCK|, and do lookups for *.exe, *.lnk etc. in parallel
?
That might not help for all filesystem drivers, but some of them are
multithreaded (e.g. SMB, ms-nfs41-client, ...) and would greatly
benefit from that since the requests would go async&&parallel over the
network, instead of sequentially like it is now.

----

Bye,
Roland
-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz@nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2024-06-07 13:22 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-01  4:23 Cygwin generates syscalls for *.lnk files on filesystems with native symlink support? Cedric Blancher
2023-09-01 10:57 ` Corinna Vinschen
2023-09-26  5:12   ` Cedric Blancher
2023-12-18 12:04   ` Cedric Blancher
2024-01-08 13:56     ` Corinna Vinschen
2024-01-08 17:11       ` matthew patton
2024-01-08 18:11         ` Corinna Vinschen
2024-01-08 18:44           ` matthew patton
2024-01-08 19:05             ` Rainer Emrich
2024-01-08 19:17             ` Jeffrey Altman
2024-01-08 19:21               ` Corinna Vinschen
2024-01-08 19:57               ` matthew patton
2024-01-08 20:27                 ` Brian Inglis
2024-06-07 13:22       ` Roland Mainz

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).