Re: rfe: CYGWIN fslinktypes option? Re: Catastrophic Cygwin find . -ls, grep performance on samba share compared to WSL&Linux

[Kaz Kylheku <kaz@kylheku.com>]

On 2023-12-21 04:16, Martin Wege via Cygwin wrote:
> On Wed, Dec 20, 2023 at 6:21 PM Kaz Kylheku via Cygwin
> <cygwin@cygwin.com> wrote:
>>
>> On 2023-12-17 22:22, Dan Shelton via Cygwin wrote:
>> > It would be nice if someone from the Cygwin authors could assist me in
>> > figuring out why this happens.
>>
>> Cygwin is famously slow; this is nothing new. We are grateful
>> for Cygwin because it makes stuff work at all; if it were blazing
>> fast that would be a bonus.
>>
>> E.g. git operations (clone, rebase, ...); ./configure scripts; ...: all
>> run like molasses.
>>
>> The following is just my fast and loose opinion, shot from the hip,
>> and possibly off or wrong, but it likely has to do with the layering.
>> Cygwin's core API is based on a C library called Newlib. Cygwin bolts
>> Newlib to Windows by means of an additional shim below Newlib that
>> is based on C++ objects, where there is path munging going on and such,
>> and that's where the Win32 calls get made. It's an additional abstraction.
> 
> I disagree with that. Ok, part of that is that the layering causes
> more memory allocations and copies, but this is not the root cause.

I seem to recall that most operations that take a path argument have
to convert the path from Cygwin to Win32, and I think that also involves
going from 8 bit to UTF-16 also. That's gotta hurt a bit.
 
> The root cause is IMO the extra Win32 syscalls (>= 3 per file lookup,
> compared to 1 on Linux) to lookup the *.lnk and *.exe.lnk files on
> filesystems which have native link support (NTFS, ReFS, SMBFS, NFS).
> On SMBFS and NFS it hurts the most, because access latency is the
> highest for networked filesystems.

Could some intelligent caching be added there? (Discussion of
associated invalidation problem in 3... 2.... 1... )

Can you discuss more details, so people don't have to dive into code
to understand it? If we are accessing some file "foo", the application
or user may actually be referring to a "foo.lnk" link. But in the
happy case that "foo" exists, why would we bother looking for "foo.lnk"?

If "foo" does not exist, but "foo.lnk" does, that could probably be
cached, so that next time "foo" is accessed, we go straight for "foo.lnk",
and keep using that while it exists.

If someone has both "foo" and "foo.lnk" in the same directory,
that's a bit of a degenerate case; how important is it to be "correct",
anyway.

> So my proposal would be to add an option ('fslinktypes') to the CYGWIN
> environment variable to define which types of links are supported:
> default 'all'. which is an shortcut for 'native,lnk,lnkexe'.
> So in case people do not want 'lnk' link support they just add
> CYGWIN+=' fslinktypes:native' to env, to turn off support for
> lnk/lnk.exe style links, and be happy.

So this complements the winsymlinks option? winsymlinks has to do
with how the Cygwin DLL creates symbolic links, whereas this has to do
with what objects are recognized as links.

The implementation would probably want to compare fslinktypes
and winsymlinks to make sure they are harmonized together;
if winsymlinks tells Cygwin to make .lnk files, but then fslinktypes
banishes them, that's something diagnosable somehow.