Most git executables are hard links to git.exe?

Most git executables are hard links to git.exe?

[Jim Garrison]

Git comes with over 100 executables, mostly in /usr/libexec/git-core,
that all appear to be *hard* links to /bin/git, in both Cygwin and
Windows. The Windows fsutil command shows they're all hard linked:

     C:\Users\jim>fsutil hardlink list "c:\cygwin64\bin\git.exe"
     \cygwin64\usr\libexec\git-core\git-add.exe
     \cygwin64\bin\git-upload-archive.exe
     \cygwin64\bin\git-upload-pack.exe
     \cygwin64\bin\git.exe
     \cygwin64\bin\git-receive-pack.exe
     \cygwin64\usr\libexec\git-core\git-am.exe
     \cygwin64\usr\libexec\git-core\git-annotate.exe
     [snip]

This is confirmed in a Cygwin bash session showing they all have the
same inode number.

     jim@HOME10 /usr/libexec/git-core
     $ ls -il
     total 492832
      34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 git.exe
      34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 
git-add.exe
      34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 
git-am.exe
      34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 
git-annotate.exe
      34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 
git-apply.exe
      [snip]

This causes other Windoze programs not aware of hard links (such as even
Windows Explorer :-( and Retrospect backup) to treat them as actual
files. In the case of Retrospect (and I suspect most backup programs)
this results in backing up 400MB instead of 3MB.

I'm curious to know if there's a specific reason for this implementation
that would make it the choice over symbolic links.

-- 
Jim Garrison
jhg@acm.org

Re: Most git executables are hard links to git.exe?

[Brian Inglis]

On 2023-07-21 14:59, Jim Garrison via Cygwin wrote:
> Git comes with over 100 executables, mostly in /usr/libexec/git-core,
> that all appear to be *hard* links to /bin/git, in both Cygwin and
> Windows. The Windows fsutil command shows they're all hard linked:
> 
>      C:\Users\jim>fsutil hardlink list "c:\cygwin64\bin\git.exe"
>      \cygwin64\usr\libexec\git-core\git-add.exe
>      \cygwin64\bin\git-upload-archive.exe
>      \cygwin64\bin\git-upload-pack.exe
>      \cygwin64\bin\git.exe
>      \cygwin64\bin\git-receive-pack.exe
>      \cygwin64\usr\libexec\git-core\git-am.exe
>      \cygwin64\usr\libexec\git-core\git-annotate.exe
>      [snip]
> 
> This is confirmed in a Cygwin bash session showing they all have the
> same inode number.
> 
>      jim@HOME10 /usr/libexec/git-core
>      $ ls -il
>      total 492832
>       34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 git.exe
>       34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 git-add.exe
>       34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 git-am.exe
>       34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 
> git-annotate.exe
>       34621422135881770 -rwxr-xr-x 142 jim None 3474451 Dec 12  2022 git-apply.exe
>       [snip]
> 
> This causes other Windoze programs not aware of hard links (such as even
> Windows Explorer :-( and Retrospect backup) to treat them as actual
> files. In the case of Retrospect (and I suspect most backup programs)
> this results in backing up 400MB instead of 3MB.
> 
> I'm curious to know if there's a specific reason for this implementation
> that would make it the choice over symbolic links.

For the same reason you are complaining about backups not taking hardlinks into 
account: to avoid distributing 400MB instead of 3MB.

Cygwin backup utilities should be able to deal with these e.g. rsync -H, 
--hard-links, although it appears xcopy and robocopy may not under Windows 10; 
don't know about other utilities or Windows 11.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: Most git executables are hard links to git.exe?

[Jim Garrison]

On 07/21/23 14:52, Brian Inglis wrote:
> On 2023-07-21 14:59, Jim Garrison via Cygwin wrote:
>> Git comes with over 100 executables, mostly in /usr/libexec/git-core,
>> that all appear to be *hard* links to /bin/git, in both Cygwin and
>> Windows. The Windows fsutil command shows they're all hard linked:
[snip]
>> I'm curious to know if there's a specific reason for this implementation
>> that would make it the choice over symbolic links.
> 
> For the same reason you are complaining about backups not taking 
> hardlinks into account: to avoid distributing 400MB instead of 3MB.
> 
> Cygwin backup utilities should be able to deal with these e.g. rsync -H, 
> --hard-links, although it appears xcopy and robocopy may not under 
> Windows 10; don't know about other utilities or Windows 11.

But why not use symbolic links to accomplish the same thing?

-- 
Jim Garrison
jhg@acm.org

Re: Most git executables are hard links to git.exe?

[Larry]

On 7/21/23 17:54, Jim Garrison via Cygwin wrote:
> On 07/21/23 14:52, Brian Inglis wrote:
>> On 2023-07-21 14:59, Jim Garrison via Cygwin wrote:
>>> Git comes with over 100 executables, mostly in /usr/libexec/git-core,
>>> that all appear to be *hard* links to /bin/git, in both Cygwin and
>>> Windows. The Windows fsutil command shows they're all hard linked:
> [snip]
>>> I'm curious to know if there's a specific reason for this implementation
>>> that would make it the choice over symbolic links.
>>
>> For the same reason you are complaining about backups not taking hardlinks 
>> into account: to avoid distributing 400MB instead of 3MB.
>>
>> Cygwin backup utilities should be able to deal with these e.g. rsync -H, 
>> --hard-links, although it appears xcopy and robocopy may not under Windows 
>> 10; don't know about other utilities or Windows 11.
> 
> But why not use symbolic links to accomplish the same thing?

If you're wondering what the limitations and complications of symbolic links
on Windows are, I'd recommend reading item 5.8 in the FAQ - "How do symbolic
links work?" (https://cygwin.com/faq.html#faq.api.symlinks)  Keep in
mind that this applies to symbolic links in the Cygwin world.  As long as
that's all you care about, then the various trade-offs and limitations can
certainly be mitigated in your personal environment to the extent that you
might actually prefer and choose to use Cygwin symbolic links over hard
links for your needs.  But in the general case, they really don't compare to
simplicity of hard links which are fully supported in Windows on NTFS file
systems along with all Cygwin tools and transparently degrade to duplicated
files on file systems and with software that doesn't support them.  In these
degraded use cases, they take up more space because the link semantics
aren't maintained but they are still 100% valid and useful.  The same cannot 
be said of symbolic links.

Re: Most git executables are hard links to git.exe?

[Adam Dinwoodie]

On Fri, 21 Jul 2023 at 22:54, Jim Garrison via Cygwin wrote:
>
> On 07/21/23 14:52, Brian Inglis wrote:
> > On 2023-07-21 14:59, Jim Garrison via Cygwin wrote:
> >> Git comes with over 100 executables, mostly in /usr/libexec/git-core,
> >> that all appear to be *hard* links to /bin/git, in both Cygwin and
> >> Windows. The Windows fsutil command shows they're all hard linked:
> [snip]
> >> I'm curious to know if there's a specific reason for this implementation
> >> that would make it the choice over symbolic links.
> >
> > For the same reason you are complaining about backups not taking
> > hardlinks into account: to avoid distributing 400MB instead of 3MB.
> >
> > Cygwin backup utilities should be able to deal with these e.g. rsync -H,
> > --hard-links, although it appears xcopy and robocopy may not under
> > Windows 10; don't know about other utilities or Windows 11.
>
> But why not use symbolic links to accomplish the same thing?

A few reasons off the top of my head:

- This is what the Git build tooling does out of the box. Minimising
the number of changes we're making as a downstream packager makes my
life easier as package maintainer.

- This is what happens on *nix systems, and Cygwin generally
prioritises matching function with *nix systems over interoperability
with Windows tools; if you want interoperability with Windows tools,
you might be better off with Git for Windows. That's not trying to
brush you off; the reason Cygwin Git and Git for Windows both exist is
that they're both serving different user needs.

- As others have said, Windows in general has good support for
hardlinks, while it has no inherent support for Cygwin's symlinks.
That means a Windows application would need to be aware of Cygwin to
have any chance of usefully interacting with those files if they were
symlinks, whereas a Windows application doesn't need to be aware of
Cygwin at all to be able to handle hardlinks, it only needs to know
how to handle hardlinks on Windows.

- Although I've not measured it, I expect there's a small runtime cost
from using symlinks over hardlinks. Cygwin's Git is already slow, for
a variety of difficult-to-solve reasons, and I'm reluctant to do
anything that might make that worse.

- Inertia. The current situation works well for most people, and
changing things takes effort and risks breaking other folks' use
cases.

I do acknowledge that while many Windows tools *could* handle
hardlinks, many don't. I'm not at all surprised that some backup
utilities don't handle them well and back up each file separately. I
think switching to using symlinks for Cygwin's executables is the
wrong solution, though.

Instead, I'd suggest (a) finding a backup tool that can handle
hardlinks, (b) finding a backup tool that uses compression so the
"duplicate" data gets deduplicated as part of the backup process, (c)
not backing up most of Cygwin's /usr directory in the first place – in
most cases I wouldn't expect there to be anything in that folder that
couldn't be readily recovered elsewhere anyway – or (d) switching to a
disk imaging backup system rather than a file-based one if it's really
important that you have everything on disk ready to restore.

Hopefully that's all useful and/or interesting, even if it's not the
answer you were hoping for!

Re: Most git executables are hard links to git.exe?

[Jim Garrison]

On 07/22/23 10:33, Adam Dinwoodie wrote:
> On Fri, 21 Jul 2023 at 22:54, Jim Garrison via Cygwin wrote:
>>
>> On 07/21/23 14:52, Brian Inglis wrote:
>>> On 2023-07-21 14:59, Jim Garrison via Cygwin wrote:
>>>> Git comes with over 100 executables, mostly in /usr/libexec/git-core,
>>>> that all appear to be *hard* links to /bin/git, in both Cygwin and
>>>> Windows. The Windows fsutil command shows they're all hard linked:
>> [snip]
>>>> I'm curious to know if there's a specific reason for this implementation
>>>> that would make it the choice over symbolic links.
>>>
>>> For the same reason you are complaining about backups not taking
>>> hardlinks into account: to avoid distributing 400MB instead of 3MB.
>>>
>>> Cygwin backup utilities should be able to deal with these e.g. rsync -H,
>>> --hard-links, although it appears xcopy and robocopy may not under
>>> Windows 10; don't know about other utilities or Windows 11.
>>
>> But why not use symbolic links to accomplish the same thing?
> 
> A few reasons off the top of my head:
> 
> - This is what the Git build tooling does out of the box. Minimising
> the number of changes we're making as a downstream packager makes my
> life easier as package maintainer.
> 
[snip]
> Hopefully that's all useful and/or interesting, even if it's not the
> answer you were hoping for!

Yes, that's very useful and now I understand and appreciate all the
good reasons for hard links.  Thanks!

-- 
Jim Garrison
jhg@acm.org

Re: Most git executables are hard links to git.exe?

[L A Walsh]

On 2023/07/22 10:35, Jim Garrison via Cygwin wrote:
> On 07/22/23 10:33, Adam Dinwoodie wrote:
>   
>> On Fri, 21 Jul 2023 at 22:54, Jim Garrison via Cygwin wrote:
>>     
>>> On 07/21/23 14:52, Brian Inglis wrote:
>>>       
>>>> On 2023-07-21 14:59, Jim Garrison via Cygwin wrote:
>>>>         
>>>>> Git comes with over 100 executables, mostly in /usr/libexec/git-core,
>>>>> that all appear to be *hard* links to /bin/git, in both Cygwin and
>>>>> Windows. The Windows fsutil command shows they're all hard linked:
>>>>>           
>>> [snip]
>>>       
>>>>> I'm curious to know if there's a specific reason for this implementation
>>>>> that would make it the choice over symbolic links.
>>>>>           
The hardlink implementation on windows is very similar to the
implementation on linux.  I'm pretty sure that utils that want to save
on space will look at the inode-number and notice that the hardlinked files
all have the same inode-number (windows has a similar concept though it is
called something else).

On linux, utils that are ignorant of inode numbers, will see hardlinked 
files
as separate files -- just as windows does.

The symlink files will break if their targets move (same on lin+win).

Re: Most git executables are hard links to git.exe?

[Andrey Repin]

Greetings, Jim Garrison via Cygwin!

> On 07/21/23 14:52, Brian Inglis wrote:
>> On 2023-07-21 14:59, Jim Garrison via Cygwin wrote:
>>> Git comes with over 100 executables, mostly in /usr/libexec/git-core,
>>> that all appear to be *hard* links to /bin/git, in both Cygwin and
>>> Windows. The Windows fsutil command shows they're all hard linked:
> [snip]
>>> I'm curious to know if there's a specific reason for this implementation
>>> that would make it the choice over symbolic links.
>> > For the same reason you are complaining about backups not taking > hardlinks into account: to avoid distributing 400MB instead of 3MB.
>> > Cygwin backup utilities should be able to deal with these e.g. rsync -H, > --hard-links, although it appears xcopy and robocopy may not under > Windows 10; don't know about other utilities or Windows 11.

> But why not use symbolic links to accomplish the same thing?

Adding to the bullet list:
cmd's handling of symbolik links is such that given a symlink path, it would
actually execute the symlink target, thoroughly ruining the executable name
and associated CWD. Making them unusable as command processors in general.


-- 
With best regards,
Andrey Repin
Monday, July 31, 2023 16:33:49

Sorry for my terrible english...