native symlinks and non-existent targets

native symlinks and non-existent targets

[clayne]

Since I recently sent an email about symlinks and cygdrive mounts, I
figured I'd report another issue that's plagued me over the years and
that I know others have reported in the past: You can't create native
symlinks to non-existent targets and this causes a bunch of issues when
rsyncing directories containing symlinks unless one does a multi-pass
approach which takes special precautions to sync all the non-symlink
contents first and then syncs the symlinks right after (note: this also
has its own problems with links to links).

Example:

clayne@sv590:/tmp/link-test $ ls -la
total 32
drwxr-xr-x 1 clayne None 0 Apr 24 14:13 .
drwxrwxrwt 1 clayne None 0 Apr 24 14:34 ..
lrwxrwxrwx 7 clayne None 3 Apr 24 14:11 _foo -> foo
-rw-r--r-- 5 clayne None 0 Apr 24 14:11 foo
lrwxrwxrwx 4 clayne None 3 Apr 24 14:13 foo-ln -> foo

clayne@sv590:/tmp/link-test $ rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/
sending incremental file list
created directory /tmp/link-test-sync-test
./
rsync: [generator] symlink "/tmp/link-test-sync-test/_foo" -> "foo" failed: No such file or directory (2)
rsync: [generator] symlink "/tmp/link-test-sync-test/foo-ln" -> "foo" failed: No such file or directory (2)
foo
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=1/4)

sent 178 bytes  received 89 bytes  534.00 bytes/sec
total size is 6  speedup is 0.02
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]

clayne@sv590:/tmp/link-test $ rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/
sending incremental file list
_foo -> foo
foo-ln -> foo

sent 134 bytes  received 18 bytes  304.00 bytes/sec
total size is 6  speedup is 0.04

This only works in a straight-forward manner when using non-native symlinks
(which can also _change_ the symlinks such that they're broken/unusable
outside of cygwin):

clayne@sv590:/tmp/link-test $ rm -rf /tmp/link-test-sync-test
clayne@sv590:/tmp/link-test $ CYGWIN= rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/
sending incremental file list
created directory /tmp/link-test-sync-test
./
_foo -> foo
foo
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=1/4)
foo-ln -> foo

sent 184 bytes  received 95 bytes  558.00 bytes/sec
total size is 6  speedup is 0.02

Or another very simple test-case not involving rsync:

clayne@sv590:/tmp/link-test $ ln -s this-does-not-exist some-link
ln: failed to create symbolic link 'some-link': No such file or directory

Relevant sections of an strace for the above:

  102   83054 [main] ln 1379 symlink_info::check: 0x0 = NtCreateFile (\??\C:\cygwin64\tmp\link-test)
  114   83168 [main] ln 1379 symlink_info::check: not a symlink
   91   83259 [main] ln 1379 symlink_info::check: 0 = symlink.check(C:\cygwin64\tmp\link-test, 0x7FFFFB140) (mount_flags 0x30008, path_flags 0x0)
   92   83351 [main] ln 1379 path_conv::check: this->path(C:\cygwin64\tmp\link-test\this-does-not-exist), has_acls(1)
   91   83442 [main] ln 1379 seterrno_from_win_error: /usr/src/debug/cygwin-3.5.3-1/winsup/cygwin/path.cc:2063 windows error 2
   90   83532 [main] ln 1379 geterrno_from_win_error: windows error 2 == errno 2
   83   83615 [main] ln 1379 symlink_worker: -1 = symlink_worker(this-does-not-exist, /tmp/link-test/some-link, 0)

However, you _can_ do this:

clayne@sv590:/tmp/link-test $ tmp="$(mktemp)" && ln -s "$tmp" some-link && rm -f "$tmp" && ls -la some-link
lrwxrwxrwx 1 clayne None 19 Apr 24 14:58 some-link -> /tmp/tmp.o7xpJxaqig

And here's a case showing where "hard-linked" symlinks work with
non-existent targets:

clayne@sv590:/tmp/link-test $ find -type l -print0 | rsync -avSHP --files-from=- --from0 --link-dest=/tmp/link-test /tmp/link-test /tmp/link-test-link-dest
building file list ... 
3 files to consider
created directory /tmp/link-test-link-dest

sent 113 bytes  received 59 bytes  344.00 bytes/sec
total size is 6  speedup is 0.03

clayne@sv590:/tmp/link-test $ ls -l /tmp/link-test-link-dest
total 0
lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 _foo -> foo
lrwxrwxrwx 5 clayne None 3 Apr 24 14:13 foo-ln -> foo

clayne@sv590:/tmp/link-test $ ls -lai _foo /tmp/link-test-link-dest/_foo
48413695994484407 lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 /tmp/link-test-link-dest/_foo -> foo
48413695994484407 lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 _foo -> foo

This only works because the symlinks are being straight up cloned
and not being newly created on the destination side. This "works" if
one is doing a --link-dest of an entire directory to essentially create
a hard-linked copy but anything outside of that use case will still
require multiple runs.

Based on past threads I've read I believe the issue is actually with
windows not allowing a symlink to be created with a non-existent target,
but I do know windows does not care if you break a link after the fact.
For the non-existent target case, is there any way we could just hack-fix
this or workaround it at the cygwin layer by create a symlink to a temp
file representing the eventual target (I realize this is way easier said
than done) and then immediately removing the temp file? Obviously there
are some atomicity issues here because one might not be able to just
create an in-situ file without creating the directory hierarchy it needs
to live in first. On the other hand it doesn't seem ideal to require an
rsync to run multiple times nor would it make sense to require every
application or utility involved with symlinks to somehow know of this
cygwin/windows specific issue (breaks abstraction). There has got to be
a better way of handling this issue such that it's transparent to the
caller.

-cl

Re: native symlinks and non-existent targets

[clayne]

On Wed, Apr 24, 2024 at 10:11:52PM +0000, Christopher Layne via Cygwin wrote:
> Based on past threads I've read I believe the issue is actually with
> windows not allowing a symlink to be created with a non-existent target,
> but I do know windows does not care if you break a link after the fact.

Actually, after referring to some microsoft documentation, is this even
true?

https://learn.microsoft.com/en-us/windows/win32/fileio/symbolic-link-programming-considerations
---
Programming Considerations (Local File Systems)

Article 03/03/2021 5 contributors

Keep the following programming considerations in mind when working with symbolic links:

* Symbolic links can point to a non-existent target.
* When creating a symbolic link, the operating system does not check to see if the target exists.
* If an application tries to open a non-existent target, ERROR_FILE_NOT_FOUND is returned.
* Symbolic links are reparse points. For more information, see Determining Whether a Directory Is a Mounted Folder.
* There is a maximum of 63 reparse points (and therefore symbolic links) allowed in a particular path.
  (Windows Server 2003 and Windows XP: There is a limit of 31 reparse points on any given path.)
---

If it isn't true then this seems trivial to fix.

-cl

Re: native symlinks and non-existent targets

[Jon Turney]

On 24/04/2024 23:36, Christopher Layne via Cygwin wrote:
> On Wed, Apr 24, 2024 at 10:11:52PM +0000, Christopher Layne via Cygwin wrote:
>> Based on past threads I've read I believe the issue is actually with
>> windows not allowing a symlink to be created with a non-existent target,
>> but I do know windows does not care if you break a link after the fact.
> 
> Actually, after referring to some microsoft documentation, is this even
> true?

No.

However, native symlinks do record the type (file or directory) of the 
destination, and (absent special knowledge) this cannot be determined if 
the destination doesn't exist (yet).

If I recall correctly, Cygwin doesn't care if the type recorded in the 
symlink is incorrect, but some parts of Windows do...

> If it isn't true then this seems trivial to fix.

This assertion is trivially disproven by the lack of a patch attached. :)

Re: native symlinks and non-existent targets

[Csaba Ráduly]

On 25/04/2024 15:15, Jon Turney via Cygwin wrote:
> On 24/04/2024 23:36, Christopher Layne via Cygwin wrote:
> [...]
>> If it isn't true then this seems trivial to fix.
>
> This assertion is trivially disproven by the lack of a patch attached. :)
>
>
I don't think this is worth a gold star, but a jester's cap is surely 
warranted :)

Csaba

-- 
Life is complex, with real and imaginary parts.

Re: native symlinks and non-existent targets

[Lee]

On Mon, Apr 29, 2024 at 4:11 PM Csaba Ráduly wrote:
>
> On 25/04/2024 15:15, Jon Turney via Cygwin wrote:
> > On 24/04/2024 23:36, Christopher Layne via Cygwin wrote:
> > [...]
> >> If it isn't true then this seems trivial to fix.
> >
> > This assertion is trivially disproven by the lack of a patch attached. :)
> >
> >
> I don't think this is worth a gold star, but a jester's cap is surely
> warranted :)

I disagree.
  "This assertion is trivially disproven by the lack of a patch attached."
is totally worth a gold star

Regards,
Lee