From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mercury.signalpunk.com (mercury.signalpunk.com [169.45.121.165]) by sourceware.org (Postfix) with ESMTP id F40E3384640E for ; Wed, 24 Apr 2024 22:11:53 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org F40E3384640E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=anodized.com Authentication-Results: sourceware.org; spf=none smtp.mailfrom=signalpunk.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org F40E3384640E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=169.45.121.165 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713996716; cv=none; b=ak4NDz8Hn3ulzzX6yErpNTlrOH6NUyXmYFDGqJK/WJv6uQprdd5UlIcbSz9FxbEEXwQj/2T9G6FgBrGOW/k7sSk5uuiTSqPt8ys+kjnmS6OV0BtreNszFHfiRLz5k4lugeuShoW0HGleVEw9GVebQquAMY0xtVdjtztlEk9gKzs= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713996716; c=relaxed/simple; bh=+7aGeTS4F6+kZtamYmRah6vwpCj4ai5OVCacXpDU0Og=; h=Date:From:To:Subject:Message-ID:MIME-Version; b=HnSzfMTIgor9N7G7WIgCJitScRp6piMYzpqui9VUT86++bo+SMcUtoDaP3SQZxy1tzX0EmOvbv21InkWbLoSiTtDGnDqqqCH0rceRQKbbQIuvWpUoSRYGE/4bC4EC06G76W3ZouuQK6FjCKf3P6qATUbji2wroOdHydBCLTUJIs= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mercury.signalpunk.com (Postfix, from userid 1000) id B200E4155B376; Wed, 24 Apr 2024 22:11:52 +0000 (UTC) Date: Wed, 24 Apr 2024 22:11:52 +0000 From: clayne@anodized.com To: cygwin@cygwin.com Subject: native symlinks and non-existent targets Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-0.6 required=5.0 tests=BAYES_00,HEADER_FROM_DIFFERENT_DOMAINS,KAM_DMARC_STATUS,KAM_LAZY_DOMAIN_SECURITY,SPF_HELO_NONE,SPF_NONE,WEIRD_PORT autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Since I recently sent an email about symlinks and cygdrive mounts, I figured I'd report another issue that's plagued me over the years and that I know others have reported in the past: You can't create native symlinks to non-existent targets and this causes a bunch of issues when rsyncing directories containing symlinks unless one does a multi-pass approach which takes special precautions to sync all the non-symlink contents first and then syncs the symlinks right after (note: this also has its own problems with links to links). Example: clayne@sv590:/tmp/link-test $ ls -la total 32 drwxr-xr-x 1 clayne None 0 Apr 24 14:13 . drwxrwxrwt 1 clayne None 0 Apr 24 14:34 .. lrwxrwxrwx 7 clayne None 3 Apr 24 14:11 _foo -> foo -rw-r--r-- 5 clayne None 0 Apr 24 14:11 foo lrwxrwxrwx 4 clayne None 3 Apr 24 14:13 foo-ln -> foo clayne@sv590:/tmp/link-test $ rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/ sending incremental file list created directory /tmp/link-test-sync-test ./ rsync: [generator] symlink "/tmp/link-test-sync-test/_foo" -> "foo" failed: No such file or directory (2) rsync: [generator] symlink "/tmp/link-test-sync-test/foo-ln" -> "foo" failed: No such file or directory (2) foo 0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/4) sent 178 bytes received 89 bytes 534.00 bytes/sec total size is 6 speedup is 0.02 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7] clayne@sv590:/tmp/link-test $ rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/ sending incremental file list _foo -> foo foo-ln -> foo sent 134 bytes received 18 bytes 304.00 bytes/sec total size is 6 speedup is 0.04 This only works in a straight-forward manner when using non-native symlinks (which can also _change_ the symlinks such that they're broken/unusable outside of cygwin): clayne@sv590:/tmp/link-test $ rm -rf /tmp/link-test-sync-test clayne@sv590:/tmp/link-test $ CYGWIN= rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/ sending incremental file list created directory /tmp/link-test-sync-test ./ _foo -> foo foo 0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/4) foo-ln -> foo sent 184 bytes received 95 bytes 558.00 bytes/sec total size is 6 speedup is 0.02 Or another very simple test-case not involving rsync: clayne@sv590:/tmp/link-test $ ln -s this-does-not-exist some-link ln: failed to create symbolic link 'some-link': No such file or directory Relevant sections of an strace for the above: 102 83054 [main] ln 1379 symlink_info::check: 0x0 = NtCreateFile (\??\C:\cygwin64\tmp\link-test) 114 83168 [main] ln 1379 symlink_info::check: not a symlink 91 83259 [main] ln 1379 symlink_info::check: 0 = symlink.check(C:\cygwin64\tmp\link-test, 0x7FFFFB140) (mount_flags 0x30008, path_flags 0x0) 92 83351 [main] ln 1379 path_conv::check: this->path(C:\cygwin64\tmp\link-test\this-does-not-exist), has_acls(1) 91 83442 [main] ln 1379 seterrno_from_win_error: /usr/src/debug/cygwin-3.5.3-1/winsup/cygwin/path.cc:2063 windows error 2 90 83532 [main] ln 1379 geterrno_from_win_error: windows error 2 == errno 2 83 83615 [main] ln 1379 symlink_worker: -1 = symlink_worker(this-does-not-exist, /tmp/link-test/some-link, 0) However, you _can_ do this: clayne@sv590:/tmp/link-test $ tmp="$(mktemp)" && ln -s "$tmp" some-link && rm -f "$tmp" && ls -la some-link lrwxrwxrwx 1 clayne None 19 Apr 24 14:58 some-link -> /tmp/tmp.o7xpJxaqig And here's a case showing where "hard-linked" symlinks work with non-existent targets: clayne@sv590:/tmp/link-test $ find -type l -print0 | rsync -avSHP --files-from=- --from0 --link-dest=/tmp/link-test /tmp/link-test /tmp/link-test-link-dest building file list ... 3 files to consider created directory /tmp/link-test-link-dest sent 113 bytes received 59 bytes 344.00 bytes/sec total size is 6 speedup is 0.03 clayne@sv590:/tmp/link-test $ ls -l /tmp/link-test-link-dest total 0 lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 _foo -> foo lrwxrwxrwx 5 clayne None 3 Apr 24 14:13 foo-ln -> foo clayne@sv590:/tmp/link-test $ ls -lai _foo /tmp/link-test-link-dest/_foo 48413695994484407 lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 /tmp/link-test-link-dest/_foo -> foo 48413695994484407 lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 _foo -> foo This only works because the symlinks are being straight up cloned and not being newly created on the destination side. This "works" if one is doing a --link-dest of an entire directory to essentially create a hard-linked copy but anything outside of that use case will still require multiple runs. Based on past threads I've read I believe the issue is actually with windows not allowing a symlink to be created with a non-existent target, but I do know windows does not care if you break a link after the fact. For the non-existent target case, is there any way we could just hack-fix this or workaround it at the cygwin layer by create a symlink to a temp file representing the eventual target (I realize this is way easier said than done) and then immediately removing the temp file? Obviously there are some atomicity issues here because one might not be able to just create an in-situ file without creating the directory hierarchy it needs to live in first. On the other hand it doesn't seem ideal to require an rsync to run multiple times nor would it make sense to require every application or utility involved with symlinks to somehow know of this cygwin/windows specific issue (breaks abstraction). There has got to be a better way of handling this issue such that it's transparent to the caller. -cl