Forcing setup.exe not to create WSL symlinks

Forcing setup.exe not to create WSL symlinks

[Michael Wild]

Hi all

Is there a way to disable WSL symlinks when installing Cygwin with
setup.exe? Problem is that Docker on Windows apparently doesn't support
them (see https://github.com/moby/moby/issues/41058). I would like to use a
custom Docker image for GitHub Actions to reproducibly build my packages
but am failing to build the base image because of this.

Kind regards

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Andrey Repin]

Greetings, Michael Wild!

> Is there a way to disable WSL symlinks when installing Cygwin with
> setup.exe? Problem is that Docker on Windows apparently doesn't support
> them (see https://github.com/moby/moby/issues/41058). I would like to use a
> custom Docker image for GitHub Actions to reproducibly build my packages
> but am failing to build the base image because of this.

What is the value of your CYGWIN environment variable?
Do you run setup elevated?


-- 
With best regards,
Andrey Repin
Wednesday, August 26, 2020 18:19:10

Sorry for my terrible english...

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

Hi Andrey

On Wed, Aug 26, 2020 at 5:20 PM Andrey Repin wrote:

> Greetings, Michael Wild!
>
> > Is there a way to disable WSL symlinks when installing Cygwin with
> > setup.exe? Problem is that Docker on Windows apparently doesn't support
> > them (see https://github.com/moby/moby/issues/41058). I would like to
> use a
> > custom Docker image for GitHub Actions to reproducibly build my packages
> > but am failing to build the base image because of this.
>
> What is the value of your CYGWIN environment variable?
> Do you run setup elevated?
>
>
I tried setting CYGWIN=winsymlinks:lnk but no luck. The docker build
process uses an administrator account inside the container, so yes, it is
elevated.

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Achim Gratz]

Michael Wild via Cygwin writes:
> Is there a way to disable WSL symlinks when installing Cygwin with
> setup.exe? Problem is that Docker on Windows apparently doesn't support
> them (see https://github.com/moby/moby/issues/41058). I would like to use a
> custom Docker image for GitHub Actions to reproducibly build my packages
> but am failing to build the base image because of this.

That description is not very clear.  If the symlinks are created during
postinstall, then they should obey the value of CYGWIN environment that
is in effect at that time.  If so, then the behaviour should start
appearing with cygwin-3.1.5, which is the first release that has the
code supporting these (and makes them the default).  I'm not aware that
the behaviour of setup.exe has changed in that respect recently, but I
haven't checked thoroughly.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

On Wed, Aug 26, 2020 at 8:31 PM Achim Gratz wrote:

> Michael Wild via Cygwin writes:
> > Is there a way to disable WSL symlinks when installing Cygwin with
> > setup.exe? Problem is that Docker on Windows apparently doesn't support
> > them (see https://github.com/moby/moby/issues/41058). I would like to
> use a
> > custom Docker image for GitHub Actions to reproducibly build my packages
> > but am failing to build the base image because of this.
>
> That description is not very clear.  If the symlinks are created during
> postinstall, then they should obey the value of CYGWIN environment that
> is in effect at that time.  If so, then the behaviour should start
> appearing with cygwin-3.1.5, which is the first release that has the
> code supporting these (and makes them the default).  I'm not aware that
> the behaviour of setup.exe has changed in that respect recently, but I
> haven't checked thoroughly.
>
>
> Regards,
> Achim.
>
>
In order to reproduce, run the following commands on Windows:

> docker run --name setup -it
mcr.microsoft.com/windows/servercore:10.0.17763.1397-amd64

Once inside the container, run:

> powershell
PS> $env:CYGWIN=winsymlinks:lnk
PS> mkdir \tmp
PS> cd \tmp
PS> Invoke-WebRequest -Uri https://www.cygwin.com/setup-x86_64.exe -OutFile
setup-x86_64.exe
PS> .\setup-x86_64.exe -W -n -q -R C:\Cygwin64 -s
http://mirrors.kernel.org/sourceware/cygwin/ -P default
PS> Wait-Process -Name setup-x86_64
PS> fsutil reparsePoint query C:\cygwin64\dev\fd

The last command will output something like:

Reparse Tag Value : 0xa000001d
Tag value: Microsoft
Tag value: Name Surrogate

Reparse Data Length: 0x00000011
Reparse Data:
0000: 02 00 00 00 2f 70 72 6f 63 2f 73 65 6c 66 2f 66 ..../proc/self/f
0010: 64

So, the CYGWIN variable got ignored.

If you want, you can try to run a Cygwin shell:

PS> C:\Cygwin64\bin\bash.exe --login -i

For me this works fine. Then, exit the container and try to commit it:

PS> exit
> exit
> docker commit setup cygwin:latest

The last command then errors out with:

Error response from daemon: re-exec error: exit status 1: output:
time="2020-08-26T21:00:55+02:00" level=error msg="hcsshim::ImportLayer -
failed failed in Win32: The system cannot find th
e path specified. (0x3)" error="hcsshim::ImportLayer - failed failed in
Win32: The system cannot find the path specified. (0x3)"
importFolderPath="C:\\ProgramData\\Docker\\tmp\\hcs00092524
3"
path="\\\\?\\C:\\ProgramData\\Docker\\windowsfilter\\c31ac616438460478e6cc6e5526ee664e620ba4479da433bcabb6d2a76880ef4"
hcsshim::ImportLayer - failed failed in Win32: The system cannot find the
path specified. (0x3)

Does this help diagnosing the problem?

Kind regards

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

On Wed, Aug 26, 2020 at 9:06 PM Michael Wild wrote:

>
>
> On Wed, Aug 26, 2020 at 8:31 PM Achim Gratz wrote:
>
>> Michael Wild via Cygwin writes:
>> > Is there a way to disable WSL symlinks when installing Cygwin with
>> > setup.exe? Problem is that Docker on Windows apparently doesn't support
>> > them (see https://github.com/moby/moby/issues/41058). I would like to
>> use a
>> > custom Docker image for GitHub Actions to reproducibly build my packages
>> > but am failing to build the base image because of this.
>>
>> That description is not very clear.  If the symlinks are created during
>> postinstall, then they should obey the value of CYGWIN environment that
>> is in effect at that time.  If so, then the behaviour should start
>> appearing with cygwin-3.1.5, which is the first release that has the
>> code supporting these (and makes them the default).  I'm not aware that
>> the behaviour of setup.exe has changed in that respect recently, but I
>> haven't checked thoroughly.
>>
>>
>> Regards,
>> Achim.
>>
>>
> In order to reproduce, run the following commands on Windows:
>
> > docker run --name setup -it
> mcr.microsoft.com/windows/servercore:10.0.17763.1397-amd64
>
> Once inside the container, run:
>
> > powershell
> PS> $env:CYGWIN=winsymlinks:lnk
> PS> mkdir \tmp
> PS> cd \tmp
> PS> Invoke-WebRequest -Uri https://www.cygwin.com/setup-x86_64.exe
> -OutFile setup-x86_64.exe
> PS> .\setup-x86_64.exe -W -n -q -R C:\Cygwin64 -s
> http://mirrors.kernel.org/sourceware/cygwin/ -P default
> PS> Wait-Process -Name setup-x86_64
> PS> fsutil reparsePoint query C:\cygwin64\dev\fd
>
> The last command will output something like:
>
> Reparse Tag Value : 0xa000001d
> Tag value: Microsoft
> Tag value: Name Surrogate
>
> Reparse Data Length: 0x00000011
> Reparse Data:
> 0000: 02 00 00 00 2f 70 72 6f 63 2f 73 65 6c 66 2f 66 ..../proc/self/f
> 0010: 64
>
> So, the CYGWIN variable got ignored.
>
> If you want, you can try to run a Cygwin shell:
>
> PS> C:\Cygwin64\bin\bash.exe --login -i
>
> For me this works fine. Then, exit the container and try to commit it:
>
> PS> exit
> > exit
> > docker commit setup cygwin:latest
>
> The last command then errors out with:
>
> Error response from daemon: re-exec error: exit status 1: output:
> time="2020-08-26T21:00:55+02:00" level=error msg="hcsshim::ImportLayer -
> failed failed in Win32: The system cannot find th
> e path specified. (0x3)" error="hcsshim::ImportLayer - failed failed in
> Win32: The system cannot find the path specified. (0x3)"
> importFolderPath="C:\\ProgramData\\Docker\\tmp\\hcs00092524
> 3"
> path="\\\\?\\C:\\ProgramData\\Docker\\windowsfilter\\c31ac616438460478e6cc6e5526ee664e620ba4479da433bcabb6d2a76880ef4"
> hcsshim::ImportLayer - failed failed in Win32: The system cannot find the
> path specified. (0x3)
>
> Does this help diagnosing the problem?
>
> Kind regards
>
> Michael
>

I tried snooping into the setup code at
https://cygwin.com/git/?p=cygwin-apps/setup.git;a=summary. The word CYGWIN
doesn't show up, so I don't assume it checks the variable. I do see code to
create links when unpacking archives, but to me it looks like it only does
hard-links and the old-style link files. Next I'll try to find where the
post-install scripts are called and put in some tracing code to find out
what's going on...

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Jon Turney]

On 26/08/2020 19:29, Achim Gratz wrote:
> Michael Wild via Cygwin writes:
>> Is there a way to disable WSL symlinks when installing Cygwin with
>> setup.exe? Problem is that Docker on Windows apparently doesn't support
>> them (see https://github.com/moby/moby/issues/41058). I would like to use a

As a comment in that issue identifies [1], this isn't actually an issue 
with setup per se, but the post-install scripts it runs.

[1] https://github.com/moby/moby/issues/41058#issuecomment-674402289

setup itself always creates symlinks when unpacking packages as 
traditional Cygwin symlinks (a plain file containing a magic value and 
the path, which are only understood by Cygwin).

(Setup probably should match Cygwin's behaviour and/or have an option to 
specify how it creates symlinks)

>> custom Docker image for GitHub Actions to reproducibly build my packages
>> but am failing to build the base image because of this.
> 
> That description is not very clear.  If the symlinks are created during
> postinstall, then they should obey the value of CYGWIN environment that
> is in effect at that time.  If so, then the behaviour should start
> appearing with cygwin-3.1.5, which is the first release that has the
> code supporting these (and makes them the default).  I'm not aware that
> the behaviour of setup.exe has changed in that respect recently, but I
> haven't checked thoroughly.

It turns out to be the case that setup doesn't propagate the CYGWIN 
environment variable into the environment for scripts it runs, so 
setting that isn't going to make any difference.  That should probably 
be considered a bug.

However, even if that's fixed, there's no value for winsymlinks in 
CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous 
(i.e. always create traditional symlinks, don't use WSL symlink reparse 
points)

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

On Wed, Aug 26, 2020 at 10:57 PM Jon Turney wrote:

>
> It turns out to be the case that setup doesn't propagate the CYGWIN
> environment variable into the environment for scripts it runs, so
> setting that isn't going to make any difference.  That should probably
> be considered a bug.
>
> However, even if that's fixed, there's no value for winsymlinks in
> CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous
> (i.e. always create traditional symlinks, don't use WSL symlink reparse
> points)
>

Thanks Jon for the elaboration. Isn't winsymlinks=lnk supposed to do this
per the documentation? Could Corinna be convinced to include an option
where this can be done? To be honest, my hopes are greater to get this
worked-around in a useful timeframe from the Cygwin side than getting the
proper fix into Docker.

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Corinna Vinschen]

On Aug 27 07:48, Michael Wild via Cygwin wrote:
> On Wed, Aug 26, 2020 at 10:57 PM Jon Turney wrote:
> 
> >
> > It turns out to be the case that setup doesn't propagate the CYGWIN
> > environment variable into the environment for scripts it runs, so
> > setting that isn't going to make any difference.  That should probably
> > be considered a bug.
> >
> > However, even if that's fixed, there's no value for winsymlinks in
> > CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous
> > (i.e. always create traditional symlinks, don't use WSL symlink reparse
> > points)
> >
> 
> Thanks Jon for the elaboration. Isn't winsymlinks=lnk supposed to do this
> per the documentation? Could Corinna be convinced to include an option
> where this can be done? To be honest, my hopes are greater to get this
> worked-around in a useful timeframe from the Cygwin side than getting the
> proper fix into Docker.

Isn't it sufficient to fix setup so you can create lnk-style symlinks in
your scenario?  The CYGWIN env variable and the number of options is
such a mess.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

Re: Forcing setup.exe not to create WSL symlinks

[Andrey Repin]

Greetings, Corinna Vinschen!

> On Aug 27 07:48, Michael Wild via Cygwin wrote:
>> On Wed, Aug 26, 2020 at 10:57 PM Jon Turney wrote:
>> 
>> >
>> > It turns out to be the case that setup doesn't propagate the CYGWIN
>> > environment variable into the environment for scripts it runs, so
>> > setting that isn't going to make any difference.  That should probably
>> > be considered a bug.
>> >
>> > However, even if that's fixed, there's no value for winsymlinks in
>> > CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous
>> > (i.e. always create traditional symlinks, don't use WSL symlink reparse
>> > points)
>> >
>> 
>> Thanks Jon for the elaboration. Isn't winsymlinks=lnk supposed to do this
>> per the documentation? Could Corinna be convinced to include an option
>> where this can be done? To be honest, my hopes are greater to get this
>> worked-around in a useful timeframe from the Cygwin side than getting the
>> proper fix into Docker.

> Isn't it sufficient to fix setup so you can create lnk-style symlinks in
> your scenario?  The CYGWIN env variable and the number of options is
> such a mess.

The only relevant option is winsymlinks, as far as I can see, as setup is only
unpacking archives.
But my opinion is that setup should first try to create native symlinks, then
.LNK style links, as both are compatible with Windows.


-- 
With best regards,
Andrey Repin
Thursday, August 27, 2020 12:29:54

Sorry for my terrible english...

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

On Thu, Aug 27, 2020 at 11:02 AM Corinna Vinschen wrote:

> On Aug 27 07:48, Michael Wild via Cygwin wrote:
> > On Wed, Aug 26, 2020 at 10:57 PM Jon Turney wrote:
> >
> > >
> > > It turns out to be the case that setup doesn't propagate the CYGWIN
> > > environment variable into the environment for scripts it runs, so
> > > setting that isn't going to make any difference.  That should probably
> > > be considered a bug.
> > >
> > > However, even if that's fixed, there's no value for winsymlinks in
> > > CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous
> > > (i.e. always create traditional symlinks, don't use WSL symlink reparse
> > > points)
> > >
> >
> > Thanks Jon for the elaboration. Isn't winsymlinks=lnk supposed to do this
> > per the documentation? Could Corinna be convinced to include an option
> > where this can be done? To be honest, my hopes are greater to get this
> > worked-around in a useful timeframe from the Cygwin side than getting the
> > proper fix into Docker.
>
> Isn't it sufficient to fix setup so you can create lnk-style symlinks in
> your scenario?  The CYGWIN env variable and the number of options is
> such a mess.
>
>
> Corinna
>

I am right now modifying setup to not sanitize CYGWIN away when invoking
the post-install scripts and then will report back.

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

On Thu, Aug 27, 2020 at 11:44 AM Michael Wild wrote:

>
> On Thu, Aug 27, 2020 at 11:02 AM Corinna Vinschen wrote:
>
>> On Aug 27 07:48, Michael Wild via Cygwin wrote:
>> > On Wed, Aug 26, 2020 at 10:57 PM Jon Turney wrote:
>> >
>> > >
>> > > It turns out to be the case that setup doesn't propagate the CYGWIN
>> > > environment variable into the environment for scripts it runs, so
>> > > setting that isn't going to make any difference.  That should probably
>> > > be considered a bug.
>> > >
>> > > However, even if that's fixed, there's no value for winsymlinks in
>> > > CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous
>> > > (i.e. always create traditional symlinks, don't use WSL symlink
>> reparse
>> > > points)
>> > >
>> >
>> > Thanks Jon for the elaboration. Isn't winsymlinks=lnk supposed to do
>> this
>> > per the documentation? Could Corinna be convinced to include an option
>> > where this can be done? To be honest, my hopes are greater to get this
>> > worked-around in a useful timeframe from the Cygwin side than getting
>> the
>> > proper fix into Docker.
>>
>> Isn't it sufficient to fix setup so you can create lnk-style symlinks in
>> your scenario?  The CYGWIN env variable and the number of options is
>> such a mess.
>>
>>
>> Corinna
>>
>
> I am right now modifying setup to not sanitize CYGWIN away when invoking
> the post-install scripts and then will report back.
>
> Michael
>

Hmm, OK, changing script.cc to not strip out CYGWIN is trivial. But the
harder part is that main.cc uses ShellExecuteEx() with
SHELLEXECUTEINFO.verb set to "runas" in order to re-run setup elevated.
This resets all environment variables back to default. In my instance I can
circumvent this by passing --no-admin and running from an elevated shell
where CYGWIN is set already. And indeed, this works and resolves my Docker
problem. I attached this fix as a patch. But on the other hand, I find this
behavior to be a bit confusing. However, I don't see an easy way of
resolving the "runas" issue, because it is by design. An option would be
that the calling process passes the CYGWIN variable as a command line
argument to the elevated process. But that is also ugly. What do you guys
think?

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

[-- Attachment #1: Type: text/plain, Size: 123 bytes --]

On Thu, Aug 27, 2020 at 2:48 PM Michael Wild wrote:

>
>
> [...] I attached this fix as a patch. [...]
>
>
No, I didn't...

[-- Attachment #2: 0001-Keep-CYGWIN-environment-variable-when-running-script.patch --]
[-- Type: application/octet-stream, Size: 683 bytes --]

From 2d112b26d8f29b083989ba919364d47c6d43661f Mon Sep 17 00:00:00 2001
From: Michael Wild <themiwi@gmail.com>
Date: Thu, 27 Aug 2020 14:45:42 +0200
Subject: [PATCH 1/1] Keep CYGWIN environment variable when running scripts

---
 script.cc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/script.cc b/script.cc
index 92c3122..67e12ad 100644
--- a/script.cc
+++ b/script.cc
@@ -89,6 +89,7 @@ init_run_script ()
 	  char *eq = strchr (p, '=');
 	  *eq = '\0';
 	  if (strcasecmp (p, "comspec") != 0
+	      && strncasecmp (p, "cygwin", 7) != 0
 	      && strcasecmp (p, "path") != 0
 	      && strncasecmp (p, "system", 7) != 0
 	      && strncasecmp (p, "user", 4) != 0
-- 
2.28.0

Re: Forcing setup.exe not to create WSL symlinks

[Jon Turney]

On 27/08/2020 13:49, Michael Wild via Cygwin wrote:
> On Thu, Aug 27, 2020 at 2:48 PM Michael Wild wrote:
> 
>>
>>
>> [...] I attached this fix as a patch. [...]

Thanks. Applied.

Re: Forcing setup.exe not to create WSL symlinks

[Jon Turney]

On 27/08/2020 06:48, Michael Wild via Cygwin wrote:
> On Wed, Aug 26, 2020 at 10:57 PM Jon Turney wrote:
>>
>> However, even if that's fixed, there's no value for winsymlinks in
>> CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous
>> (i.e. always create traditional symlinks, don't use WSL symlink reparse
>> points)
> 
> Thanks Jon for the elaboration. Isn't winsymlinks=lnk supposed to do this
> per the documentation?

No.

I'm not sure what documentation you are looking at, but [1][2] seems 
reasonably clear that the behaviour of Cygwin 3.1.4 and earlier (what I 
referred to as "traditional cygwin symlinks") isn't the same as ".lnk 
symlinks".

[1] https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks
[2] https://cygwin.com/cygwin-ug-net/using-cygwinenv.html

Re: Forcing setup.exe not to create WSL symlinks

[Michael Wild]

On Sun, 30 Aug 2020, 22:24 Jon Turney wrote:

> On 27/08/2020 06:48, Michael Wild via Cygwin wrote:
> > On Wed, Aug 26, 2020 at 10:57 PM Jon Turney wrote:
> >>
> >> However, even if that's fixed, there's no value for winsymlinks in
> >> CYGWIN env var to specify the behaviour of Cygwin 3.1.4 and previous
> >> (i.e. always create traditional symlinks, don't use WSL symlink reparse
> >> points)
> >
> > Thanks Jon for the elaboration. Isn't winsymlinks=lnk supposed to do this
> > per the documentation?
>
> No.
>
> I'm not sure what documentation you are looking at, but [1][2] seems
> reasonably clear that the behaviour of Cygwin 3.1.4 and earlier (what I
> referred to as "traditional cygwin symlinks") isn't the same as ".lnk
> symlinks".
>
> [1] https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks
> [2] https://cygwin.com/cygwin-ug-net/using-cygwinenv.html


Ah, I was jumping ahead there. My intention was having no WSL symlink
reparse points. And per my experiments, winsymlinks:lnk gets that job done.

Michael

Re: Forcing setup.exe not to create WSL symlinks

[Jon Turney]

On 27/08/2020 13:48, Michael Wild via Cygwin wrote:
> Hmm, OK, changing script.cc to not strip out CYGWIN is trivial. But the
> harder part is that main.cc uses ShellExecuteEx() with
> SHELLEXECUTEINFO.verb set to "runas" in order to re-run setup elevated.
> This resets all environment variables back to default. In my instance I can
> circumvent this by passing --no-admin and running from an elevated shell
> where CYGWIN is set already. And indeed, this works and resolves my Docker
> problem. I attached this fix as a patch. But on the other hand, I find this
> behavior to be a bit confusing. However, I don't see an easy way of
> resolving the "runas" issue, because it is by design. An option would be
> that the calling process passes the CYGWIN variable as a command line
> argument to the elevated process. But that is also ugly. What do you guys
> think?

I think an option which explicitly specifies the symlink style would be 
best (rather than option to specify the CYGWIN env var).

You'd also need to teach setup to create syminks in these different 
ways, and set the CYGWIN env var for scripts appropriately.