pipe hang issue when running as SYSTEM

pipe hang issue when running as SYSTEM

[Jeremy Drake]

This issue was reported to MSYS2 as a hang when trying to build libxml2 in
a Windows docker container.  Another user was able to come up with a
simple reproducer and a reasonable theory as to why it happens.  The msys2
issue is https://github.com/msys2/msys2-runtime/issues/77, and I will
quote the reproducer and analysis here:

I am having the same issue when building mingw. I managed to reduce the
issue to the following shell script:

#!/bin/sh
seq 1 99999 > big_file
eval '$(eval cmd.exe /c "type big_file" | : )'

When running as a normal user this completes immediately, but when run as
a system service it hangs forever. The issue appears to be that when
running under the SYSTEM account, the third sh process holds open the read
end of the pipe. Since cmd has too much output to write all at once, it
waits until the pipe's buffer has room to write more, but since sh isn't
actually reading from the pipe, this hangs forever. When running as a
normal user the read end of the pipe is not kept open, and so cmd.exe gets
an error when attempting to write and exits immediately.

My suspicion is that this is caused by f79a461 (which keeps the read end
of the pipe open) and b531d6b (which changes the behavior depending on
whether or not the program is running as a service).

Re: pipe hang issue when running as SYSTEM

[Takashi Yano]

On Mon, 21 Mar 2022 11:45:57 -0700 (PDT)
Jeremy Drake wrote:
> This issue was reported to MSYS2 as a hang when trying to build libxml2 in
> a Windows docker container.  Another user was able to come up with a
> simple reproducer and a reasonable theory as to why it happens.  The msys2
> issue is https://github.com/msys2/msys2-runtime/issues/77, and I will
> quote the reproducer and analysis here:
> 
> I am having the same issue when building mingw. I managed to reduce the
> issue to the following shell script:
> 
> #!/bin/sh
> seq 1 99999 > big_file
> eval '$(eval cmd.exe /c "type big_file" | : )'
> 
> When running as a normal user this completes immediately, but when run as
> a system service it hangs forever. The issue appears to be that when
> running under the SYSTEM account, the third sh process holds open the read
> end of the pipe. Since cmd has too much output to write all at once, it
> waits until the pipe's buffer has room to write more, but since sh isn't
> actually reading from the pipe, this hangs forever. When running as a
> normal user the read end of the pipe is not kept open, and so cmd.exe gets
> an error when attempting to write and exits immediately.
> 
> My suspicion is that this is caused by f79a461 (which keeps the read end
> of the pipe open) and b531d6b (which changes the behavior depending on
> whether or not the program is running as a service).

Thanks for the report. This is expected problem as mentioned
in b531d6b commit message. However, I could not imagin the
situation that the service has multiple writer for the pipe
and one of them is a non-cygwin app.

Question is: Does the docker invoke the command using SYSTEM
account? Or is the processes in docker determined as running
as a service?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: pipe hang issue when running as SYSTEM

[Takashi Yano]

On Tue, 22 Mar 2022 19:18:07 +0900
Takashi Yano wrote:
> On Mon, 21 Mar 2022 11:45:57 -0700 (PDT)
> Jeremy Drake wrote:
> > This issue was reported to MSYS2 as a hang when trying to build libxml2 in
> > a Windows docker container.  Another user was able to come up with a
> > simple reproducer and a reasonable theory as to why it happens.  The msys2
> > issue is https://github.com/msys2/msys2-runtime/issues/77, and I will
> > quote the reproducer and analysis here:
> > 
> > I am having the same issue when building mingw. I managed to reduce the
> > issue to the following shell script:
> > 
> > #!/bin/sh
> > seq 1 99999 > big_file
> > eval '$(eval cmd.exe /c "type big_file" | : )'
> > 
> > When running as a normal user this completes immediately, but when run as
> > a system service it hangs forever. The issue appears to be that when
> > running under the SYSTEM account, the third sh process holds open the read
> > end of the pipe. Since cmd has too much output to write all at once, it
> > waits until the pipe's buffer has room to write more, but since sh isn't
> > actually reading from the pipe, this hangs forever. When running as a
> > normal user the read end of the pipe is not kept open, and so cmd.exe gets
> > an error when attempting to write and exits immediately.
> > 
> > My suspicion is that this is caused by f79a461 (which keeps the read end
> > of the pipe open) and b531d6b (which changes the behavior depending on
> > whether or not the program is running as a service).
> 
> Thanks for the report. This is expected problem as mentioned
> in b531d6b commit message. However, I could not imagin the
> situation that the service has multiple writer for the pipe
> and one of them is a non-cygwin app.
> 
> Question is: Does the docker invoke the command using SYSTEM
> account? Or is the processes in docker determined as running
> as a service?

I confirmed the processes in Windows docker are running as
well_known_service_sid. Let me consider a while.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: pipe hang issue when running as SYSTEM

[Jeremy Drake]

On Tue, 22 Mar 2022, Takashi Yano wrote:

> > Thanks for the report. This is expected problem as mentioned
> > in b531d6b commit message. However, I could not imagin the
> > situation that the service has multiple writer for the pipe
> > and one of them is a non-cygwin app.
> >
> > Question is: Does the docker invoke the command using SYSTEM
> > account? Or is the processes in docker determined as running
> > as a service?
>
> I confirmed the processes in Windows docker are running as
> well_known_service_sid. Let me consider a while.
>

Just got another report to MSYS2 of this behavior, this time from a Gitlab
CI runner that is running as a service and trying to pipe output from
Microsoft cl.exe while running configure.  This user reports that 3.3.4
with your "[PATCH v2] Cygwin: pipe: Avoid deadlock for non-cygwin writer."
applied to it does not solve the problem.

https://github.com/msys2/MSYS2-packages/issues/2893

Re: pipe hang issue when running as SYSTEM

[Takashi Yano]

On Thu, 24 Mar 2022 20:42:50 -0700 (PDT)
Jeremy Drake wrote:
> On Tue, 22 Mar 2022, Takashi Yano wrote:
> 
> > > Thanks for the report. This is expected problem as mentioned
> > > in b531d6b commit message. However, I could not imagin the
> > > situation that the service has multiple writer for the pipe
> > > and one of them is a non-cygwin app.
> > >
> > > Question is: Does the docker invoke the command using SYSTEM
> > > account? Or is the processes in docker determined as running
> > > as a service?
> >
> > I confirmed the processes in Windows docker are running as
> > well_known_service_sid. Let me consider a while.
> >
> 
> Just got another report to MSYS2 of this behavior, this time from a Gitlab
> CI runner that is running as a service and trying to pipe output from
> Microsoft cl.exe while running configure.  This user reports that 3.3.4
> with your "[PATCH v2] Cygwin: pipe: Avoid deadlock for non-cygwin writer."
> applied to it does not solve the problem.
> 
> https://github.com/msys2/MSYS2-packages/issues/2893

I can confirm the sample script hangs indeed if the script is
running as SYSTEM account. However, in my test, the patch solves
the issue.

It would be very helpfull if a simple github repository with
github actions, which can reproduce the issue, is provided.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: pipe hang issue when running as SYSTEM

[Jeremy Drake]

On Fri, 25 Mar 2022, Takashi Yano wrote:

> I can confirm the sample script hangs indeed if the script is
> running as SYSTEM account. However, in my test, the patch solves
> the issue.
>
> It would be very helpfull if a simple github repository with
> github actions, which can reproduce the issue, is provided.

After much fighting with docker on windows,
https://github.com/jeremyd2019/msys2-pipe-hang-test/runs/5688176578?check_suite_focus=true

Re: pipe hang issue when running as SYSTEM

[Takashi Yano]

On Fri, 25 Mar 2022 00:09:59 -0700 (PDT)
Jeremy Drake wrote:
> On Fri, 25 Mar 2022, Takashi Yano wrote:
> 
> > I can confirm the sample script hangs indeed if the script is
> > running as SYSTEM account. However, in my test, the patch solves
> > the issue.
> >
> > It would be very helpfull if a simple github repository with
> > github actions, which can reproduce the issue, is provided.
> 
> After much fighting with docker on windows,
> https://github.com/jeremyd2019/msys2-pipe-hang-test/runs/5688176578?check_suite_focus=true

Thanks much! I could reproduce the issue even with the patch.
I also noticed what is the cause.

The problem is that the command:
docker run msys2-base c:\msys64\usr\bin\bash -lc "/c/test.sh"
does not allocate tty. Therefore, kill_pgrp(__SIGNONCYGCHLD)
is not called.

If you use:
docker run -t msys2-base c:\msys64\usr\bin\bash -lc "/c/test.sh"
the problem does not occur.

I will submit v3 patch shortly.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: pipe hang issue when running as SYSTEM

[Jeremy Drake]

On Fri, 25 Mar 2022, Takashi Yano wrote:

> I will submit v3 patch shortly.

I applied your v3 patch to 3.3.4 in
https://github.com/msys2/msys2-runtime/pull/88 and re-ran my test action,
and it worked as expected this time.

I've put out a call to test on the msys2-runtime issue to confirm that it
also solves the real-world cases.

Re: pipe hang issue when running as SYSTEM

[Takashi Yano]

On Fri, 25 Mar 2022 10:01:53 -0700 (PDT)
Jeremy Drake wrote:
> On Fri, 25 Mar 2022, Takashi Yano wrote:
> 
> > I will submit v3 patch shortly.
> 
> I applied your v3 patch to 3.3.4 in
> https://github.com/msys2/msys2-runtime/pull/88 and re-ran my test action,
> and it worked as expected this time.
> 
> I've put out a call to test on the msys2-runtime issue to confirm that it
> also solves the real-world cases.

I have submitted v4 patch, which fixes the issue caused
when multiple pipes in the same process group exist.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: pipe hang issue when running as SYSTEM

[Roumen Petrov]

Hi,

Takashi Yano wrote:
> [SNIP]
>> Just got another report to MSYS2 of this behavior, this time from a Gitlab
>> CI runner that is running as a service and trying to pipe output from
>> Microsoft cl.exe while running configure.  This user reports that 3.3.4
>> with your "[PATCH v2] Cygwin: pipe: Avoid deadlock for non-cygwin writer."
>> applied to it does not solve the problem.
>>
>> https://github.com/msys2/MSYS2-packages/issues/2893
> I can confirm the sample script hangs indeed if the script is
> running as SYSTEM account. However, in my test, the patch solves
> the issue.
>
> It would be very helpfull if a simple github repository with
> github actions, which can reproduce the issue, is provided.

In my environment with AD user account:

$ cmd.exe /c "type big_file" | :

The process tried to write to a nonexistent pipe.

$ cat big_file | :

*** starting debugger for pid NNN tid KKKKK

$ uname -a

CYGWIN_NT-10.0 ZZZZZZZ 3.3.4(0.341/5/3) 2022-01-31 19:35 x86_64 Cygwin

Remarks:
- no way to debug
- no error if run as administrator

Regards,
Roumen Petrov