Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[David Finnie]

Hi Ken,

Thanks for having a look at my issue.

On 1/01/2020 06:20, Ken Brown wrote:
> On 12/30/2019 6:10 PM, David Finnie wrote:
>> I recently upgraded my cygwin64 installation to get latest packages.
>>
>> After the install, if I run a fairly lengthy GNU make with multiple concurrent
>> jobs (-j option) specified, some of the sub-makes fail with "Resource
>> temporarily unavailable" errors. In all cases, this has been when make is
>> starting a shell (i.e. $(shell ...) ) to help with setting up some of the
>> variables required in the make processing. The failure, however, occurs in
>> different places in the make files used (the make spawns several sub-makes).
> Does this happen with every parallel make, or is it one specific program that
> you're building?  If the latter, can you give a detailed recipe so that others
> can try to reproduce the problem?

It's happening consistently across a bunch of libraries and executables 
that I build (with a series of invocations of make). While each of those 
builds has their own Makefile, each of them includes a common set of 
makefiles that does the vast majority of the work, so I guess you could 
say that it really is just isolated (as far as I can tell), to one build 
environment. The build is not of open source software - it is for 
software created by the company I work for, so unfortunately I can't 
share all of the source code etc., but I can certainly give details of 
what we're doing.

The make builds a cross-platform set of products, and so invokes both 
Windows native compilers, and also cross compilers for the other 
platforms (Linux, and HPE NonStop). So there are many tools being 
invoked simultaneously.

Keep in mind, though, that this has been working perfectly for me for 
over a decade. I will say that I did run into the need to run rebase 
after upgrading cygwin many years ago, but that fixed the issue for me 
immediately. Not so this time, and there has been nothing else changed 
in my environment. I actually ran the build successfully just minutes 
before upgrading the cygwin packages, and then it failed immediately 
after that.

I have not tried building for any other product. I might try that.

> Have you tried reverting to cygwin-3.0.7-1? 


No, I haven't - I'll try that now. Thanks.


>
> The only thing that jumps out at me from your cygcheck output is that your PATH
> is very long.  I don't know if that's relevant, but you might try cutting it
> down before running make.


I did try changing PATH to a much smaller value 
(/bin:/usr/bin:/usr/local/bin) but I still got exactly the same 
behaviour :-( A good thing to try, though.

I'll see how I go after downgrading cygwin.

Thanks again.

Dave


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[David Finnie]

Hi all,

I did as Ken suggested and reverted to 3.0.7-1. (Thanks, Ken !)

That has fixed the issue for me, and the make it now running again at 
full speed. I forgot to mention that it did seem somewhat slower with 
the new version of cygwin.

That strongly implies that there is an issue with changes made since 
then. I noticed that in fork.cc, at line 540, this new code exists:

   /* Do not attach to the child before it has successfully initialized.
      Otherwise we may wait forever, or deliver an orphan SIGCHILD. */
   if (!child.reattach ())
     {
       this_errno = EAGAIN;
#ifdef DEBUGGING0
       error ("child reattach failed");
#endif
       goto cleanup;
     }

Since I am getting "Resource temporarily unavailable", which equates to 
EAGAIN, and this code is new, this looks like the most likely candidate 
to investigate first (but rest assured that I'm not saying that this 
code is definitely the culprit !).

The reattach and parent/child logic looks reasonably complex and I've 
only just downloaded the code, so I'm not sure that I want to try fixing 
anything for fear of breaking much more.

Could someone please scan an eye over this code ? I noticed that there 
were 2 commits related to this - moving the reattach() until later 
processing. Is it possible that there is still an opportunity that the 
reattach() fails even when things are working properly ?

Thanks.

Dave

On 2/01/2020 09:32, David Finnie wrote:
> Hi Ken,
>
> Thanks for having a look at my issue.
>
> On 1/01/2020 06:20, Ken Brown wrote:
>> On 12/30/2019 6:10 PM, David Finnie wrote:
>>> I recently upgraded my cygwin64 installation to get latest packages.
>>>
>>> After the install, if I run a fairly lengthy GNU make with multiple 
>>> concurrent
>>> jobs (-j option) specified, some of the sub-makes fail with "Resource
>>> temporarily unavailable" errors. In all cases, this has been when 
>>> make is
>>> starting a shell (i.e. $(shell ...) ) to help with setting up some 
>>> of the
>>> variables required in the make processing. The failure, however, 
>>> occurs in
>>> different places in the make files used (the make spawns several 
>>> sub-makes).
>> Does this happen with every parallel make, or is it one specific 
>> program that
>> you're building?  If the latter, can you give a detailed recipe so 
>> that others
>> can try to reproduce the problem?
>
> It's happening consistently across a bunch of libraries and 
> executables that I build (with a series of invocations of make). While 
> each of those builds has their own Makefile, each of them includes a 
> common set of makefiles that does the vast majority of the work, so I 
> guess you could say that it really is just isolated (as far as I can 
> tell), to one build environment. The build is not of open source 
> software - it is for software created by the company I work for, so 
> unfortunately I can't share all of the source code etc., but I can 
> certainly give details of what we're doing.
>
> The make builds a cross-platform set of products, and so invokes both 
> Windows native compilers, and also cross compilers for the other 
> platforms (Linux, and HPE NonStop). So there are many tools being 
> invoked simultaneously.
>
> Keep in mind, though, that this has been working perfectly for me for 
> over a decade. I will say that I did run into the need to run rebase 
> after upgrading cygwin many years ago, but that fixed the issue for me 
> immediately. Not so this time, and there has been nothing else changed 
> in my environment. I actually ran the build successfully just minutes 
> before upgrading the cygwin packages, and then it failed immediately 
> after that.
>
> I have not tried building for any other product. I might try that.
>
>> Have you tried reverting to cygwin-3.0.7-1? 
>
>
> No, I haven't - I'll try that now. Thanks.
>
>
>>
>> The only thing that jumps out at me from your cygcheck output is that 
>> your PATH
>> is very long.  I don't know if that's relevant, but you might try 
>> cutting it
>> down before running make.
>
>
> I did try changing PATH to a much smaller value 
> (/bin:/usr/bin:/usr/local/bin) but I still got exactly the same 
> behaviour :-( A good thing to try, though.
>
> I'll see how I go after downgrading cygwin.
>
> Thanks again.
>
> Dave
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Ken Brown]

On 1/1/2020 6:43 PM, David Finnie wrote:
> That strongly implies that there is an issue with changes made since then. I 
> noticed that in fork.cc, at line 540, this new code exists:
> 
>    /* Do not attach to the child before it has successfully initialized.
>       Otherwise we may wait forever, or deliver an orphan SIGCHILD. */
>    if (!child.reattach ())
>      {
>        this_errno = EAGAIN;
> #ifdef DEBUGGING0
>        error ("child reattach failed");
> #endif
>        goto cleanup;
>      }
> 
> Since I am getting "Resource temporarily unavailable", which equates to EAGAIN, 
> and this code is new, this looks like the most likely candidate to investigate 
> first (but rest assured that I'm not saying that this code is definitely the 
> culprit !).

You could try building cygwin with DEBUGGING0 defined to see if this is where 
the EAGAIN is coming from.  I think the error message would show up in strace 
output, so you'd have to be able to reproduce the problem under strace.

And it would be great if you could find a testcase that you can share with the 
list, so that others can try to reproduce the problem.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Peter Dons Tychsen]

Hi Cygwinners,

On Thu, 2020-01-02 at 10:43 +1100, David Finnie wrote:
> I did as Ken suggested and reverted to 3.0.7-1. (Thanks, Ken !)
> 
> That has fixed the issue for me, and the make it now running again
> at 
> full speed. I forgot to mention that it did seem somewhat slower
> with 
> the new version of cygwin.

I second this. I have the exact same problem on some of my setups.
Reverting to 3.0.7-1 also fixes it for me. Without reverting i get
stuck in processes all the time (especially with make -jx). Also the
slowdown seems apparent here, even though that is not my biggest
concern ATM, but hanging build servers are :-).

My guess would also be the fork changes, even though i have zero proof
so far. Have not had time to debug.

Thanks,

/pedro


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[David Finnie]

Hi Pedro,

That's good to hear confirmation.

I have started down the road of building cygwin, but did run into a few
issues so don't have a debuggable version yet. If you beat me to it,
please let me know. Thanks!

Dave

On Fri, 3 Jan. 2020, 08:58 Peter Dons Tychsen, <pdt@dontech.dk> wrote:

> Hi Cygwinners,
>
> On Thu, 2020-01-02 at 10:43 +1100, David Finnie wrote:
> > I did as Ken suggested and reverted to 3.0.7-1. (Thanks, Ken !)
> >
> > That has fixed the issue for me, and the make it now running again
> > at
> > full speed. I forgot to mention that it did seem somewhat slower
> > with
> > the new version of cygwin.
>
> I second this. I have the exact same problem on some of my setups.
> Reverting to 3.0.7-1 also fixes it for me. Without reverting i get
> stuck in processes all the time (especially with make -jx). Also the
> slowdown seems apparent here, even though that is not my biggest
> concern ATM, but hanging build servers are :-).
>
> My guess would also be the fork changes, even though i have zero proof
> so far. Have not had time to debug.
>
> Thanks,
>
> /pedro
>
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Ken Brown]

[Please don't top-post on this list.  Thanks.]

On 1/2/2020 5:29 PM, David Finnie wrote:
> Hi Pedro,
> 
> That's good to hear confirmation.
> 
> I have started down the road of building cygwin, but did run into a few
> issues so don't have a debuggable version yet. If you beat me to it,
> please let me know. Thanks!

You could also try the various test releases between cygwin-3.0.7 and 
cygwin-3.1.0 using the Cygwin Time Machine.  See

   https://cygwin.com/ml/cygwin/2020-01/msg00018.html

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Peter Dons Tychsen]

Hi David & Co,

> I have started down the road of building cygwin, but did run into a
> few issues so don't have a debuggable version yet. If you beat me to
> it,  please let me know. Thanks!

Any findings?

I have found something interesting:

1) If i run the terminal without mintty, the problem goes away. I can
run my makefiles with "make -j20" without any issues if i start with
"bash --login" instead of the mintty shortcut, where it just uses the
normal windows terminal.

2) As soon as i run via mintty, even "make -j2" quickly goes south,
causes various fork() issues as we have seen. I will see if i can
create a small makefile that shows this.

So either the change we are looking for affects mintty, or the actual
bug we are looking for is in mintty and not the DLL (or both).

Can anyone shed light on this? Another problem is that if i rewind the
DLL (building previous commits) too much then mintty stops working...

Any pointers would be great!

Thanks,

/pedro


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Ken Brown]

On 2/5/2020 4:45 PM, Peter Dons Tychsen wrote:
> Hi David & Co,
> 
>> I have started down the road of building cygwin, but did run into a
>> few issues so don't have a debuggable version yet. If you beat me to
>> it,  please let me know. Thanks!
> 
> Any findings?
> 
> I have found something interesting:
> 
> 1) If i run the terminal without mintty, the problem goes away. I can
> run my makefiles with "make -j20" without any issues if i start with
> "bash --login" instead of the mintty shortcut, where it just uses the
> normal windows terminal.
> 
> 2) As soon as i run via mintty, even "make -j2" quickly goes south,
> causes various fork() issues as we have seen. I will see if i can
> create a small makefile that shows this.
> 
> So either the change we are looking for affects mintty, or the actual
> bug we are looking for is in mintty and not the DLL (or both).
> 
> Can anyone shed light on this? Another problem is that if i rewind the
> DLL (building previous commits) too much then mintty stops working...

That last issue is probably due to changes in /usr/bin/cygwin-console-helper.exe 
related to the new PTY code.  So in addition to rewinding the DLL, you have to 
rewind cygwin-console-helper.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Takashi Yano]

On Wed, 5 Feb 2020 17:07:38 -0500
Ken Brown wrote:
> That last issue is probably due to changes in /usr/bin/cygwin-console-helper.exe 
> related to the new PTY code.  So in addition to rewinding the DLL, you have to 
> rewind cygwin-console-helper.

Indeed, cygwin-console-helper.exe was changed related to the pseudo
console support, however, it suppose to be upper compatible.
At least cygwin 3.0.7 works with new cygwin-console-helper.exe.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[David Finnie]

Hi Pedro,
>> I have started down the road of building cygwin, but did run into a
>> few issues so don't have a debuggable version yet. If you beat me to
>> it,  please let me know. Thanks!
> Any findings?

Unfortunately no, I did get a clean build, but "make install" following 
it just created complete havoc in my cygwin environment (lots of errors 
indicating that a rebase was required, but rebasing didn't fix them). 
Reinstalls of the cygwin base dll (and other packages) didn't fix it, 
and I was forced to delete my entire cygwin directory and reinstall all 
cygwin packages that I needed from scratch. I'm not inclined to go 
through that pain again any time soon.

I have on my to-do list to use the cygwin time machine, as Ken 
suggested, and will try to isolate when the problem first appeared. 
However, my day job has been getting in the way on that, so I haven't 
yet found the time.

For the moment, I'm continuing to run on 3.0.7 without any issues.

> I have found something interesting:
>
> 1) If i run the terminal without mintty, the problem goes away. I can
> run my makefiles with "make -j20" without any issues if i start with
> "bash --login" instead of the mintty shortcut, where it just uses the
> normal windows terminal.
>
> 2) As soon as i run via mintty, even "make -j2" quickly goes south,
> causes various fork() issues as we have seen. I will see if i can
> create a small makefile that shows this.
>
That would be awesome if you could create a small test case.

Dave


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Ken Brown]

[Please don't top-post on this list.  Thanks.]

On 2/5/2020 5:18 PM, David Finnie wrote:
> Hi Pedro,
>>> I have started down the road of building cygwin, but did run into a
>>> few issues so don't have a debuggable version yet. If you beat me to
>>> it,  please let me know. Thanks!
>> Any findings?
> 
> Unfortunately no, I did get a clean build, but "make install" following it just 
> created complete havoc in my cygwin environment (lots of errors indicating that 
> a rebase was required, but rebasing didn't fix them). Reinstalls of the cygwin 
> base dll (and other packages) didn't fix it, and I was forced to delete my 
> entire cygwin directory and reinstall all cygwin packages that I needed from 
> scratch. I'm not inclined to go through that pain again any time soon.

You should build as suggested in

   https://www.cygwin.com/faq.html#faq.programming.building-cygwin

so that "make install" installs everything into a temporary location from which 
you can copy what you need.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[David Finnie]

> [Please don't top-post on this list.  Thanks.]
>
>
What do you mean by top posting, then ? How was my post a top post ?

Thanks.

Dave


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Ken Brown]

On 2/5/2020 6:07 PM, David Finnie wrote:
> 
>> [Please don't top-post on this list.  Thanks.]
>>
>>
> What do you mean by top posting, then ? How was my post a top post ?

It wasn't.  I misread it.  Sorry.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Brian Inglis]

On 2020-02-05 16:35, Ken Brown wrote:
> On 2/5/2020 6:07 PM, David Finnie wrote:
>>> [Please don't top-post on this list.  Thanks.]
>> What do you mean by top posting, then ? How was my post a top post ?
> It wasn't.  I misread it.  Sorry.

DF's post immediately preceding that KB post at the start of *January* *was*
TOFU:	https://cygwin.com/ml/cygwin/2020-01/msg00021.html

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[David Finnie]

On 8/02/2020 05:13, Brian Inglis wrote:
> DF's post immediately preceding that KB post at the start of*January*  *was*

Brian, indeed. And, as I'm sure you're aware - given that you were 
trawling through past posts - Ken already pulled me up on it, and since 
then I haven't. It has been my *only* top post to the list.

https://cygwin.com/ml/cygwin/2020-01/msg00022.html

I sent this latest message because Ken *again* pulled me up on it, even 
though I haven't done it since.

As I mentioned, if this is a "rule" for the list, then it should be 
mentioned in the "Reporting guidelines" here: 
https://cygwin.com/problems.html

I would not have done it in the first place if it was mentioned there, 
because I did carefully read that prior to my first post.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Peter Dons Tychsen]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

Hi all,

On Thu, 2020-02-06 at 09:18 +1100, David Finnie wrote:
> That would be awesome if you could create a small test case

OK, i put together a test-case:

1) Put attached makefile somewhere
2) Download https://nuwen.net/files/mingw/mingw-17.1-without-git.exe
and unzip it in same place.
3) Now run "make create"
4) Now run "make clean && make -j32". Try a couple of times.

-----------

a) If running via mintty - it hangs fairly quickly, and the task-
manager will show hanging gcc's and sh's.

b) If running in normal terminal via "bash --login" shortcut,
everything is ok. Ran 100s of times with no error.

-----------
It seems to be related to the fact the is is spawning non-cygwin
programs. If i do the same test with normal GCC (default cygwin gcc)
then everything is fine.

My guess would be that this is related to the pseudo-console changes.
I think it relates to an terminal I/O problem when the tasks are
starting/stopping.

Thanks,

/pedro



[-- Attachment #2: Makefile --]
[-- Type: text/x-makefile, Size: 598 bytes --]

test: hang

define test
test$(1).d: test$(1).c
	echo running thread $(1)
	MinGW/bin/gcc -MP -MM -MT $$@ -M -o $$@ $$^ &&\
	MinGW/bin/gcc -c -o $$(^:.c=.o) $$^
outputs += test$(i).d
inputs += test$(i).c
endef

loops := 0 1 2 3 4 5 6 7 8 9
loops := $(foreach i, $(loops),$(i)0 $(i)1 $(i)2 $(i)3 $(i)4 $(i)5 $(i)6 $(i)7 $(i)8 $(i)9)

$(foreach i,$(loops),$(eval $(call test,$(i))))

hang: $(outputs)

print: 
	echo $(outputs)
	echo $(inputs)

%.c:
	echo "#include <stdio.h>" > $@
	echo "#include <stdlib.h>" >> $@
	echo "#include <string.h>" >> $@

create: $(inputs)
	echo

clean:
	rm -rf *.c *.d *.o

[-- Attachment #3: Type: text/plain, Size: 219 bytes --]


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[David Finnie]

On 11/02/2020 07:58, Peter Dons Tychsen wrote:
> It seems to be related to the fact the is is spawning non-cygwin
> programs. If i do the same test with normal GCC (default cygwin gcc)
> then everything is fine.

That's great work, and I can confirm that it is the same in my 
environment. Our makefiles start the following:

  * Several cygwin utilities (e.g. cygpath, cut, sed, etc.)
  * Linux cross-compilers/linkers that are cygwin programmes
  * Windows native compilers/linkers (not cygwin)
  * HPE Nonstop cross-compilers/linkers (not cygwin)

So what you've found is consistent with the issue that we're getting.

Dave


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Takashi Yano]

On Mon, 10 Feb 2020 21:58:09 +0100
Peter Dons Tychsen wrote:
> 1) Put attached makefile somewhere
> 2) Download https://nuwen.net/files/mingw/mingw-17.1-without-git.exe
> and unzip it in same place.
> 3) Now run "make create"
> 4) Now run "make clean && make -j32". Try a couple of times.
> 
> -----------
> 
> a) If running via mintty - it hangs fairly quickly, and the task-
> manager will show hanging gcc's and sh's.

Thanks for the test case. 

With cygwin 3.1.2, I confirmed make stops for a minute at:

.....
.....
echo running thread 98
echo running thread 99
running thread 98
cc    -c -o hang.o hang.c
running thread 99
MinGW/bin/gcc -MP -MM -MT test98.d -M -o test98.d test98.c && MinGW/bin/gcc -c -
o test98.o test98.c
MinGW/bin/gcc -MP -MM -MT test99.d -M -o test99.d test99.c && MinGW/bin/gcc -c -
o test99.o test99.c
cc    -c -o test.o test.c

and displays following after a minute.

cc   hang.o test00.d test01.d test02.d test03.d test04.d test05.d test06.d test07.d test08.d test09.d test10.d test11.d test12.d test13.d test14.d test15.d test16.d test17.d test18.d test19.d test20.d test21.d test22.d test23.d test24.d test25.d test26.d test27.d test28.d test29.d test30.d test31.d test32.d test33.d test34.d test35.d test36.d test37.d test38.d test39.d test40.d test41.d test42.d test43.d test44.d test45.d test46.d test47.d test48.d test49.d test50.d test51.d test52.d test53.d test54.d test55.d test56.d test57.d test58.d test59.d test60.d test61.d test62.d test63.d test64.d test65.d test66.d test67.d test68.d test69.d test70.d test71.d test72.d test73.d test74.d test75.d test76.d test77.d test78.d test79.d test80.d test81.d test82.d test83.d test84.d test85.d test86.d test87.d test88.d test89.d test90.d test91.d test92.d test93.d test94.d test95.d test96.d test97.d test98.d test99.d   -o hang
/usr/lib/gcc/x86_64-pc-cygwin/7.4.0/../../../../x86_64-pc-cygwin/bin/ld:test00.d: file format not recognized; treating as linker script
/usr/lib/gcc/x86_64-pc-cygwin/7.4.0/../../../../x86_64-pc-cygwin/bin/ld:test00.d:1: syntax error
collect2: error: ld returned 1 exit status
make: *** [<builtin>: hang] Error 1
make: *** Waiting for unfinished jobs....
rm hang.o test.o hang.c test.c

Is this the same as your problem?

If so, it goes without stopping 1 minute with patch:
https://cygwin.com/ml/cygwin-patches/2020-q1/msg00148.html

In any case, the following error occurs.

/usr/lib/gcc/x86_64-pc-cygwin/7.4.0/../../../../x86_64-pc-cygwin/bin/ld:test00.d: file format not recognized; treating as linker script
/usr/lib/gcc/x86_64-pc-cygwin/7.4.0/../../../../x86_64-pc-cygwin/bin/ld:test00.d:1: syntax error
collect2: error: ld returned 1 exit status
make: *** [<builtin>: hang] Error 1
make: *** Waiting for unfinished jobs....

Is this as you expected?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Peter Dons Tychsen]

Hi Takashi,

Thanks for looking at this & your great work on Cygwin!

On Tue, 2020-02-11 at 11:20 +0900, Takashi Yano wrote:
> Is this the same as your problem?

Yeah, it could be. Could this result in fork error messages as we are
seeing all over the place?

> If so, it goes without stopping 1 minute with patch:
> https://cygwin.com/ml/cygwin-patches/2020-q1/msg00148.html

Nice trick to use pinfo as a work-around, but i could fear that others
also call kill() at this critical time. Is there an actual fix for
kill() in the cooking? I could fear that tools like "make" actually
call this all over the place, especially if its aborting running
processes. And why did this work before?

> In any case, the following error occurs.
> 
> /usr/lib/gcc/x86_64-pc-cygwin/7.4.0/../../../../x86_64-pc-
> cygwin/bin/ld:test00.d: file format not recognized; treating as
> linker script

That looks OK. Ignore the actual result of the makefile, 
its just garbage :-). Its only to reproduce this error.

Thanks again for helping,

/pedro


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Peter Dons Tychsen]

On Tue, 2020-02-11 at 11:38 +0100, Peter Dons Tychsen wrote:
> processes. And why did this work before?

And why does it work when running without minnty? How does that play
into this?

Thanks,

/pedro


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Takashi Yano]

On Tue, 11 Feb 2020 11:38:35 +0100
Peter Dons Tychsen wrote:
> On Tue, 2020-02-11 at 11:20 +0900, Takashi Yano wrote:
> > Is this the same as your problem?
> 
> Yeah, it could be. Could this result in fork error messages as we are
> seeing all over the place?

No. Fork error is not seen in my environment. This could be another
problem. However, the code using kill() below is called also by fork().
So, it may be related.

I have not investigated your problem on make, so the mechanism to
cause the problem is not clear. I do not have confidence that this
patch really fixes your problem.

> > If so, it goes without stopping 1 minute with patch:
> > https://cygwin.com/ml/cygwin-patches/2020-q1/msg00148.html
> 
> Nice trick to use pinfo as a work-around, but i could fear that others
> also call kill() at this critical time. Is there an actual fix for
> kill() in the cooking? I could fear that tools like "make" actually
> call this all over the place, especially if its aborting running
> processes. And why did this work before?

The commit message says:
>  This seems to be caused by invalid pointer access which occurs
>  when the process exits during the kill() code is execuetd. This
>  patch avoids the issue by not using kill().
however, I found the real cause is that errno is accidentally set
by kill() in pty system calls. That is, the problem is not in the
kill() itself but in usage of it. Cygwin older than 3.1.0 does not
have these code in pty. 

On Tue, 11 Feb 2020 11:40:45 +0100
Peter Dons Tychsen wrote:
> And why does it work when running without minnty? How does that play
> into this?

The code which has this problem is in the pty code. So, this occurs
only in pty such as mintty, xterm, etc. Console (cygwin in command
prompt) does not use pty, so problem does not occur in console.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Ken Brown]

On 2/11/2020 8:16 AM, Takashi Yano wrote:
> On Tue, 11 Feb 2020 11:38:35 +0100
> Peter Dons Tychsen wrote:
>> On Tue, 2020-02-11 at 11:20 +0900, Takashi Yano wrote:
>>> Is this the same as your problem?
>>
>> Yeah, it could be. Could this result in fork error messages as we are
>> seeing all over the place?
> 
> No. Fork error is not seen in my environment. This could be another
> problem. However, the code using kill() below is called also by fork().
> So, it may be related.

FWIW, I'm not seeing the fork error in my environment either.  I see the hang 
prior to your kill() patch, and then no problem (except for the ld error).

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Takashi Yano]

On Tue, 11 Feb 2020 09:21:32 -0500
Ken Brown wrote:
> On 2/11/2020 8:16 AM, Takashi Yano wrote:
> > On Tue, 11 Feb 2020 11:38:35 +0100
> > Peter Dons Tychsen wrote:
> >> On Tue, 2020-02-11 at 11:20 +0900, Takashi Yano wrote:
> >>> Is this the same as your problem?
> >>
> >> Yeah, it could be. Could this result in fork error messages as we are
> >> seeing all over the place?
> > 
> > No. Fork error is not seen in my environment. This could be another
> > problem. However, the code using kill() below is called also by fork().
> > So, it may be related.
> 
> FWIW, I'm not seeing the fork error in my environment either.  I see the hang 
> prior to your kill() patch, and then no problem (except for the ld error).

I tested under Win10 1809 and found fork error occurs in that
environment. Also confirmed the kill() patch fixes the issue.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Peter Dons Tychsen]

Hi Takashi,

Thanks for your effort.

On Tue, 2020-02-11 at 22:16 +0900, Takashi Yano wrote:
> however, I found the real cause is that errno is accidentally set
> by kill() in pty system calls. That is, the problem is not in the
> kill() itself but in usage of it. Cygwin older than 3.1.0 does not
> have these code in pty. 

OK, is there a fix for that or is that the fix you already pushed? And
what is wrong with the usage of kill(). How can kill() be used
incorrectly?

Is there are more things here it would be great to get them in for the
3.1.3 release.

Thanks,

/pedro


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Takashi Yano]

On Tue, 11 Feb 2020 22:31:12 +0100
Peter Dons Tychsen wrote:
> On Tue, 2020-02-11 at 22:16 +0900, Takashi Yano wrote:
> > however, I found the real cause is that errno is accidentally set
> > by kill() in pty system calls. That is, the problem is not in the
> > kill() itself but in usage of it. Cygwin older than 3.1.0 does not
> > have these code in pty. 
> 
> OK, is there a fix for that or is that the fix you already pushed? And
> what is wrong with the usage of kill(). How can kill() be used
> incorrectly?

POSIX system calls set global variable errno to appropriate
value when the system calls fail.
http://man7.org/linux/man-pages/man3/errno.3.html

My failure in pty code was that I used kill() in pty system
calls. kill() can be used check if the process is still alive
by passing signal number of 0 to the second argument. It returns
-1 if the process already exited. However, in this case errno
is set to ESRCH. As a result, the errno value which pty system
call set is accidentally overwritten by kill().

The patch was already accepted and pushed to git repository.
I hope cygwin 3.1.3 which applied this patch will be released
shortly.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

On Feb 12 11:24, Takashi Yano wrote:
> On Tue, 11 Feb 2020 22:31:12 +0100
> Peter Dons Tychsen wrote:
> > On Tue, 2020-02-11 at 22:16 +0900, Takashi Yano wrote:
> > > however, I found the real cause is that errno is accidentally set
> > > by kill() in pty system calls. That is, the problem is not in the
> > > kill() itself but in usage of it. Cygwin older than 3.1.0 does not
> > > have these code in pty. 
> > 
> > OK, is there a fix for that or is that the fix you already pushed? And
> > what is wrong with the usage of kill(). How can kill() be used
> > incorrectly?
> 
> POSIX system calls set global variable errno to appropriate
> value when the system calls fail.
> http://man7.org/linux/man-pages/man3/errno.3.html
> 
> My failure in pty code was that I used kill() in pty system
> calls. kill() can be used check if the process is still alive
> by passing signal number of 0 to the second argument. It returns
> -1 if the process already exited. However, in this case errno
> is set to ESRCH. As a result, the errno value which pty system
> call set is accidentally overwritten by kill().
> 
> The patch was already accepted and pushed to git repository.
> I hope cygwin 3.1.3 which applied this patch will be released
> shortly.

For a first test, please try the latest developer snapshot on
https://cygwin.com/snapshots/


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Takashi Yano]

On Wed, 12 Feb 2020 11:54:02 +0100
Peter Dons Tychsen wrote:
> Hi Takashi,
> 
> On Wed, 2020-02-12 at 11:24 +0900, Takashi Yano wrote:
> > My failure in pty code was that I used kill() in pty system
> > calls. kill() can be used check if the process is still alive
> > by passing signal number of 0 to the second argument. It returns
> > -1 if the process already exited. However, in this case errno
> > is set to ESRCH. As a result, the errno value which pty system
> > call set is accidentally overwritten by kill().
> 
> OK, thanks for clearing all that up. Just wanted to make sure kill()
> was not buggy :-).
> 
> I will test the committed patch and the upcoming release vigerously to
> see if there are still bugs to be found in this area. Not sure why i
> get fork() errors and you do not (my tests with the example i made
> always causes fork() errors in 3.1.2).
> 
> Thanks for all the help and your snappy replies,

Could you please try latest snapshot?
https://cygwin.com/snapshots/

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Peter Dons Tychsen]

Hi Takashi,

On Thu, 2020-02-13 at 13:13 +0900, Takashi Yano wrote:
> Could you please try latest snapshot?
> https://cygwin.com/snapshots/

The commit regarding use of kill() does the trick.
It's a go from here :-).

Thanks for all the help,

/pedro


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: fork: Resource temporarily unavailable errors after upgrading cygwin packages

[Takashi Yano]

On Thu, 13 Feb 2020 22:42:31 +0100
Peter Dons Tychsen wrote:
> On Thu, 2020-02-13 at 13:13 +0900, Takashi Yano wrote:
> > Could you please try latest snapshot?
> > https://cygwin.com/snapshots/
> 
> The commit regarding use of kill() does the trick.
> It's a go from here :-).
> 
> Thanks for all the help,

Thanks for testing.

I had looked into the make problem further, and found this
was not caused by the errno overwriting, but by calling
sig_dispatch_pending() in fixup_after_exec() via kill().

fixup_after_exec() is called from dll_crt0_0(). It seems
that calling sig_dispatch_pending() from dll_crl0_0()
causes hangs everywhere (even outside PTY).

I am not sure what exactly does sig_dispatch_pending()
and why this causes the problem, anyway, using pinfo()
rather than kill() solves the issue.

Thanks.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple