Re: Deadlock of the process tree when running make

[Alexey Izbyshev <izbyshev@ispras.ru>]

On 2022-04-08 20:04, Brian Inglis wrote:
> On 2022-04-08 02:42, Alexey Izbyshev wrote:
>> There is also an additional detail that I forgot to mention: in the 
>> stack trace of all leaf processes as displayed by ProcessHacker, it 
>> seems that the executable entry point is not reached yet. The only 
>> non-Windows-DLL location is in cygwin1.dll, so I suspect that all 
>> processes hang at early initialization in Cygwin's DLL entry point.
> 
> That sounds like BLODA interference from AntiVirus programs:
> 
> 	https://cygwin.com/faq/faq.html#faq.using.bloda
> 
> and can also happen if you use Windows AD, and your users have a lot
> of rights, and a slow server, firewall filtering, or network link, but
> known issues were fixed a few releases ago.
> 

It seems that a potential cause of the hang has been identified in 
another discussion thread, and I'm now testing a patched version 
provided by Takashi Yano.

Anyway, thanks for your time! And just in case the identified cause 
turns out to be wrong, I'm answering your questions below.

We don't use any third-party AV products on this machine (it's an 
internal box used only for CI), and we've disabled Real-Time Protection 
in the Windows AV (it causes a terrible performance degradation, 
something like 1.5-2 times).

> Any idea how much address space is used by Cygwin DLLs, and memory by
> all the processes running: run rebase -is to see if you could be out
> of address space for Cygwin and DLLs, and how much is left for
> processes?
> 
> Do you have a decent amount of memory free on your system while
> running, and Windows paging space allocated to back it up - total
> twice memory, and do you have multiple drives to spread it across?
> Check your system memory and paging activity while those processes are 
> running.
> 
The peak memory consumption of our tests never exceeds 30% of RAM. Also, 
Cygwin is used (almost) exclusively for the test harness (the actual 
software under testing is native), and there are no heavyweight 
processes in it, mostly just make, bash and some coreutils. So I don't 
think we could hit address space issues even on 32-bit Cygwin.

> Could you try installing Cygwin64 packages and running those instead
> of Cygwin32 (recommended as Cygwin32 support will be dropped next
> release) as there is more address space available as well as usable
> memory for processes?

We test both 32-bit and 64-bit builds of our software, and a couple of 
tests need to run (Cygwin) make under debugging. Because a 32-bit 
process can't debug a 64-bit one, we simply use 32-bit Cygwin for both 
cases. But if need to reproduce under 64-bit Cygwin arises, I can simply 
exclude the problematic tests (they're unlikely to be relevant to the 
hang).

Alexey