From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.ispras.ru (mail.ispras.ru [83.149.199.84]) by sourceware.org (Postfix) with ESMTPS id 939C33858C54 for ; Mon, 11 Apr 2022 13:27:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 939C33858C54 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=ispras.ru Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=ispras.ru Received: from mail.ispras.ru (unknown [83.149.199.84]) by mail.ispras.ru (Postfix) with ESMTPSA id C48E64076242; Mon, 11 Apr 2022 13:27:37 +0000 (UTC) MIME-Version: 1.0 Date: Mon, 11 Apr 2022 16:27:37 +0300 From: Alexey Izbyshev To: cygwin@cygwin.com Subject: Re: Deadlock of the process tree when running make In-Reply-To: <80ad73a8-16f0-81bd-aa92-1cd22550c2ba@SystematicSw.ab.ca> References: <9388316255ada0e0fcb2d849cce5a894@ispras.ru> <23e627a3-39a7-e22f-6372-65d50f1a63d6@SystematicSw.ab.ca> <5c00c0cd21bac6b4564fe5d2721520f1@ispras.ru> <80ad73a8-16f0-81bd-aa92-1cd22550c2ba@SystematicSw.ab.ca> User-Agent: Roundcube Webmail/1.4.4 Message-ID: X-Sender: izbyshev@ispras.ru Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_00, DOS_RCVD_IP_TWICE_B, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Apr 2022 13:27:50 -0000 On 2022-04-08 20:04, Brian Inglis wrote: > On 2022-04-08 02:42, Alexey Izbyshev wrote: >> There is also an additional detail that I forgot to mention: in the >> stack trace of all leaf processes as displayed by ProcessHacker, it >> seems that the executable entry point is not reached yet. The only >> non-Windows-DLL location is in cygwin1.dll, so I suspect that all >> processes hang at early initialization in Cygwin's DLL entry point. > > That sounds like BLODA interference from AntiVirus programs: > > https://cygwin.com/faq/faq.html#faq.using.bloda > > and can also happen if you use Windows AD, and your users have a lot > of rights, and a slow server, firewall filtering, or network link, but > known issues were fixed a few releases ago. > It seems that a potential cause of the hang has been identified in another discussion thread, and I'm now testing a patched version provided by Takashi Yano. Anyway, thanks for your time! And just in case the identified cause turns out to be wrong, I'm answering your questions below. We don't use any third-party AV products on this machine (it's an internal box used only for CI), and we've disabled Real-Time Protection in the Windows AV (it causes a terrible performance degradation, something like 1.5-2 times). > Any idea how much address space is used by Cygwin DLLs, and memory by > all the processes running: run rebase -is to see if you could be out > of address space for Cygwin and DLLs, and how much is left for > processes? > > Do you have a decent amount of memory free on your system while > running, and Windows paging space allocated to back it up - total > twice memory, and do you have multiple drives to spread it across? > Check your system memory and paging activity while those processes are > running. > The peak memory consumption of our tests never exceeds 30% of RAM. Also, Cygwin is used (almost) exclusively for the test harness (the actual software under testing is native), and there are no heavyweight processes in it, mostly just make, bash and some coreutils. So I don't think we could hit address space issues even on 32-bit Cygwin. > Could you try installing Cygwin64 packages and running those instead > of Cygwin32 (recommended as Cygwin32 support will be dropped next > release) as there is more address space available as well as usable > memory for processes? We test both 32-bit and 64-bit builds of our software, and a couple of tests need to run (Cygwin) make under debugging. Because a 32-bit process can't debug a 64-bit one, we simply use 32-bit Cygwin for both cases. But if need to reproduce under 64-bit Cygwin arises, I can simply exclude the problematic tests (they're unlikely to be relevant to the hang). Alexey