From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.135]) by sourceware.org (Postfix) with ESMTPS id 606303861002 for ; Fri, 28 Aug 2020 13:36:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 606303861002 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=cygwin.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=corinna-cygwin@cygwin.com Received: from calimero.vinschen.de ([217.91.18.234]) by mrelayeu.kundenserver.de (mreue009 [212.227.15.167]) with ESMTPSA (Nemesis) id 1Mzhar-1kXbMQ3e5J-00vibx for ; Fri, 28 Aug 2020 15:36:43 +0200 Received: by calimero.vinschen.de (Postfix, from userid 500) id 43BFAA83A79; Fri, 28 Aug 2020 15:36:43 +0200 (CEST) Date: Fri, 28 Aug 2020 15:36:43 +0200 From: Corinna Vinschen To: cygwin@cygwin.com Subject: Re: Sv: Sv: Limit for number of child processes Message-ID: <20200828133643.GK3272@calimero.vinschen.de> Reply-To: cygwin@cygwin.com Mail-Followup-To: cygwin@cygwin.com References: <000401d67ba0$8b1f33b0$a15d9b10$@gmail.com> <20200826175724.GQ3272@calimero.vinschen.de> <000701d67c6c$10bcf720$3236e560$@gmail.com> <6d698a32-06bb-a47b-58e6-ceeecca722c9@cornell.edu> <001101d67d16$aa5db9f0$ff192dd0$@gmail.com> <6242991f-e5ec-150b-bd6c-15a8c348c7cf@cornell.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <6242991f-e5ec-150b-bd6c-15a8c348c7cf@cornell.edu> X-Provags-ID: V03:K1:GqPf6K7fq5JP8vLBDur1p3QQPhwP7d7DmvUuHdDLE2HofvAUkpx PcuRzcJnrZmUVyooCXGSxOU9Ufkqw7aZjsZoNF8K901Px6wgnfnluJN8b99zhhPRArT9LCX c5cmNU6jBI3+7Jf+ey1ogFSlACdZlGZmIfESHqbHeMvwN06IAjswdZeGSKW9Dok3UqxmVqx 0zpuKgYnhVwsyu0359phA== X-UI-Out-Filterresults: notjunk:1;V03:K0:vGIwFQ8VcpA=:51g84ujsYyTFuw5Iy7NL2M MW0j3KH5OEPcc02a36igTQKWUPqo6eEgiKZCxzYzZfFemiljeUiwys+ERRmEmGt1igK0sCuUg 7co15i37dY0zC+jRMZdX1hHBsxxpUhSS3UvGPtQvlbpDh3MYqtflr263nQeiWVnHbyWYHl0sA KiX/RoBcBmpANnaA1fCfVGuxnJCvtf9n7vnnCX8T8/yeYb2VwjUE578xQVxGYejOsfRA9+1be gBKTMDKoHl3CTiBOkNEt/v9gSdZV9RTte8KrncAxlR9wUFdO/iToIU6i8wcDCfKK7PI10XR/s SeFXJx+fGcn82hAq7H0hfsaLuu5JVCh2ycPXoK/cqxTg/2oYpH9XKw1+xGRQWaKmXsaFHd59t BuawoJjtiTa9We7ZKUACnZi8CZ+Jx+n03B0rWYgiMoHV/99EEja2yEs78agXL4N8l/tURjQpr Mn/UaKQur2VIxwwEhNwmQAru9gIASZRW8AqX6XCHFtCYyqOUZ8FWZbtIA5IT5zYg1GEPw2QLO wqmzDwkvn1Yu6cGK1rTa/zxmQKzXdBscemJsx1e4v3FKq6vcbGlb5WZDEOQIQKWJgc5GpMpuF ZBE9AFrQnaQzClUeGKe92MfnPY/czeZyrW6Tb//yuvcLGCmec2huDlqQ/5H0unvGIUuh7Aw6K tBSZuR6OAU1oglacGcktJa156maM4V4+lJuSXCxlgSymNOnxh+VvLU6VnT+mh3V/ReAQa2wIz 9GzQi7IjKxHaAsyec9m+JUtk8C0LeuozH/SV9BgQWdSXJWxrbfaFIheXhP1q6f94PClUPF4At gM6f0ldl1jVcBCOjQzThmGb7SZMPo/uW++zbGDPw2yACv1up1WqQbR0ABfbzYE4FaQ3mzBjsg 3TSTS99FxCpOzEpjAJPw== X-Spam-Status: No, score=-99.9 required=5.0 tests=BAYES_00, GOOD_FROM_CORINNA_CYGWIN, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NEUTRAL, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2020 13:36:47 -0000 On Aug 28 08:29, Ken Brown via Cygwin wrote: > On 8/28/2020 4:38 AM, sten.kristian.ivarsson@gmail.com wrote: > > > > > > It seems like there's a limit of the number of possible child > > > > > > processes defined to 256 with 'NPROCS' in > > > > > > //winsup/cygwin/child_info.h used in 'cprocs' in > > > > > > //winsup/cygwin/sigproc.cc > > > > > > > > > > > > 256 is quite few possible children in an enterprise environment and > > > > > > perhaps the limit should be limited by the physical resources or > > > > > possibly Windows ? > > > > > > > > > > The info has to be kept available in the process itself so we need > > > > > this array of NPROCS * sizeof (pinfo). > > > > > > > > > > Of course, there's no reason to use a static array, the code could > > > > > just as well use a dynamically allocated array or a linked list. > > > > > It's just not the way it is right now and would need a patch or > > > rewrite. > > > > > [...] > > > > A linked list could be used if you wanna optimize (dynamic) memory > > > > usage but an (amortized) array would probably provide faster linear > > > > search but I guess simplicity of the code and external functionality > > > > is the most important demands for this choice > > > > > > Any change here (aside from just increasing NPROCS) would have to be done > > > with care to avoid a performance hit. I looked at the history of changes > > > to sigproc.cc, and I found commit 4ce15a49 in 2001 in which a static array > > > something like cprocs was replaced by a dynamically allocated buffer in > > > order to save DLL space. This was reverted 3 days later (commit e2ea684e) > > > because of performance issues. > > > > I wonder what kind of performance issue ? > > [...] > I don't know for sure, but I doubt if it had anything to do with memory > access. My guess is that the performance hit came from the need to free the > allocated memory after every fork call (see sigproc_fixup_after_fork). Either way, I rewrote this partially so we now have a default array size for 255 child processes on 32 bit and 1023 child processes on 64 bit. The new code is mainly a minor update in that it convertes the code directly accessing stuff into using a class, encapsulating the mechanism used under the hood behind a class barrier and access methods. As POC, I added a bit of code to maintain a second array, which is only allocated (using HeapAlloc so as not to spill into the child processes) if the default array overflows. This second array adds room for another 1023 (32 bit) or 4095 (64 bit) child processes, raising the number of max child processes per process to 1278 on 32 bit and 5118 on 64 bit. My STC just forking like crazy overflowed my 4 Gigs RAM + 2.5 Gigs pagefile after roughly 1450 child processes. I'm pretty confident that this POC implementation is sufficient for a while, even in enterprise scenarios. And if not, we can now easily tweak the numbers without having to tweak much of the code. For testing purposes I uploaded a developer snapshot to https://cygwin.com/snapshots/ Corinna -- Corinna Vinschen Cygwin Maintainer