From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sonic311-23.consmr.mail.ne1.yahoo.com (sonic311-23.consmr.mail.ne1.yahoo.com [66.163.188.204]) by sourceware.org (Postfix) with ESMTPS id B659D385E02A for ; Thu, 2 Apr 2020 12:47:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org B659D385E02A X-YMail-OSG: Szd3kzkVM1lsQlVVpgAM3G2.l1K0_suU4h7Q0pKe7TvCOT1BxJoppWTd7TYTK6h _S_dpnHGjVBe4FAV0E04aQHkOoGWA5JUN78fc9E8v9w4gq5_ft8wdsB1rZxRHPQGkq_nj4ABWvUE NETxLUjfXuyDXXaswMFf_jWPoiHwFsbHTKIrp6LwhuWCjOlF9G1vd.i_3t1tCdxKgKeaux2tfJjI 93gixioPx2sv5YGt1aZZdOY00qX_koYlLRNWGaMKMQ1wWdXCvdQb_5sG1MrlCU_IoHxDJIyMwf5d dUo3EC24QjzYH.qYKh_AjNjq39dY0mjva97k2BttcSlqHvC9pHyeeF8FS__GIhh_D9.QVthcLL2K BYeHWCnnmxELhjgd50_6I1V.4esgd.ySNy4h3KYAbAuaXPZAAw5Yjo4FY1FHnivNQdRn6XZDtxI0 HTkWeUFLwygvM0Xo4LP1Kp8.BNpQ_ksnhpTsA_Vefe9ajCIReDfg286w3UQjuU0SX.qfLaNyhJmT XuohgbY_QlTf9B2qmPq5_aPs9bP5LK7sJ1zPPRPHBD_lW1tkmctpx20.iiOFNoDZfZZBtJic9OGo jz85qcgCVAp8.5iFCBS_w4LdytLGm770mgHH847zzAcsErgEkFNgSHFsORvjjV9doB24zwVS9Qhg .JWbslTxGt6bstpQfXFSrRhMcUN4rncXCvLMHyPm7TkZqW5B1lZMaBT3v9aBYvdevRV8WG97jU.G 4svfWs1ULv4qAiLL1OrfgGz4cgNzvC7CB11e1gm4yR4TWboEZIOoZnAfY8C6qztIkrAhC9jQRPeq Ja16vulIwd7g8XqL.vS8VW3EOI3HKRAOB2GbQDqoyKRmyludEE01mi6P.igYdZIiudiSP1AtT0mq pLBViNRjsqUUpMYZ9VfzcDW.PjwnvyjiXuhuiLprNbCFp0IIaFpjRc7ugy9nGZ_isdLJRTAR3Iwc rf8X1vyKIZdLzsil3VgZbqPXRJ8Og.gunXA8daEfJQYBIxZ_cocdjs1sFHrCSbxzPTeQdKNLcKnG 6IQxRWV_mTCxYf7CfS7jChbDjrXW7FAIb.2YcxzElxugl2NCf3AKtT0BEXsPCztps47J58AESEs9 iThWivXjW56Q8p7_kzML0J0lP_JVR1KMN7ByRCOI2jVxiKfcw13a4gSz4kY0Wnem93JXzrL5QE3D 1qm_.E9PHcUuJpQs5z9hKTPy6jZWPCKC6Ctjb0lSpvMU58tDkTQLFxtR9VLZhqpiKI5.7ENUd6bd 5ESvcUywcx07jK9A5HcWUasUvcM.mGxmLLZuxHzCt8HGL2LgIddHuM8hmP8_z7RkAtrYbPJ_wXf8 wM2Q_wT7EwFt4eUIne9sNwbVeqwhI4gt1.W8Ag01wVf1UfmaeCfv5JYg6g9zlIYJrejfT0fBDe6B smqVjPrWt6t6.5sBcV1a7FYBz1vTOGIGl Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.ne1.yahoo.com with HTTP; Thu, 2 Apr 2020 12:47:07 +0000 Date: Thu, 2 Apr 2020 12:47:02 +0000 (UTC) From: Gregery Barton Reply-To: "gregery20@yahoo.com.au" To: "sten.kristian.ivarsson@gmail.com" , Kristian Ivarsson via Cygwin , 'Ken Brown' Cc: 'cygwin' Message-ID: <1644706448.958021.1585831622425@mail.yahoo.com> In-Reply-To: <000901d608c5$86361880$92a24980$@gmail.com> References: <1b1401d60296$2769e690$763db3b0$@gmail.com> <716e2076-f607-454e-2723-937c3959e2a3@cornell.edu> <18be01d602ab$0bbfca30$233f5e90$@gmail.com> <35b43b59-6410-f21f-710c-385e39cbae0b@cornell.edu> <005201d603ba$2bc8ab20$835a0160$@gmail.com> <472d1df6-531a-ebd7-4ffa-583a06e270ff@cornell.edu> <00b901d60447$7ecb4c50$7c61e4f0$@gmail.com> <00e001d604f9$d0aa0720$71fe1560$@gmail.com> <8c6c5655-c162-8361-9f44-376bbd7cf114@cornell.edu> <3fe06192-7300-382a-8c98-f1bc2ff81e36@cornell.edu> <003701d607a0$c975f140$5c61d3c0$@gmail.com> <249be61e-da8a-7da1-ca67-0c4c6433a415@cornell.edu> <000a01d60802$d1525900$73f70b00$@gmail.com> <001601d60848$fcffd320$f6ff7960$@gmail.com> <7b5b058e-5047-4d49-8c31-5553056f3845@cornell.edu> <7897bc10-439d-64aa-c173-f0bf4ec8246 8@cornell.edu> <7897bc10-439d-64aa-c173-f0bf4ec82468@cornell.edu> <000901d608c5$86361880$92a24980$@gmail.com> Subject: Re: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Sv: Named pipes and multiple wri MIME-Version: 1.0 X-Mailer: WebService/1.1.15585 YahooMailAndroidMobile YMobile/1.0 (com.yahoo.mobile.client.android.mail/6.5.3; Android/10; QP1A.190711.020; beyond1; samsung; SM-G973F; 5.58; 2042x1080; ) X-Spam-Status: No, score=0.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, FREEMAIL_REPLYTO_END_DIGIT, HTML_MESSAGE, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, REPTO_QUOTE_YAHOO, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2020 12:47:10 -0000 How do I unsubscribed from cygwin mailing lists? I tried simple unsubscribe= but it's not so simple. I don't have a password and it won't send me one. Sent from Yahoo7 Mail on Android=20 =20 On Thu, 2 Apr 2020 at 7:06 pm, Kristian Ivarsson via Cygwin wrote: > On 4/1/2020 2:34 PM, Ken Brown via Cygwin wrote: > > On 4/1/2020 1:14 PM, sten.kristian.ivarsson@gmail.com wrote: > >>> On 4/1/2020 4:52 AM, sten.kristian.ivarsson@gmail.com wrote: > >>>>> On 3/31/2020 5:10 PM, sten.kristian.ivarsson@gmail.com wrote: > >>>>>>> On 3/28/2020 10:19 PM, Ken Brown via Cygwin wrote: > >>>>>>>> On 3/28/2020 11:43 AM, Ken Brown via Cygwin wrote: > >>>>>>>>> On 3/28/2020 8:10 AM, sten.kristian.ivarsson@gmail.com wrote: > >>>>>>>>>>> On 3/27/2020 10:53 AM, sten.kristian.ivarsson@gmail.com wrote= : > >>>>>>>>>>>>> On 3/26/2020 7:19 PM, Ken Brown via Cygwin wrote: > >>>>>>>>>>>>>> On 3/26/2020 6:39 PM, Ken Brown via Cygwin wrote: > >>>>>>>>>>>>>>> On 3/26/2020 6:01 PM, sten.kristian.ivarsson@gmail.com wrote: > >>>>>>>>>>>>>>>> The ENIXIO occurs when parallel child-processes > >>>>>>>>>>>>>>>> simultaneously using O_NONBLOCK opening the descriptor. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This is consistent with my guess that the error is > >>>>>>>>>>>>>>> generated by fhandler_fifo::wait.=C2=A0 I have a feeling = that > >>>>>>>>>>>>>>> read_ready should have been created as a manual-reset > >>>>>>>>>>>>>>> event, and that more care is needed to make sure it's > >>>>>>>>>>>>>>> set > >> when it should be. > >> > >> [snip] > >> > >>>>>>>> Never mind.=C2=A0 I was able to reproduce the problem and find t= he cause. > >>>>>>>> What happens is that when the first subprocess exits, > >>>>>>>> fhandler_fifo::close resets read_ready.=C2=A0 That causes the se= cond > >>>>>>>> and subsequent subprocesses to think that there's no reader > >>>>>>>> open, so their attempts to open a writer with O_NONBLOCK fail with ENXIO. > >> > >> [snip] > >> > >>>> I wrote in a previous mail in this topic that it seemed to work > >>>> fine for me as well, but when I bumped up the numbers of writers > >>>> and/or the number of messages (e.g. 25/25) it starts to fail again > >> > >> [snip] > >> > >>> Yes, it is a resource issue.=C2=A0 There is a limit on the number of > >>> writers > >> that can be open at one > >>> time, currently 64.=C2=A0 I chose that number arbitrarily, with no id= ea > >>> what > >> might actually be > >>> needed in practice, and it can easily be changed. > >> > >> Does it have to be a limit at all ? We would rather see that the > >> application decide how much resources it would like to use. In our > >> particular case there will be a process-manager with an incoming pipe > >> that possible several thousands of processes will write to > > > > I agree. > > > >> Just for fiddling around (to figure out if this is the limit that > >> make other things work a bit odd), where's this 64 limit defined now ? > > > > It's MAX_CLIENTS, defined in fhandler.h.=C2=A0 But there seem to be oth= er > > resource issues also; simply increasing MAX_CLIENTS doesn't solve the > > problem.=C2=A0 I think there are also problems with the number of threa= ds, > > for example.=C2=A0 Each time your program forks, the subprocess inherit= s > > the rfd file descriptor and its "fifo_reader_thread" starts up.=C2=A0 T= his > > is unnecessary for your application, so I tried disabling it (in > fhandler_fifo::fixup_after_fork), just as an experiment. > > > > But then I ran into some deadlocks, suggesting that one of the locks > > I'm using isn't robust enough.=C2=A0 So I've got a lot of things to wor= k on. > > > >>> In addition, a writer isn't recognized as closed until a reader > >>> tries to > >> read and gets an error. > >>> In your example with 25/25, the list of writers quickly gets to 64 > >>> before > >> the parent ever tries > >>> to read. > >> > >> That explains the behaviour, but should there be some error returned > >> from open/write (maybe it is but I'm missing it) ? > > > > The error is discovered in add_client_handler, called from > > thread_func.=C2=A0 I think you'll only see it if you run the program un= der > > strace.=C2=A0 I'll see if I can find a way to report it.=C2=A0 Currentl= y, > > there's a retry loop in fhandler_fifo::open when a writer tries to > > open, and I think I need to limit the number of retries and then error out. >=20 > I pushed a few improvements and bug fixes, and your 25/25 example now run= s without a > problem.=C2=A0 I increased MAX_CLIENTS to 1024 just for the sake of this example, but I'll > work on letting the number of writers increase dynamically as needed. I pulled it and tried it out and yes, the sample test program with 25/25 worked well and a whole bunch of our unit-tests passed with ok result now We still do have some issues, but I cannot yet tell if they are related to named pipes or not It is great that you're looking into a totally dynamic solution Kristian > Ken -- Problem reports:=C2=A0 =C2=A0 =C2=A0 https://cygwin.com/problems.html FAQ:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://= cygwin.com/faq/ Documentation:=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://cygwin.com/docs.html Unsubscribe info:=C2=A0 =C2=A0 https://cygwin.com/ml/#unsubscribe-simple = =20