From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from conssluserg-05.nifty.com (conssluserg-05.nifty.com [210.131.2.90]) by sourceware.org (Postfix) with ESMTPS id 7603E3858414 for ; Wed, 8 Sep 2021 11:33:04 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 7603E3858414 Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=nifty.ne.jp Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nifty.ne.jp Received: from Express5800-S70 (z221123.dynamic.ppp.asahi-net.or.jp [110.4.221.123]) (authenticated) by conssluserg-05.nifty.com with ESMTP id 188BWXL4029646 for ; Wed, 8 Sep 2021 20:32:34 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conssluserg-05.nifty.com 188BWXL4029646 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.ne.jp; s=dec2015msa; t=1631100754; bh=qQJlUdPQjswtaas9o51Nv3VXkOV7CwAcScu9lqgXOxQ=; h=Date:From:To:Subject:In-Reply-To:References:From; b=W+8PM1CvxxLYApl3okM6sLn9Np4A6TeWZxvwA8X0GnyidGJEVRCOwD0OaDw9sMV+G puMQW+tRfk+TOXttOs32LucHMnqRAYRktoWIJm+nwxsop0uuvl9SZj1uMpCRGwQwEL z7QsAS8AhXqzLI91OzkpAUIwK0cXT5sxmspl/vsgsW4Wx+dkvMMLWDSJROmMHFufzR zkpeHE6gPxP34JlLZHH0oM4obVXwvxHiPAXmODchJNhMU7gZczD/a/hNO9PoYNIG1b MPsgvZ0qIGWO5EoToOH+dyx1MwTqAHNPZaK6ChRoFaRQkzlt7atLxmTClrgWiGIqkZ 9iFXSDxqXLlNw== X-Nifty-SrcIP: [110.4.221.123] Date: Wed, 8 Sep 2021 20:32:34 +0900 From: Takashi Yano To: cygwin-developers@cygwin.com Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? Message-Id: <20210908203234.8cc6215e05108d9ad0507bd3@nifty.ne.jp> In-Reply-To: References: <20210901091652.6bf3cccbcaed4a22f6ffa6b0@nifty.ne.jp> <20210901172339.1039604b7067e0492534a20f@nifty.ne.jp> <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; i686-pc-mingw32) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-5.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2021 11:33:06 -0000 Hi Corinna, On Thu, 2 Sep 2021 21:35:21 +0200 Corinna Vinschen wrote: > On Sep 2 21:00, Corinna Vinschen wrote: > > On Sep 2 09:01, Ken Brown wrote: > > > On 9/2/2021 4:17 AM, Corinna Vinschen wrote: > > > > What if the readers never request more than, say, 50 or even 25% of the > > > > available buffer space? Our buffer is 64K and there's no guarantee that > > > > any read > PIPE_BUF (== 4K) is atomic anyway. This can work without > > > > having to check the other side of the pipe. Something like this, > > > > ignoring border cases: > > > > > > > > pipe::create() > > > > { > > > > [...] > > > > mutex = CreateMutex(); > > > > } > > > > > > > > pipe::raw_read(char *buf, size_t num_requested) > > > > { > > > > if (blocking) > > > > { > > > > WFSO(mutex); > > > > NtQueryInformationFile(FilePipeLocalInformation); > > > > if (!fpli.ReadDataAvailable > > > > && num_requested > fpli.InboundQuota / 4) > > > > num_requested = fpli.InboundQuota / 4; > > > > NtReadFile(pipe, buf, num_requested); > > > > ReleaseMutex(mutex); > > > > } > > > > } > > > > > > > > It's not entirely foolproof, but it should fix 99% of the cases. > > > > > > I like it! > > > > > > Do you think there's anything we can or should do to avoid a deadlock in the > > > rare cases where this fails? The only thing I can think of immediately is > > > to always impose a timeout if select is called with infinite timeout on the > > > write side of a pipe, after which we report that the pipe is write ready. > > > After all, we've lived since 2008 with a bug that caused select to *always* > > > report write ready. > > > > Indeed. Hmm. What timeout are you thinking of? Seconds? Minutes? > > > > > Alternatively, we could just wait and see if there's an actual use case in > > > which someone encounters a deadlock. > > > > Or that. Fixing up select isn't too hard in that case, I guess. > > It's getting too late again. I drop off for tonight, but I attached > my POC code I have so far. It also adds the snippets from my previous > patch which fixes stuff Takashi found during testing. It also fixes > something which looks like a bug in raw_write: > > - ptr = ((char *) ptr) + chunk; > + ptr = ((char *) ptr) + nbytes_now; > > Incrementing ptr by chunk bytes while only nbytes_now have been written > looks incorrect. > > As for the reader, it makes the # of bytes to read dependent on the > number of reader handles. I don't know if that's such a bright idea, > but this can be changed easily. > > Anyway, this runs all my testcases successfully but they are anything > but thorough. > > Patch relativ to topic/pipe attached. Would you both mind to take a > scrutinizing look? Sorry for replying to old post. As for this patch, read_mtx was introduced. This handle is initialized only for read pipe. However, this seems to be NULL even without initialization in write pipe. I wonder why initializing read_mtx in the constructor is not necessary. How do you guarantee that read_mtx is NULL on the write pipe? -- Takashi Yano