From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from conssluserg-03.nifty.com (conssluserg-03.nifty.com [210.131.2.82]) by sourceware.org (Postfix) with ESMTPS id AEB7C385840F for ; Fri, 3 Sep 2021 10:01:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org AEB7C385840F Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=nifty.ne.jp Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nifty.ne.jp Received: from Express5800-S70 (z221123.dynamic.ppp.asahi-net.or.jp [110.4.221.123]) (authenticated) by conssluserg-03.nifty.com with ESMTP id 183A0kiF002763 for ; Fri, 3 Sep 2021 19:00:47 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conssluserg-03.nifty.com 183A0kiF002763 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.ne.jp; s=dec2015msa; t=1630663247; bh=U8Kzc4JGZUX4AEgD2aTZWlm7qk+viYem+NrM5uJx0RQ=; h=Date:From:To:Subject:In-Reply-To:References:From; b=kthY7424FNrGDeBG/xFfzmaCBlT86zxTbD/1wwLhUhCL5GB6Bw1YAE6hLKHqF7s/2 MrRd7xooRZZUiPwvcEQOU31F12+viMLAEkoijHclqRQyWiJZTHsFug+OmY3KxoDzLS McYtHgqaYevgAjuDXXnRcg1H3ZffTTm5cK99WFVCbxm9xNfl/puMSNLZ7iVx6Pi8bB xlYn+RMVxo5dLpBRJXWL2phPA/8DGqzm+KPw6csiqv7NmfxkQEc1PsN4M5p0T4Ejt8 ENNftkoQ7CLaqjWSdXOfoYHhkzrT9ELX8O1OtpNpqvt6BjfLj276Nu3ej/V4UU88Co 2DNtxjYL+R3CQ== X-Nifty-SrcIP: [110.4.221.123] Date: Fri, 3 Sep 2021 19:00:46 +0900 From: Takashi Yano To: cygwin-developers@cygwin.com Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? Message-Id: <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> In-Reply-To: References: <20210901091652.6bf3cccbcaed4a22f6ffa6b0@nifty.ne.jp> <20210901172339.1039604b7067e0492534a20f@nifty.ne.jp> <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; i686-pc-mingw32) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2021 10:01:18 -0000 On Thu, 2 Sep 2021 21:35:21 +0200 Corinna Vinschen wrote: > On Sep 2 21:00, Corinna Vinschen wrote: > > On Sep 2 09:01, Ken Brown wrote: > > > On 9/2/2021 4:17 AM, Corinna Vinschen wrote: > > > > What if the readers never request more than, say, 50 or even 25% of the > > > > available buffer space? Our buffer is 64K and there's no guarantee that > > > > any read > PIPE_BUF (== 4K) is atomic anyway. This can work without > > > > having to check the other side of the pipe. Something like this, > > > > ignoring border cases: > > > > > > > > pipe::create() > > > > { > > > > [...] > > > > mutex = CreateMutex(); > > > > } > > > > > > > > pipe::raw_read(char *buf, size_t num_requested) > > > > { > > > > if (blocking) > > > > { > > > > WFSO(mutex); > > > > NtQueryInformationFile(FilePipeLocalInformation); > > > > if (!fpli.ReadDataAvailable > > > > && num_requested > fpli.InboundQuota / 4) > > > > num_requested = fpli.InboundQuota / 4; > > > > NtReadFile(pipe, buf, num_requested); > > > > ReleaseMutex(mutex); > > > > } > > > > } > > > > > > > > It's not entirely foolproof, but it should fix 99% of the cases. > > > > > > I like it! > > > > > > Do you think there's anything we can or should do to avoid a deadlock in the > > > rare cases where this fails? The only thing I can think of immediately is > > > to always impose a timeout if select is called with infinite timeout on the > > > write side of a pipe, after which we report that the pipe is write ready. > > > After all, we've lived since 2008 with a bug that caused select to *always* > > > report write ready. > > > > Indeed. Hmm. What timeout are you thinking of? Seconds? Minutes? > > > > > Alternatively, we could just wait and see if there's an actual use case in > > > which someone encounters a deadlock. > > > > Or that. Fixing up select isn't too hard in that case, I guess. > > It's getting too late again. I drop off for tonight, but I attached > my POC code I have so far. It also adds the snippets from my previous > patch which fixes stuff Takashi found during testing. It also fixes > something which looks like a bug in raw_write: > > - ptr = ((char *) ptr) + chunk; > + ptr = ((char *) ptr) + nbytes_now; > > Incrementing ptr by chunk bytes while only nbytes_now have been written > looks incorrect. > > As for the reader, it makes the # of bytes to read dependent on the > number of reader handles. I don't know if that's such a bright idea, > but this can be changed easily. > > Anyway, this runs all my testcases successfully but they are anything > but thorough. > > Patch relativ to topic/pipe attached. Would you both mind to take a > scrutinizing look? Thanks. Your code seems that read() returns only the partial data even if the pipe stil has more data. Is this by design? This happes in both blocking and non-blocking case. -- Takashi Yano