From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from conssluserg-06.nifty.com (conssluserg-06.nifty.com [210.131.2.91]) by sourceware.org (Postfix) with ESMTPS id 3991C385842C for ; Fri, 3 Sep 2021 10:38:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3991C385842C Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=nifty.ne.jp Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nifty.ne.jp Received: from Express5800-S70 (z221123.dynamic.ppp.asahi-net.or.jp [110.4.221.123]) (authenticated) by conssluserg-06.nifty.com with ESMTP id 183Ac7pk030917 for ; Fri, 3 Sep 2021 19:38:08 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conssluserg-06.nifty.com 183Ac7pk030917 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.ne.jp; s=dec2015msa; t=1630665488; bh=SPfyaSufFnrJrW7lq19lprcQDqSUP2aMd4qZs/kD7+4=; h=Date:From:To:Subject:In-Reply-To:References:From; b=g6IBoCrkPuUkzerOGMrtyOTtkAd/QV/O1kdTPyTtD+Dr+v5LeP285cLDVJoFrf5Au NpbffBuL0KH8jGJtBJ1GBRe1W6AIQoD7WctvyW76WK6zDTvtWIO3z9JhPGnrJUfiSG CPn0pLwSwcZrLGBV3Hw8LexjGEqOBecBXs3aoFFizKnb+rTSCqphCvpryDI+6zM7zP 6Wd4T7DWq65M2z01AOe4znL1mjgpjn/7I0SFCulcNvnpfi2oVly766cN5JPmouTKnJ 7ymTTv66PNnkTv9bH6D5Kploi421Wdi07EcSO5+x5/vxULFQao3KwB4G3k3JnWN0kD qfODkE/UGZ41Q== X-Nifty-SrcIP: [110.4.221.123] Date: Fri, 3 Sep 2021 19:38:08 +0900 From: Takashi Yano To: cygwin-developers@cygwin.com Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? Message-Id: <20210903193808.68ed354a53ae5016f58a0a9c@nifty.ne.jp> In-Reply-To: <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> References: <20210901091652.6bf3cccbcaed4a22f6ffa6b0@nifty.ne.jp> <20210901172339.1039604b7067e0492534a20f@nifty.ne.jp> <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; i686-pc-mingw32) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2021 10:38:40 -0000 On Fri, 3 Sep 2021 19:00:46 +0900 Takashi Yano wrote: > On Thu, 2 Sep 2021 21:35:21 +0200 > Corinna Vinschen wrote: > > On Sep 2 21:00, Corinna Vinschen wrote: > > > On Sep 2 09:01, Ken Brown wrote: > > > > On 9/2/2021 4:17 AM, Corinna Vinschen wrote: > > > > > What if the readers never request more than, say, 50 or even 25% of the > > > > > available buffer space? Our buffer is 64K and there's no guarantee that > > > > > any read > PIPE_BUF (== 4K) is atomic anyway. This can work without > > > > > having to check the other side of the pipe. Something like this, > > > > > ignoring border cases: > > > > > > > > > > pipe::create() > > > > > { > > > > > [...] > > > > > mutex = CreateMutex(); > > > > > } > > > > > > > > > > pipe::raw_read(char *buf, size_t num_requested) > > > > > { > > > > > if (blocking) > > > > > { > > > > > WFSO(mutex); > > > > > NtQueryInformationFile(FilePipeLocalInformation); > > > > > if (!fpli.ReadDataAvailable > > > > > && num_requested > fpli.InboundQuota / 4) > > > > > num_requested = fpli.InboundQuota / 4; > > > > > NtReadFile(pipe, buf, num_requested); > > > > > ReleaseMutex(mutex); > > > > > } > > > > > } > > > > > > > > > > It's not entirely foolproof, but it should fix 99% of the cases. > > > > > > > > I like it! > > > > > > > > Do you think there's anything we can or should do to avoid a deadlock in the > > > > rare cases where this fails? The only thing I can think of immediately is > > > > to always impose a timeout if select is called with infinite timeout on the > > > > write side of a pipe, after which we report that the pipe is write ready. > > > > After all, we've lived since 2008 with a bug that caused select to *always* > > > > report write ready. > > > > > > Indeed. Hmm. What timeout are you thinking of? Seconds? Minutes? > > > > > > > Alternatively, we could just wait and see if there's an actual use case in > > > > which someone encounters a deadlock. > > > > > > Or that. Fixing up select isn't too hard in that case, I guess. > > > > It's getting too late again. I drop off for tonight, but I attached > > my POC code I have so far. It also adds the snippets from my previous > > patch which fixes stuff Takashi found during testing. It also fixes > > something which looks like a bug in raw_write: > > > > - ptr = ((char *) ptr) + chunk; > > + ptr = ((char *) ptr) + nbytes_now; > > > > Incrementing ptr by chunk bytes while only nbytes_now have been written > > looks incorrect. > > > > As for the reader, it makes the # of bytes to read dependent on the > > number of reader handles. I don't know if that's such a bright idea, > > but this can be changed easily. > > > > Anyway, this runs all my testcases successfully but they are anything > > but thorough. > > > > Patch relativ to topic/pipe attached. Would you both mind to take a > > scrutinizing look? > > Thanks. > > Your code seems that read() returns only the partial data even > if the pipe stil has more data. Is this by design? > > This happes in both blocking and non-blocking case. Sorry, this may only happen if pipe is blocking mode. -- Takashi Yano