From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from conssluserg-05.nifty.com (conssluserg-05.nifty.com [210.131.2.90]) by sourceware.org (Postfix) with ESMTPS id 8C669385840F for ; Fri, 3 Sep 2021 12:22:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 8C669385840F Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=nifty.ne.jp Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nifty.ne.jp Received: from Express5800-S70 (z221123.dynamic.ppp.asahi-net.or.jp [110.4.221.123]) (authenticated) by conssluserg-05.nifty.com with ESMTP id 183CM3Tx020772 for ; Fri, 3 Sep 2021 21:22:04 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conssluserg-05.nifty.com 183CM3Tx020772 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.ne.jp; s=dec2015msa; t=1630671724; bh=Dg7nMjq+JgYDf0HppzfNNn8/czWiF4HexEAPBFB0DRw=; h=Date:From:To:Subject:In-Reply-To:References:From; b=UH/nzVwwiJvU7cc3W2pT2nA7U9P/P7tM9IZtC7yf3ndVsiS+rAqcVYrdfuAwNdQhg vhVGZBq1k32rQNOFsrlai7WR2z6QFlYkO9mx5wraw16r3QRLmRHGkynFcURkOuHsl7 4N4ZbNrcHEXcEWjst/PbzJqp0Y2lPZiGZiF+m9voB0lhPXy8g8xJNGoNMOp6nmLHhe oK1wa6ghOxHgBryU37V0btuVi27C7YmPWivFhC2DwonebR6SXV4g2+62JBMu98WuTR trZ6m+oT8UbpIYXDEYaxwS0uvwpPGbmxOInFkKcVbg28BV9YJ9Doa2IFl0+KdQbbVC /xJpHtDSrQ/UA== X-Nifty-SrcIP: [110.4.221.123] Date: Fri, 3 Sep 2021 21:22:05 +0900 From: Takashi Yano To: cygwin-developers@cygwin.com Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? Message-Id: <20210903212205.acc2fc68cc4ffce9c1db3dd9@nifty.ne.jp> In-Reply-To: References: <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> <20210903191340.c28ae366e79ca14799bacc1f@nifty.ne.jp> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; i686-pc-mingw32) Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="Multipart=_Fri__3_Sep_2021_21_22_05_+0900_i=TsOMDf7zIoXOfX" X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2021 12:22:38 -0000 This is a multi-part message in MIME format. --Multipart=_Fri__3_Sep_2021_21_22_05_+0900_i=TsOMDf7zIoXOfX Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Fri, 3 Sep 2021 13:41:45 +0200 Corinna Vinschen wrote: > On Sep 3 13:31, Corinna Vinschen wrote: > > On Sep 3 19:13, Takashi Yano wrote: > > > On Fri, 3 Sep 2021 19:00:46 +0900 > > > Takashi Yano wrote: > > > > On Thu, 2 Sep 2021 21:35:21 +0200 > > > > Corinna Vinschen wrote: > > > > > On Sep 2 21:00, Corinna Vinschen wrote: > > > > > It's getting too late again. I drop off for tonight, but I attached > > > > > my POC code I have so far. It also adds the snippets from my previous > > > > > patch which fixes stuff Takashi found during testing. It also fixes > > > > > something which looks like a bug in raw_write: > > > > > > > > > > - ptr = ((char *) ptr) + chunk; > > > > > + ptr = ((char *) ptr) + nbytes_now; > > > > > > > > > > Incrementing ptr by chunk bytes while only nbytes_now have been written > > > > > looks incorrect. > > > > > > > > > > As for the reader, it makes the # of bytes to read dependent on the > > > > > number of reader handles. I don't know if that's such a bright idea, > > > > > but this can be changed easily. > > > > > > > > > > Anyway, this runs all my testcases successfully but they are anything > > > > > but thorough. > > > > > > > > > > Patch relativ to topic/pipe attached. Would you both mind to take a > > > > > scrutinizing look? > > > > > > > > Thanks. > > > > > > > > Your code seems that read() returns only the partial data even > > > > if the pipe stil has more data. Is this by design? > > > > > > > > This happes in both blocking and non-blocking case. > > > > > > The patch attached seems to fix the issue. > > > > I'm sorry, but I don't see what your patch is supposed to do in the > > first place. What I see is that it now calls NtQueryInformationFile > > even in the non-blocking case, which is not supposed to happen. > > > > I'm a bit puzzled what the actual bug is. > > > > The code changing len is only called if there's no data in the pipe. > > In that case we only request a partial buffer so as not to block > > the writer on select. > > > > If there *is* data in the pipe, it will just go straight to the > > NtReadFile code without changing len. > > > > Where's the mistake? > > Oh, wait. Do you mean, if we only request less than len bytes, but > after NtReadFile there's still data in the buffer, we should try to > deplete the buffer up to len bytes in a subsequent NtReadFile? Yes. I am sorry, my intent was not clear because I did more than necessary in the previous patch. Please see attached patch revised. > I thought this is unnecessary, actually, because of POSIX: > > The standard developers considered adding atomicity requirements to a > pipe or FIFO, but recognized that due to the nature of pipes and FIFOs > there could be no guarantee of atomicity of reads of {PIPE_BUF} or any > other size that would be an aid to applications portability. POSIX says: The value returned may be less than nbyte if the number of bytes left in the file is less than nbyte, if the read() request was interrupted by a signal, or if the file is a pipe or FIFO or special file and has ~~~ fewer than nbyte bytes immediately available for reading. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html If it is turned over, read() should read all data immediately available, I think. -- Takashi Yano --Multipart=_Fri__3_Sep_2021_21_22_05_+0900_i=TsOMDf7zIoXOfX Content-Type: application/octet-stream; name="pipe-corinna4-fix.patch" Content-Disposition: attachment; filename="pipe-corinna4-fix.patch" Content-Transfer-Encoding: base64 ZGlmZiAtLWdpdCBhL3dpbnN1cC9jeWd3aW4vZmhhbmRsZXJfcGlwZS5jYyBiL3dpbnN1cC9jeWd3 aW4vZmhhbmRsZXJfcGlwZS5jYwppbmRleCA4NTczMGQwMzkuLmVmNzgyM2FlNSAxMDA2NDQKLS0t IGEvd2luc3VwL2N5Z3dpbi9maGFuZGxlcl9waXBlLmNjCisrKyBiL3dpbnN1cC9jeWd3aW4vZmhh bmRsZXJfcGlwZS5jYwpAQCAtMjIyLDYgKzIyMiw3IEBAIGZoYW5kbGVyX3BpcGU6OnJhd19yZWFk ICh2b2lkICpwdHIsIHNpemVfdCYgbGVuKQogICBEV09SRCB3YWl0cmV0ID0gV0FJVF9PQkpFQ1Rf MDsKICAgYm9vbCBrZWVwX2xvb3BpbmcgPSBmYWxzZTsKICAgc2l6ZV90IG9yaWdfbGVuID0gbGVu OworICBzaXplX3QgdG90YWxfbGVuID0gMDsKIAogICBpZiAoIWxlbikKICAgICByZXR1cm47CkBA IC0yMzYsNyArMjM3LDggQEAgZmhhbmRsZXJfcGlwZTo6cmF3X3JlYWQgKHZvaWQgKnB0ciwgc2l6 ZV90JiBsZW4pCiAKICAgZG8KICAgICB7Ci0gICAgICBsZW4gPSBvcmlnX2xlbjsKKyAgICAgIGNo YXIgKnB0cjEgPSAoY2hhciAqKSBwdHIgKyB0b3RhbF9sZW47CisgICAgICBsZW4gPSBvcmlnX2xl biAtIHRvdGFsX2xlbjsKICAgICAgIGtlZXBfbG9vcGluZyA9IGZhbHNlOwogICAgICAgaWYgKGV2 dCkKIAlSZXNldEV2ZW50IChldnQpOwpAQCAtMjYwLDYgKzI2MiwxMiBAQCBmaGFuZGxlcl9waXBl OjpyYXdfcmVhZCAodm9pZCAqcHRyLCBzaXplX3QmIGxlbikKIAkJCQkJICAgRmlsZVBpcGVMb2Nh bEluZm9ybWF0aW9uKTsKIAkgIGlmIChOVF9TVUNDRVNTIChzdGF0dXMpICYmIGZwbGkuUmVhZERh dGFBdmFpbGFibGUgPT0gMCkKIAkgICAgeworCSAgICAgIGlmICh0b3RhbF9sZW4gIT0gMCkKKwkJ eworCQkgIGxlbiA9IHRvdGFsX2xlbjsKKwkJICBSZWxlYXNlTXV0ZXggKHJlYWRfbXR4KTsKKwkJ ICBicmVhazsKKwkJfQogCSAgICAgIHJlYWRlcl9jb3VudCA9IGdldF9vYmpfaGFuZGxlX2NvdW50 IChnZXRfaGFuZGxlICgpKTsKIAkgICAgICBpZiAocmVhZGVyX2NvdW50IDwgMTApCiAJCW1heF9s ZW4gPSBmcGxpLkluYm91bmRRdW90YSAvICgyICogcmVhZGVyX2NvdW50KTsKQEAgLTI2Nyw3ICsy NzUsNyBAQCBmaGFuZGxlcl9waXBlOjpyYXdfcmVhZCAodm9pZCAqcHRyLCBzaXplX3QmIGxlbikK IAkJbGVuID0gbWF4X2xlbjsKIAkgICAgfQogCX0KLSAgICAgIHN0YXR1cyA9IE50UmVhZEZpbGUg KGdldF9oYW5kbGUgKCksIGV2dCwgTlVMTCwgTlVMTCwgJmlvLCBwdHIsCisgICAgICBzdGF0dXMg PSBOdFJlYWRGaWxlIChnZXRfaGFuZGxlICgpLCBldnQsIE5VTEwsIE5VTEwsICZpbywgcHRyMSwK IAkJCSAgIGxlbiwgTlVMTCwgTlVMTCk7CiAgICAgICBpZiAoIWlzX25vbmJsb2NraW5nICgpKQog CVJlbGVhc2VNdXRleCAocmVhZF9tdHgpOwpAQCAtMjkxLDkgKzI5OSwxMyBAQCBmaGFuZGxlcl9w aXBlOjpyYXdfcmVhZCAodm9pZCAqcHRyLCBzaXplX3QmIGxlbikKIAl9CiAgICAgICBlbHNlIGlm IChOVF9TVUNDRVNTIChzdGF0dXMpKQogCXsKLQkgIGxlbiA9IGlvLkluZm9ybWF0aW9uOwotCSAg aWYgKGxlbiA9PSAwKQorCSAgaWYgKGlvLkluZm9ybWF0aW9uID09IDApCiAJICAgIGtlZXBfbG9v cGluZyA9IHRydWU7CisJICB0b3RhbF9sZW4gKz0gaW8uSW5mb3JtYXRpb247CisJICBpZiAoIWlz X25vbmJsb2NraW5nICgpICYmIHRvdGFsX2xlbiA8IG9yaWdfbGVuKQorCSAgICBrZWVwX2xvb3Bp bmcgPSB0cnVlOworCSAgZWxzZQorCSAgICBsZW4gPSB0b3RhbF9sZW47CiAJfQogICAgICAgZWxz ZQogCXsKQEAgLTMwOCw5ICszMjAsMTMgQEAgZmhhbmRsZXJfcGlwZTo6cmF3X3JlYWQgKHZvaWQg KnB0ciwgc2l6ZV90JiBsZW4pCiAJICAgIGNhc2UgU1RBVFVTX01PUkVfRU5UUklFUzoKIAkgICAg Y2FzZSBTVEFUVVNfQlVGRkVSX09WRVJGTE9XOgogCSAgICAgIC8qIGBpby5JbmZvcm1hdGlvbicg aXMgc3VwcG9zZWRseSB2YWxpZC4gICovCi0JICAgICAgbGVuID0gaW8uSW5mb3JtYXRpb247Ci0J ICAgICAgaWYgKGxlbiA9PSAwKQorCSAgICAgIGlmIChpby5JbmZvcm1hdGlvbiA9PSAwKQorCQlr ZWVwX2xvb3BpbmcgPSB0cnVlOworCSAgICAgIHRvdGFsX2xlbiArPSBpby5JbmZvcm1hdGlvbjsK KwkgICAgICBpZiAoIWlzX25vbmJsb2NraW5nICgpICYmIHRvdGFsX2xlbiA8IG9yaWdfbGVuKQog CQlrZWVwX2xvb3BpbmcgPSB0cnVlOworCSAgICAgIGVsc2UKKwkJbGVuID0gdG90YWxfbGVuOwog CSAgICAgIGJyZWFrOwogCSAgICBjYXNlIFNUQVRVU19QSVBFX0xJU1RFTklORzoKIAkgICAgY2Fz ZSBTVEFUVVNfUElQRV9FTVBUWToK --Multipart=_Fri__3_Sep_2021_21_22_05_+0900_i=TsOMDf7zIoXOfX--