From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from conssluserg-02.nifty.com (conssluserg-02.nifty.com [210.131.2.81]) by sourceware.org (Postfix) with ESMTPS id 31211385840D for ; Sat, 4 Sep 2021 12:37:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 31211385840D Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=nifty.ne.jp Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nifty.ne.jp Received: from Express5800-S70 (z221123.dynamic.ppp.asahi-net.or.jp [110.4.221.123]) (authenticated) by conssluserg-02.nifty.com with ESMTP id 184CbBlw002911 for ; Sat, 4 Sep 2021 21:37:12 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conssluserg-02.nifty.com 184CbBlw002911 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.ne.jp; s=dec2015msa; t=1630759032; bh=weWAe5VG658OKUULKQWmBfa/q09Iq+agioE0ts0B5MA=; h=Date:From:To:Subject:In-Reply-To:References:From; b=XOK++q8HQjLmuUczvrcNF9Ym/ifuFBsy6PiiOw6exLusLtuAYgfFdoCytWSRAXdJQ DDfd+sty+ill5OwucSR4XpVHT2oaMLcl67EFamDl3Wx18lgilZ/RgGeOZh56RNpgzM dwX95oTbbw5IoeYwhwUL+88MvZxJePkpAuoAijlD7M6Hj+Vbpo6OHZkzBOwWgZZ3PY QwtP0E0wnG0GJNAr8V02Qx7GwumMWmoHdHjZpsKLyeiUAvllvx5B+Y6b+8ARz3zf/k 5T3YNcQymP3tUMipIWLJVurs+skdEYNV+e5qTCMdrlf6HZfiEr8r0hXzXN8KVkCarG ScN3qp2xmFGtQ== X-Nifty-SrcIP: [110.4.221.123] Date: Sat, 4 Sep 2021 21:37:13 +0900 From: Takashi Yano To: cygwin-developers@cygwin.com Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? Message-Id: <20210904213713.8760e7ba3a4d68fbb78d677e@nifty.ne.jp> In-Reply-To: <20210904210258.342eb795ac53f1d5352ea512@nifty.ne.jp> References: <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> <20210903191340.c28ae366e79ca14799bacc1f@nifty.ne.jp> <20210903212205.acc2fc68cc4ffce9c1db3dd9@nifty.ne.jp> <20210904210258.342eb795ac53f1d5352ea512@nifty.ne.jp> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; i686-pc-mingw32) Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="Multipart=_Sat__4_Sep_2021_21_37_13_+0900_8Nc5KCr/4i06fsrT" X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Sep 2021 12:37:37 -0000 This is a multi-part message in MIME format. --Multipart=_Sat__4_Sep_2021_21_37_13_+0900_8Nc5KCr/4i06fsrT Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Sat, 4 Sep 2021 21:02:58 +0900 Takashi Yano wrote: > Hi Corinna, Ken, > > On Fri, 3 Sep 2021 09:27:37 -0400 > Ken Brown wrote: > > On 9/3/2021 8:22 AM, Takashi Yano wrote: > > > POSIX says: > > > The value returned may be less than nbyte if the number of bytes left > > > in the file is less than nbyte, if the read() request was interrupted > > > by a signal, or if the file is a pipe or FIFO or special file and has > > > ~~~ > > > fewer than nbyte bytes immediately available for reading. > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > > > https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html > > > > > > If it is turned over, read() should read all data immediately available, > > > I think. > > > > I understand the reasoning now, but I think your patch isn't quite right. As it > > stands, if the call to NtQueryInformationFile fails but total_length != 0, > > you're trying to read again without knowing that there's data in the pipe. > > > > Also, I think you need the following: > > > > diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc > > index ef7823ae5..46bb96961 100644 > > --- a/winsup/cygwin/fhandler_pipe.cc > > +++ b/winsup/cygwin/fhandler_pipe.cc > > @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len) > > CloseHandle (evt); > > if (status == STATUS_THREAD_SIGNALED) > > { > > - set_errno (EINTR); > > - len = (size_t) -1; > > + if (total_len == 0) > > + { > > + set_errno (EINTR); > > + len = (size_t) -1; > > + } > > + else > > + len = total_len; > > } > > else if (status == STATUS_THREAD_CANCELED) > > pthread::static_cancel_self (); > > Thanks for your advice. I fixed the issue and attached new patch. > > On Fri, 3 Sep 2021 17:37:13 +0200 > Corinna Vinschen wrote: > > Hmm, I see the point, but we might have another problem with that. > > > > We can't keep the mutex while waiting on the pending read, and there > > could be more than one pending read running at the time. if so, > > chances are extremly high, that the data written to the buffer gets > > split like this: > > > > reader 1 reader 2 > > > > calls read(65536) calls read(65536) > > > > calls NtReadFile(16384 bytes) > > calls NtReadFile(16384 bytes) > > > > writer writes 65536 bytes > > > > wakes up and gets 16384 bytes > > wakes up and gets 16384 bytes > > gets the mutex, calls > > NtReadFile(32768) which > > returns immediately with > > 32768 bytes added to the > > caller's buffer. > > > > so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes > > missing in the middle of it, *without* the reader knowing about that > > fact. If reader 1 gets the first 16384 bytes, the 16384 bytes have > > been read in a single call, at least, so the byte order is not > > unknowingly broken on the application level. > > > > Does that make sense? > > Why can't we keep the mutex while waiting on the pending read? > If we can keep the mutex, the issue above mentioned does not > happen, right? > > What about the patch attached? This keeps the mutex while read() > but I do not see any defects so far. Sorry, I forgot to attach the patch. -- Takashi Yano --Multipart=_Sat__4_Sep_2021_21_37_13_+0900_8Nc5KCr/4i06fsrT Content-Type: application/octet-stream; name="pipe-corinna4-fix.patch" Content-Disposition: attachment; filename="pipe-corinna4-fix.patch" Content-Transfer-Encoding: base64 ZGlmZiAtLWdpdCBhL3dpbnN1cC9jeWd3aW4vZmhhbmRsZXJfcGlwZS5jYyBiL3dpbnN1cC9jeWd3 aW4vZmhhbmRsZXJfcGlwZS5jYwppbmRleCA4NTczMGQwMzkuLjU3MzU0NzEyZCAxMDA2NDQKLS0t IGEvd2luc3VwL2N5Z3dpbi9maGFuZGxlcl9waXBlLmNjCisrKyBiL3dpbnN1cC9jeWd3aW4vZmhh bmRsZXJfcGlwZS5jYwpAQCAtMjIyLDYgKzIyMiw3IEBAIGZoYW5kbGVyX3BpcGU6OnJhd19yZWFk ICh2b2lkICpwdHIsIHNpemVfdCYgbGVuKQogICBEV09SRCB3YWl0cmV0ID0gV0FJVF9PQkpFQ1Rf MDsKICAgYm9vbCBrZWVwX2xvb3BpbmcgPSBmYWxzZTsKICAgc2l6ZV90IG9yaWdfbGVuID0gbGVu OworICBzaXplX3QgdG90YWxfbGVuID0gMDsKIAogICBpZiAoIWxlbikKICAgICByZXR1cm47CkBA IC0yMzQsOSArMjM1LDE3IEBAIGZoYW5kbGVyX3BpcGU6OnJhd19yZWFkICh2b2lkICpwdHIsIHNp emVfdCYgbGVuKQogICAgICAgcmV0dXJuOwogICAgIH0KIAorICBEV09SRCB0aW1lb3V0ID0gaXNf bm9uYmxvY2tpbmcgKCkgPyAwIDogSU5GSU5JVEU7CisgIGlmIChXQUlUX1RJTUVPVVQgPT0gV2Fp dEZvclNpbmdsZU9iamVjdCAocmVhZF9tdHgsIHRpbWVvdXQpKQorICAgIHsKKyAgICAgIHNldF9l cnJubyAoRUFHQUlOKTsKKyAgICAgIGxlbiA9IChzaXplX3QpIC0xOworICAgICAgcmV0dXJuOwor ICAgIH0KICAgZG8KICAgICB7Ci0gICAgICBsZW4gPSBvcmlnX2xlbjsKKyAgICAgIGNoYXIgKnB0 cjEgPSAoY2hhciAqKSBwdHIgKyB0b3RhbF9sZW47CisgICAgICBsZW4gPSBvcmlnX2xlbiAtIHRv dGFsX2xlbjsKICAgICAgIGtlZXBfbG9vcGluZyA9IGZhbHNlOwogICAgICAgaWYgKGV2dCkKIAlS ZXNldEV2ZW50IChldnQpOwpAQCAtMjQ2LDcgKzI1NSw2IEBAIGZoYW5kbGVyX3BpcGU6OnJhd19y ZWFkICh2b2lkICpwdHIsIHNpemVfdCYgbGVuKQogCSAgVUxPTkcgcmVhZGVyX2NvdW50OwogCSAg VUxPTkcgbWF4X2xlbiA9IDY0OwogCi0JICBXYWl0Rm9yU2luZ2xlT2JqZWN0IChyZWFkX210eCwg SU5GSU5JVEUpOwogCiAJICAvKiBNYWtlIHN1cmUgbmV2ZXIgdG8gcmVxdWVzdCBtb3JlIGJ5dGVz IHRoYW4gaGFsZiB0aGUgcGlwZQogCSAgICAgYnVmZmVyIHNpemUuICBFdmVyeSBwZW5kaW5nIHJl YWQgbG93ZXJzIFdyaXRlUXVvdGFBdmFpbGFibGUKQEAgLTI2MCwxNyArMjY4LDI1IEBAIGZoYW5k bGVyX3BpcGU6OnJhd19yZWFkICh2b2lkICpwdHIsIHNpemVfdCYgbGVuKQogCQkJCQkgICBGaWxl UGlwZUxvY2FsSW5mb3JtYXRpb24pOwogCSAgaWYgKE5UX1NVQ0NFU1MgKHN0YXR1cykgJiYgZnBs aS5SZWFkRGF0YUF2YWlsYWJsZSA9PSAwKQogCSAgICB7CisJICAgICAgaWYgKHRvdGFsX2xlbiAh PSAwKQorCQl7CisJCSAgbGVuID0gdG90YWxfbGVuOworCQkgIGJyZWFrOworCQl9CiAJICAgICAg cmVhZGVyX2NvdW50ID0gZ2V0X29ial9oYW5kbGVfY291bnQgKGdldF9oYW5kbGUgKCkpOwogCSAg ICAgIGlmIChyZWFkZXJfY291bnQgPCAxMCkKIAkJbWF4X2xlbiA9IGZwbGkuSW5ib3VuZFF1b3Rh IC8gKDIgKiByZWFkZXJfY291bnQpOwogCSAgICAgIGlmIChsZW4gPiBtYXhfbGVuKQogCQlsZW4g PSBtYXhfbGVuOwogCSAgICB9CisJICBpZiAoIU5UX1NVQ0NFU1MgKHN0YXR1cykgJiYgdG90YWxf bGVuICE9IDApCisJICAgIHsKKwkgICAgICBsZW4gPSB0b3RhbF9sZW47CisJICAgICAgYnJlYWs7 CisJICAgIH0KIAl9Ci0gICAgICBzdGF0dXMgPSBOdFJlYWRGaWxlIChnZXRfaGFuZGxlICgpLCBl dnQsIE5VTEwsIE5VTEwsICZpbywgcHRyLAorICAgICAgc3RhdHVzID0gTnRSZWFkRmlsZSAoZ2V0 X2hhbmRsZSAoKSwgZXZ0LCBOVUxMLCBOVUxMLCAmaW8sIHB0cjEsCiAJCQkgICBsZW4sIE5VTEws IE5VTEwpOwotICAgICAgaWYgKCFpc19ub25ibG9ja2luZyAoKSkKLQlSZWxlYXNlTXV0ZXggKHJl YWRfbXR4KTsKICAgICAgIGlmIChldnQgJiYgc3RhdHVzID09IFNUQVRVU19QRU5ESU5HKQogCXsK IAkgIHdhaXRyZXQgPSBjeWd3YWl0IChldnQpOwpAQCAtMjkxLDkgKzMwNywxMSBAQCBmaGFuZGxl cl9waXBlOjpyYXdfcmVhZCAodm9pZCAqcHRyLCBzaXplX3QmIGxlbikKIAl9CiAgICAgICBlbHNl IGlmIChOVF9TVUNDRVNTIChzdGF0dXMpKQogCXsKLQkgIGxlbiA9IGlvLkluZm9ybWF0aW9uOwot CSAgaWYgKGxlbiA9PSAwKQorCSAgdG90YWxfbGVuICs9IGlvLkluZm9ybWF0aW9uOworCSAgaWYg KHRvdGFsX2xlbiA8IG9yaWdfbGVuKQogCSAgICBrZWVwX2xvb3BpbmcgPSB0cnVlOworCSAgZWxz ZQorCSAgICBsZW4gPSB0b3RhbF9sZW47CiAJfQogICAgICAgZWxzZQogCXsKQEAgLTMwMywxNyAr MzIxLDI0IEBAIGZoYW5kbGVyX3BpcGU6OnJhd19yZWFkICh2b2lkICpwdHIsIHNpemVfdCYgbGVu KQogCSAgICBjYXNlIFNUQVRVU19FTkRfT0ZfRklMRToKIAkgICAgY2FzZSBTVEFUVVNfUElQRV9C Uk9LRU46CiAJICAgICAgLyogVGhpcyBpcyByZWFsbHkgRU9GLiAgKi8KLQkgICAgICBsZW4gPSAw OworCSAgICAgIGxlbiA9IHRvdGFsX2xlbjsKIAkgICAgICBicmVhazsKIAkgICAgY2FzZSBTVEFU VVNfTU9SRV9FTlRSSUVTOgogCSAgICBjYXNlIFNUQVRVU19CVUZGRVJfT1ZFUkZMT1c6CiAJICAg ICAgLyogYGlvLkluZm9ybWF0aW9uJyBpcyBzdXBwb3NlZGx5IHZhbGlkLiAgKi8KLQkgICAgICBs ZW4gPSBpby5JbmZvcm1hdGlvbjsKLQkgICAgICBpZiAobGVuID09IDApCisJICAgICAgdG90YWxf bGVuICs9IGlvLkluZm9ybWF0aW9uOworCSAgICAgIGlmICh0b3RhbF9sZW4gPCBvcmlnX2xlbikK IAkJa2VlcF9sb29waW5nID0gdHJ1ZTsKKwkgICAgICBlbHNlCisJCWxlbiA9IHRvdGFsX2xlbjsK IAkgICAgICBicmVhazsKIAkgICAgY2FzZSBTVEFUVVNfUElQRV9MSVNURU5JTkc6CiAJICAgIGNh c2UgU1RBVFVTX1BJUEVfRU1QVFk6CisJICAgICAgaWYgKHRvdGFsX2xlbiAhPSAwKQorCQl7CisJ CSAgbGVuID0gdG90YWxfbGVuOworCQkgIGJyZWFrOworCQl9CiAJICAgICAgaWYgKGlzX25vbmJs b2NraW5nICgpKQogCQl7CiAJCSAgc2V0X2Vycm5vIChFQUdBSU4pOwpAQCAtMzI4LDEyICszNTMs MTggQEAgZmhhbmRsZXJfcGlwZTo6cmF3X3JlYWQgKHZvaWQgKnB0ciwgc2l6ZV90JiBsZW4pCiAJ ICAgIH0KIAl9CiAgICAgfSB3aGlsZSAoa2VlcF9sb29waW5nKTsKKyAgUmVsZWFzZU11dGV4IChy ZWFkX210eCk7CiAgIGlmIChldnQpCiAgICAgQ2xvc2VIYW5kbGUgKGV2dCk7CiAgIGlmIChzdGF0 dXMgPT0gU1RBVFVTX1RIUkVBRF9TSUdOQUxFRCkKICAgICB7Ci0gICAgICBzZXRfZXJybm8gKEVJ TlRSKTsKLSAgICAgIGxlbiA9IChzaXplX3QpIC0xOworICAgICAgaWYgKHRvdGFsX2xlbiA9PSAw KQorCXsKKwkgIHNldF9lcnJubyAoRUlOVFIpOworCSAgbGVuID0gKHNpemVfdCkgLTE7CisJfQor ICAgICAgZWxzZQorCWxlbiA9IHRvdGFsX2xlbjsKICAgICB9CiAgIGVsc2UgaWYgKHN0YXR1cyA9 PSBTVEFUVVNfVEhSRUFEX0NBTkNFTEVEKQogICAgIHB0aHJlYWQ6OnN0YXRpY19jYW5jZWxfc2Vs ZiAoKTsK --Multipart=_Sat__4_Sep_2021_21_37_13_+0900_8Nc5KCr/4i06fsrT--