From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.135]) by sourceware.org (Postfix) with ESMTPS id AA779384843C for ; Fri, 3 Sep 2021 15:37:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org AA779384843C Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=cygwin.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=cygwin.com Received: from calimero.vinschen.de ([24.134.7.25]) by mrelayeu.kundenserver.de (mreue009 [212.227.15.167]) with ESMTPSA (Nemesis) id 1MV6G6-1mVVC019Py-00S3t9 for ; Fri, 03 Sep 2021 17:37:14 +0200 Received: by calimero.vinschen.de (Postfix, from userid 500) id B1B1CA80F97; Fri, 3 Sep 2021 17:37:13 +0200 (CEST) Date: Fri, 3 Sep 2021 17:37:13 +0200 From: Corinna Vinschen To: cygwin-developers@cygwin.com Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? Message-ID: Reply-To: cygwin-developers@cygwin.com Mail-Followup-To: cygwin-developers@cygwin.com References: <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> <20210903191340.c28ae366e79ca14799bacc1f@nifty.ne.jp> <20210903212205.acc2fc68cc4ffce9c1db3dd9@nifty.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20210903212205.acc2fc68cc4ffce9c1db3dd9@nifty.ne.jp> X-Provags-ID: V03:K1:Ekr/Q6nklDD6UsSt+Zs5uLRn/U3rWFE++kuA4bzHSQ9m27G+cZP c/ix3xrzEGM7C4Hi8pj70XBAGdm/KTf5+gY1UkJuoYIZ3+UqTyHmGQXt54irYoUpKoOl2Cf nJVoVhKVqNN8hryowBLcWH2jNHKtiIPxgv/flKQyHskm/OGlblT0wJvBSZmB9aP3YZ/dbjH Dn0qNKZCE1KVHWCbdSL4w== X-UI-Out-Filterresults: notjunk:1;V03:K0:1S5vDmhkuYM=:wgLlCUZqtEQCVpnynMiVQm 2wppvytjrb4LlpVShx7csng5o2Qebr3878fdLAcZ+vSDaQXu9S6MMmW8FKaHidOdj5W0ZMfG1 Z50cAw4Sygtai1VPFdj06nxcrPZozB+kOwQw30kDsXXDPU7I8Lz+MUxbX80m7v18Y/nOh3zW3 qmBgBjpVkUyxXfSLNbVys6oSrcWDEdN9na9vmDsDtGHRo4SbZHalDAStfCSkCEGqenTRZJIAn 0rfY56waKs58BTZvaLfzjcnXkvTN3CgByd1nuycPAy+dFH8H/gO1IWZZyRyJ0z180jEhPbDUD UPlX0iYl1zLxk8bwiOBM7kpaN3EZmYoxoc2xOAkj7YFBTptPMOMJd43XsJbjOiMx+qlaTlBGz oNH/F/pml5ob8rtpbxLCuSDo5IeKt7w0/Rl1y0vNIZugsZHL++NsSARzMbdqp7bMss5p9EuR7 gELOvOLJP0Br2oH3DPBQLTBaSuNxhLQZgETXq7rOwZoDB4qF0qyGqdRJ0pSVpWNLchAoVi6vw RQck+/pWnt5ZoeGx3wONHXNjwx5/yDrJgcBOXJmXr7dRu79cqgh3RYGk1d9+mVZWxLQ/ESZHd 3JITDxviyIQBE+/3HKqJFVQdNA8LorE6PzK9e7tcXD8CgB78bHt8WbZmcxFR5oFXYiJUz3Xyw 89yuq2Yb1gR/T7QHDNwZWLybX6eZEYR/P6XwP9rx4HED2a+mJAgKJAwFsZUbLu563h4yI5A3Z qDYuBLor8qO5B8rDLq085ZiUQ67/Vc9lTdizesKJhC2g8bKamQigXpOuLRE= X-Spam-Status: No, score=-99.7 required=5.0 tests=BAYES_00, GOOD_FROM_CORINNA_CYGWIN, JMQ_SPF_NEUTRAL, KAM_DMARC_NONE, KAM_DMARC_STATUS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NEUTRAL, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2021 15:37:18 -0000 On Sep 3 21:22, Takashi Yano wrote: > On Fri, 3 Sep 2021 13:41:45 +0200 > Corinna Vinschen wrote: > > Oh, wait. Do you mean, if we only request less than len bytes, but > > after NtReadFile there's still data in the buffer, we should try to > > deplete the buffer up to len bytes in a subsequent NtReadFile? > > Yes. I am sorry, my intent was not clear because I did more than > necessary in the previous patch. Please see attached patch revised. > > > I thought this is unnecessary, actually, because of POSIX: > > > > The standard developers considered adding atomicity requirements to a > > pipe or FIFO, but recognized that due to the nature of pipes and FIFOs > > there could be no guarantee of atomicity of reads of {PIPE_BUF} or any > > other size that would be an aid to applications portability. > > POSIX says: > The value returned may be less than nbyte if the number of bytes left > in the file is less than nbyte, if the read() request was interrupted > by a signal, or if the file is a pipe or FIFO or special file and has > ~~~ > fewer than nbyte bytes immediately available for reading. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html > > If it is turned over, read() should read all data immediately available, > I think. Hmm, I see the point, but we might have another problem with that. We can't keep the mutex while waiting on the pending read, and there could be more than one pending read running at the time. if so, chances are extremly high, that the data written to the buffer gets split like this: reader 1 reader 2 calls read(65536) calls read(65536) calls NtReadFile(16384 bytes) calls NtReadFile(16384 bytes) writer writes 65536 bytes wakes up and gets 16384 bytes wakes up and gets 16384 bytes gets the mutex, calls NtReadFile(32768) which returns immediately with 32768 bytes added to the caller's buffer. so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes missing in the middle of it, *without* the reader knowing about that fact. If reader 1 gets the first 16384 bytes, the 16384 bytes have been read in a single call, at least, so the byte order is not unknowingly broken on the application level. Does that make sense? Corinna