From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2135.outbound.protection.outlook.com [40.107.220.135]) by sourceware.org (Postfix) with ESMTPS id B2487384A881 for ; Fri, 3 Sep 2021 13:27:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B2487384A881 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=G1C6Hg0FpTkhf+I7fcz11/Rzh9lO5awrTRFNd7q/b8xQj0g0elGxeFg+bjQktDetmVWNpETF0QPATt2cGBTwmOlL1jLU5AhIxNOLePQXOSpcvARv+lrHfOj+ENdatmfYqeSrFRw9BTMumzSSM2gYx0mi8t5hl7stcfLDRcGAdjF327O3XTrXMR+WImqzT7wzSsxj+yMBEVJLNvzQBvRxZ2xiad7KzZsJ0qpZidG6LHnhqjEKJ0ums+siPm/WBOSdonF4Va5MaZPLrRrlIQqiIpWdsb9KV9oS7g4uysaVNqjEZamQ4YQdjL1H5faPN6r3bjqcKMC87HWpj1azMyJaeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=CY7Bi8TilY9XIOY/nrySUQA6o2xdTHKclKXsRDlX9OM=; b=OVJ9umZY4mEEBk2Q2YhREQXnbcfss6oegnF+WuyWw7BMmW43i/gRxylIgAK1LqLbXrA98itJInTvv9ncONtVIdjPkcCws4o6dAYEuhtAugZ9fc5Uz048GjrBPhKmjLBMkQWyMaYa6joLM2I3RMuxGP5ijJJLopKYrpzbWbn0pJpNKl5QTW8gPkmQmx4u7M/tTUsiR397oLhLfKmO2bfzueC3vJSH3HE9rQyvgCgMbvXejK2Aqn3mTJW6uNV/FNXlAhv3Dj9D+7DpOZuboxIaxM/xplwyj+VSrN/V39F+xFz0l4EnFbtnfnQ9JD+TRtcT60Yw8WoJWGg8/1KHOj4Lhg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CY7Bi8TilY9XIOY/nrySUQA6o2xdTHKclKXsRDlX9OM=; b=ErCmLZw4emNcILi7a40RiPNjSKjNUNMjevFiAlMZK/TiogumxciUJFWWGinrJxGXqCTylQ1Q78PUzxQ+X+uzB7/xmkFpaasjdX6gsrP4VfYBKvLSRFY7wb+T56iXcBp8Tw15OrATBGjWaGTywa2utwIf8rz3tFYJkhNOy0r3qU4= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN7PR04MB4385.namprd04.prod.outlook.com (2603:10b6:406:f0::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.17; Fri, 3 Sep 2021 13:27:39 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e%7]) with mapi id 15.20.4478.020; Fri, 3 Sep 2021 13:27:38 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> <20210903191340.c28ae366e79ca14799bacc1f@nifty.ne.jp> <20210903212205.acc2fc68cc4ffce9c1db3dd9@nifty.ne.jp> From: Ken Brown Message-ID: <020f405a-a35f-0bcb-0d53-28153f7ae5a7@cornell.edu> Date: Fri, 3 Sep 2021 09:27:37 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <20210903212205.acc2fc68cc4ffce9c1db3dd9@nifty.ne.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0PR04CA0046.namprd04.prod.outlook.com (2603:10b6:610:77::21) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [IPv6:2603:7081:7e3f:3419:a4be:f19a:7570:4bb4] (2603:7081:7e3f:3419:a4be:f19a:7570:4bb4) by CH0PR04CA0046.namprd04.prod.outlook.com (2603:10b6:610:77::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.19 via Frontend Transport; Fri, 3 Sep 2021 13:27:38 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f19d49b2-7c42-4b2a-8642-08d96ede996a X-MS-TrafficTypeDiagnostic: BN7PR04MB4385: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gThKYqdKSOMDgMgfGN0GA83YWeCWfTRF64XNuEH3FZH20T89giS8NpGV3LTyyFwqICTMPVxU8zqMYNphG4kUnUCTtf1hrmm/NE6QvqGDlEgVG1I2CVhWMPM9YIo3Gcyn1HE+x6RT6pUoRenIITYiMuHPqs+iACtEw2Ym0qfBa7kE+aiy1x62Qyuwyss+UZ6Qmml4qpMNTp4aR/m+dkpXMagvqDIvwFFb9DeK3gtyrdCd1RSU0dzmvcL85JNFjF7tiuvCjAUngccVv2b+gxPCe8jOxVWEsYxM1YOYiv2Pvn4hBJ90S15ukWVVyqqUoDgEoHtwhH0vf82tYLHTa3GOvNDSeGyCVb23WFgqNcJ2/EWC22qCKl3MhBQqfqhNuKbdxV5CA1nMqFGOWRv5j7AqbD6apcNeVwDBNbYR5ubMRlA9W6vFJ4vTrB4PIHEfZ5Of6+D2qysWJkYY2fNVRu/Pdbsq4Qghc+498Yf7HOxwW9e0vzCjS/h5882Is8S6yzccl7WaAfatBJPPi/Q/NaV2iuFCNVcdwDGymF7LF0RG2s51m5yLP0QzcGcOQCJvBoraJXlMhBB/2/Sfp/Ni0vEn1pzBlO4CvNJhKLl/px7lobKAsa4NLCfGKF2s9EyXD5n3rs1YMHNudfYgdJDS62CXxnxZWDkbIUegmZrRztE1fVSNyw7yhEdnsVTJ0viINEK+Ig14uYOI++g3vsJuC6P9jmVvbnxuJLtsdg7LkgzcNVj9WK76DYeE3sEpa8X7EV0gFzuv7KvnudLzNd1MBEeBIKKfApC/4ZkU599o2OtXCL4Tyx7zOF0jSb4rYFVcyA4n X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(376002)(136003)(346002)(396003)(39860400002)(31696002)(8676002)(8936002)(31686004)(2906002)(316002)(6916009)(75432002)(5660300002)(66946007)(66556008)(186003)(66476007)(53546011)(6486002)(2616005)(36756003)(478600001)(966005)(38100700002)(86362001)(83380400001)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?5QM0nooTL5TPgL7MOP9+JGg2pK+qgyYPGl01KiPivgljEtGK07IN8kxg?= =?Windows-1252?Q?RAojNbpct1FyZpu+Bqn9KyFUZpiHfAXJ8+wLA9vtRnA0dxGWkLTrCf7R?= =?Windows-1252?Q?OnxeAjINFavC7veQWSSvr/lC1zikKIHXvAA+BSPTKFBjA3YLsfKNhuaV?= =?Windows-1252?Q?4loLeK2Nn4mzf9rEENr9bLyA46QraPHM26bSVZmrjFbNM2sxhTRPGdca?= =?Windows-1252?Q?h0oUK7rsDt4134WnDdufxcX+PZIUqNEIQaF6Mz8AWQZ7X4Y0kALzFZFN?= =?Windows-1252?Q?OrAnYF0doCHpHYF0u9E7vzaCEA5cP4cW+4qYfDDyJ944Zxg1HsRKmM9i?= =?Windows-1252?Q?+XYUpHcXi3I0RaS51PjMflJvXp3dgzafy6Tt+Loynr5TVYDBl7OE5hp4?= =?Windows-1252?Q?qAnV187kZy4iA/eWSQtft/RZrFrN+1BqopyjikDNYOJKw3+TbaByn0JA?= =?Windows-1252?Q?y64vVIccU+dFrAaK3+YpJzapwmx/svibpq2vhaPxp+emzvzei+pXJfvH?= =?Windows-1252?Q?QPoYtz334AcB9CPemqPAax0ewyPudwusPQ8wn7sagEDg4Yqa1kG9EAlf?= =?Windows-1252?Q?JNZq0s/go7NSf18dFFMycEi/LQ27wkdNIa40Qu6m/4wNIgqn5P+mkBik?= =?Windows-1252?Q?pSaL97O41+9+19WVCybxUIPKp/PDbrQkK0uvOE6QrWiJN4rba+Bxm/92?= =?Windows-1252?Q?dw2iwtWdQR72VdGNY4A/FrfB7cqggp52LSlT14Xb1a46+Xj70mlws8GF?= =?Windows-1252?Q?b0UX0nlrLj30t8+lLgQ2QJnrDhcn3NfcTs70xx+brc3dbX6iGh2owUGR?= =?Windows-1252?Q?UcNPEwggKSsntrMaSUIckjTH9ty5Iuhs8KATPkWZLOV3bI38skELPCz4?= =?Windows-1252?Q?rrfjh3NTACP8r0gTUE0AMGBGzn7ZtirA3sftY0tJndduMuWFI1wezyZk?= =?Windows-1252?Q?kT/3X2JrKB31O0ustElLX4xUoMUvqM3RcsRsxCEu48h5EHfu35zwXQuq?= =?Windows-1252?Q?ml3PpglL4RL5duxC+J49BxzUIycyRpcY3kMkopG7fuINBC5k6I/nQctW?= =?Windows-1252?Q?yHJEU3j32k2vD0F7eHv+3w/185IF0qZnE6RvNmVHBKwEdkxkA54jPRP8?= =?Windows-1252?Q?d/Rq4MERV/22R0phTe4jj8PU1g0ixL9U17XAaWVzPrUMxcZZUclqeiUY?= =?Windows-1252?Q?QEUBSy1VjyGL1p5QOnqL8ZLagvDax5urwy5oJ8M+4XQePDVtRbQgs+KF?= =?Windows-1252?Q?vbdLgDyIQ+D8l+2RSfr0mNy/mr44ydsI1B6XDm2rwMinrz9X5iBR3V7a?= =?Windows-1252?Q?0XVD96fbxMW6mdQdhdv5SL6zxR4X8nWpnU8Nfoze0fbQud6FVMxdm4Qn?= =?Windows-1252?Q?kzC7bTv9YP/evv+MbJVQrKBZeP9ofRUibcUgbVvpB8iXNoDzqPBKNrJP?= =?Windows-1252?Q?MKzfM7qFoSMorGkD2oE08Fh81Lihhfvn5P15vWyKzn16ZH14Q5S2NaN2?= =?Windows-1252?Q?lx++fPzW?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: f19d49b2-7c42-4b2a-8642-08d96ede996a X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Sep 2021 13:27:38.7884 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: oeVenL3GJt27x/8UvbdmdwS7jVe5plphBbWZQ/r1s1Vwo51CVGrjPBBPbTbGvi4jFU5OdearP2L943GHnDKf+g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR04MB4385 X-Spam-Status: No, score=-10.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, JMQ_SPF_NEUTRAL, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2021 13:27:42 -0000 On 9/3/2021 8:22 AM, Takashi Yano wrote: > On Fri, 3 Sep 2021 13:41:45 +0200 > Corinna Vinschen wrote: >> On Sep 3 13:31, Corinna Vinschen wrote: >>> On Sep 3 19:13, Takashi Yano wrote: >>>> On Fri, 3 Sep 2021 19:00:46 +0900 >>>> Takashi Yano wrote: >>>>> On Thu, 2 Sep 2021 21:35:21 +0200 >>>>> Corinna Vinschen wrote: >>>>>> On Sep 2 21:00, Corinna Vinschen wrote: >>>>>> It's getting too late again. I drop off for tonight, but I attached >>>>>> my POC code I have so far. It also adds the snippets from my previous >>>>>> patch which fixes stuff Takashi found during testing. It also fixes >>>>>> something which looks like a bug in raw_write: >>>>>> >>>>>> - ptr = ((char *) ptr) + chunk; >>>>>> + ptr = ((char *) ptr) + nbytes_now; >>>>>> >>>>>> Incrementing ptr by chunk bytes while only nbytes_now have been written >>>>>> looks incorrect. >>>>>> >>>>>> As for the reader, it makes the # of bytes to read dependent on the >>>>>> number of reader handles. I don't know if that's such a bright idea, >>>>>> but this can be changed easily. >>>>>> >>>>>> Anyway, this runs all my testcases successfully but they are anything >>>>>> but thorough. >>>>>> >>>>>> Patch relativ to topic/pipe attached. Would you both mind to take a >>>>>> scrutinizing look? >>>>> >>>>> Thanks. >>>>> >>>>> Your code seems that read() returns only the partial data even >>>>> if the pipe stil has more data. Is this by design? >>>>> >>>>> This happes in both blocking and non-blocking case. >>>> >>>> The patch attached seems to fix the issue. >>> >>> I'm sorry, but I don't see what your patch is supposed to do in the >>> first place. What I see is that it now calls NtQueryInformationFile >>> even in the non-blocking case, which is not supposed to happen. >>> >>> I'm a bit puzzled what the actual bug is. >>> >>> The code changing len is only called if there's no data in the pipe. >>> In that case we only request a partial buffer so as not to block >>> the writer on select. >>> >>> If there *is* data in the pipe, it will just go straight to the >>> NtReadFile code without changing len. >>> >>> Where's the mistake? >> >> Oh, wait. Do you mean, if we only request less than len bytes, but >> after NtReadFile there's still data in the buffer, we should try to >> deplete the buffer up to len bytes in a subsequent NtReadFile? > > Yes. I am sorry, my intent was not clear because I did more than > necessary in the previous patch. Please see attached patch revised. > >> I thought this is unnecessary, actually, because of POSIX: >> >> The standard developers considered adding atomicity requirements to a >> pipe or FIFO, but recognized that due to the nature of pipes and FIFOs >> there could be no guarantee of atomicity of reads of {PIPE_BUF} or any >> other size that would be an aid to applications portability. > > POSIX says: > The value returned may be less than nbyte if the number of bytes left > in the file is less than nbyte, if the read() request was interrupted > by a signal, or if the file is a pipe or FIFO or special file and has > ~~~ > fewer than nbyte bytes immediately available for reading. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html > > If it is turned over, read() should read all data immediately available, > I think. I understand the reasoning now, but I think your patch isn't quite right. As it stands, if the call to NtQueryInformationFile fails but total_length != 0, you're trying to read again without knowing that there's data in the pipe. Also, I think you need the following: diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc index ef7823ae5..46bb96961 100644 --- a/winsup/cygwin/fhandler_pipe.cc +++ b/winsup/cygwin/fhandler_pipe.cc @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len) CloseHandle (evt); if (status == STATUS_THREAD_SIGNALED) { - set_errno (EINTR); - len = (size_t) -1; + if (total_len == 0) + { + set_errno (EINTR); + len = (size_t) -1; + } + else + len = total_len; } else if (status == STATUS_THREAD_CANCELED) pthread::static_cancel_self (); Ken