From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2102.outbound.protection.outlook.com [40.107.237.102]) by sourceware.org (Postfix) with ESMTPS id BD5DD3858403 for ; Sun, 29 Aug 2021 15:57:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org BD5DD3858403 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=N5f75N4A/HoXfx3atuyBHaeuBaKpZ3b8HGxqpNged1wGsaQPpLeZhIf1j4qFug5Y/5Mxr+J+DAN86H3aI1f1XlGytQBwjqSkrKXoiqtn/WeITDg/m1kbBmMNSasX1oQfXvWAgN73xZ9CmVMSHih96O1Md0/MksfGhqIt0oIlfwTlzX0QWsqiAbcO7TYAmJlfxA62PD7BPBtrvy2QQ3kfM0uvHlJPMZLSSUCpaeknOkhLmk5vTsIb/YIIRxW7Gdyu1oDXpfsO70c1WSbSc7VdOTji5P9DbvTRgmFGbEjkDVFUayc0oGfFFjS6+Xs5/U1vSQtPgWbPT4IMtWQ+5BXs+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6MNxanTzMOAqA61fxpMLI+4Azhvjbhd+QbwJiojGkEY=; b=MdqhQBSZviMt1CSjIo5C+Iy0MErE8BxuYYmpL8QEF4HthYqtmCmjO+CPdS+zTp7QmP2aJ9Woxo3UIbbBZmeWqnZFOg5nT8FWJ0zZ5mAsNa0K+luCR2jRztu75r4Yk5NpfnEacDKwok5y1vxu/LxJn0sQvy52r/Gsny5qqXvbVjGTs1si8DfKzRx/0oZaXzaHDWl96tsssPGGvwUeUABYh9C4+wvTEtAYfK0IV/1w8UeaXZ+UtzhfKT+MPM8Yl5VY1vkwviRvKZ5X3GVLKlYGRh0N87KSp64KUgOa32PEzY+9h9mbYIjQyUAf3JMJvJIVu33DVcZZB7YadodjcnkqTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN7PR04MB3956.namprd04.prod.outlook.com (2603:10b6:406:c2::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.18; Sun, 29 Aug 2021 15:57:06 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::38bd:b608:234f:9ec6]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::38bd:b608:234f:9ec6%7]) with mapi id 15.20.4457.024; Sun, 29 Aug 2021 15:57:05 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin@cygwin.com References: <41A583E1-C8E7-42AB-9F24-EEC33A41EC60@house.org> <20210825201845.07b6400b79dc5558a7761efe@nifty.ne.jp> <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp> <3b560051-ab27-f392-ca4b-d1fd9b5733b0@cornell.edu> <20210827202440.47706fc2fc07c5e9a1bc0047@nifty.ne.jp> <4f2cb5f3-ce9c-c617-f65f-841a5eca096e@cornell.edu> <20210828022111.91ef5b4ff24f6da9fadb489e@nifty.ne.jp> <20210828184102.f2206a8a9e5fe5cf24bf5e45@nifty.ne.jp> <20210829180729.48b4e877f773cb3980c5766d@nifty.ne.jp> From: Ken Brown Message-ID: <789f056a-f164-d71d-1dc9-230f5a41846d@cornell.edu> Date: Sun, 29 Aug 2021 11:57:04 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <20210829180729.48b4e877f773cb3980c5766d@nifty.ne.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0PR03CA0018.namprd03.prod.outlook.com (2603:10b6:610:b0::23) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.1.211] (74.69.128.111) by CH0PR03CA0018.namprd03.prod.outlook.com (2603:10b6:610:b0::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23 via Frontend Transport; Sun, 29 Aug 2021 15:57:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 14a5333f-8bc4-4d94-605f-08d96b05a5ff X-MS-TrafficTypeDiagnostic: BN7PR04MB3956: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Vw004yVz70uSViL9VY4yEsEjaQ5JYqJABYlTqFgHW9Aig9jNtbCk7qc/kcIzDk6bYw91LytM8O3+tEvBpIwOQ8Ns1NnoJqZy5OQwaGM3ymf8zXUfm0ZJq9S19AhqFOkcKWTswGFsKZgmLMoXWqT7G5x49ZdUi0Eyn3f5zTwWqKP61D0+kYdtw39bx26O70S5fer0EqCKDB/S6gs5ibuw3gIcQeY3IXxk+PiSrI4qH+vKMijhPVIeLQF4wWg1n6ndf9AMAvu4X+7f5fEZ9fdAp2szC+vI23kOgsUuNX+xt+linwiV+L+4MQ3L9MjcCXgYf1nJ2msw7z8w1ROAhL2Q1RS1yk4QAk8+o391bQnSC2sRLsFhYjhdpvBwRAmb56EZd3BEHJQXxKvk8V1vwWTTJ45en1FopMPXErPWiTiHZy92ldepfuPBZTePuK+qhTS4wl+qfoejnWEBvtzMFIFWT9sI7eJuL76iHD8EqW0DTmGpb1SxFhm3bU8u9pxEx0wJ7vUmufPSge9vBxrdJPxbtQNlQykGx/zaYMV6afU7Gr4kzp6/kw/gWqrbnxUUTpKZhxAZKg9xQ3uTQ0QH/YlgulEta/C7roemFF4k7TN75Zp30WlvRO2qvXy5p/eB4XyGCOvmBl6xEmClNef/jaXmwbBopQdFnqcSt+7ky5LQNlTl7NNcXe1L/DFI+B+faipL3eyuNqjBn1tFAkzNs6tz1JUoTk9l5Dk9SMjrUaGb28g= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(136003)(366004)(396003)(39860400002)(376002)(38100700002)(31696002)(5660300002)(478600001)(86362001)(36756003)(6916009)(53546011)(16576012)(786003)(316002)(83380400001)(75432002)(66476007)(2616005)(956004)(66556008)(66946007)(2906002)(186003)(8676002)(6486002)(31686004)(8936002)(26005)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?rfPYK7/UsX163Ca5agMJyTWjtqQl4qkoGx0m3nufx11uXaTnjk7oBsaU?= =?Windows-1252?Q?uXHvS1o/EbGfcDxR6bidlckd3ZSVhRGwwU+hUtEKJLi1HH8Ed10WyHzh?= =?Windows-1252?Q?j7/IK8Uwpy0SxXogqVNrW/+y7MoWSeOnbCPvE0wqgkJZ1IaEs3cH74xB?= =?Windows-1252?Q?F+88h4PTeriTGCgrJcD4nmVnxnZ/BhZhfU+hDxiCxorSbeknxC9BPoIO?= =?Windows-1252?Q?3qJaQ86i2y4u21qy3DS3rkOFpjcjqCR2Dzqj8s4n/q9gP5Ua763vJvTW?= =?Windows-1252?Q?PPzzAxqudOLVliqCOr3L4JYzufWA7uY8yTe/MhG3McHZT1Z0HqO0roo5?= =?Windows-1252?Q?bivI/mJmYSqafvWmwbegHDblDv6SZrLCqRIYbkY3rH3qNBkXBogvEg6D?= =?Windows-1252?Q?s+9Is0AtEfXa94pGvwrzVOwyMbrmkvNCiQIAALPZ3a6XzN4B4E7UkSfS?= =?Windows-1252?Q?1VFKVU74bUpTz5n+/0nuUI8ptDd4GduBey/29jQUdIRr92qQonz2GgKb?= =?Windows-1252?Q?6IyrycSX8UOD6kR0zYj8luOMS1Dq8lygCMYAlp4kA0xnQqWGsE2iAMOI?= =?Windows-1252?Q?d6Fv5/llYlLqyWG19XGmE6Tkk9GPaeJq2DjUpsVjc5OSHJWoCURiluM6?= =?Windows-1252?Q?jEILtS3TZSYttKjXjwYYbMhBuuOIwIoL92KM92+l8qfzK12P1HXGdEM4?= =?Windows-1252?Q?UtsA9qoTbdeiwuGS6Pi1ym6By6PW1VBtek+NAcfy6pwv4XhCgwraiD2o?= =?Windows-1252?Q?WlW3V4k2tmhESK8DWE78JeMqhVh2Q/KC9+rLlZZ1Tc7aDiLw29RP+m/k?= =?Windows-1252?Q?UpOtpLQ4fDf7bb1raeGTJUQN1QKTWWJ6jLOLoyVFETAlGsbM4L+P+0wv?= =?Windows-1252?Q?PAWDiB674x/2ILIW0lKadegfdyv9p+2YBj/MoHG6045PKaZJN82p2tAh?= =?Windows-1252?Q?BiEDmOIdQYFWaPQgNTdvTiIYfJYPN0NJbs0v6xA20TidyLGqRJCo5NBX?= =?Windows-1252?Q?ri/k45JFRPoXoaX0tdsz+7DjawbWQFbG0a0OzkFOrTqKX7K/CPABdjz8?= =?Windows-1252?Q?2AkJvcrVA19g5gARxfBGb9oCHAvbEzhVr4Lmnmw2R1LN1P7L3Ho1mAA8?= =?Windows-1252?Q?holYReePtg4hwK5YMKHrM8bspWLJQjoIeJ5xZku6YQv/Lf/ltgVg/nKC?= =?Windows-1252?Q?BRMXXHoXKGUBOl0lprEeUSbQwe+bfUUEsIzXB2pVM7t6gUkzzOxuturs?= =?Windows-1252?Q?QfKvyJ6kATGKqZN+iUmArIHcOZmJws5dx0WkxLlJPR/Q+WOwj4XwbVCP?= =?Windows-1252?Q?3/kN8MIYeHgdGiH3DXePNAu4IdB+1mN+MV73aN2MAoZ8xa9Z23+ZPjjU?= =?Windows-1252?Q?RM8Meefp883gc/vkzWG/rRSMwbT20Y6rI52iNoRf+LvBFmxxVYBmurzN?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 14a5333f-8bc4-4d94-605f-08d96b05a5ff X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Aug 2021 15:57:05.5190 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bE2Ze1aCpMsC1KCbii4qImzAPdONyf5zj9iq++p1Q892vFHhjrKaPrtPuZNWajEKY2/R1+p52s04aNJ6JF/1sg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR04MB3956 X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Aug 2021 15:57:19 -0000 On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote: > On Sat, 28 Aug 2021 18:41:02 +0900 > Takashi Yano wrote: >> On Sat, 28 Aug 2021 10:43:27 +0200 >> Corinna Vinschen wrote: >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote: >>>> On Fri, 27 Aug 2021 12:00:50 -0400 >>>> Ken Brown wrote: >>>>> Two years ago I thought I needed nt_create to avoid problems when calling >>>>> set_pipe_non_blocking. Are you saying that's not an issue? Is >>>>> set_pipe_non_blocking unnecessary? Is that the point of your modification to >>>>> raw_read? >>>> >>>> Yes. Instead of making windows read function itself non-blocking, >>>> it is possible to check if the pipe can be read before read using >>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is >>>> returned. >>> >>> The problem is this: >>> >>> if (PeekNamedPipe()) >>> ReadFile(blocking); >>> >>> is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile >>> call. And as soon as ReadFile runs, it hangs indefinitely and we can't >>> stop it via a signal. >> >> Hmm, you are right. Mutex guard seems to be necessary like pty code >> if we go this way. > > I have found that set_pipe_non_blocking() succeeds for both read and > write pipes if the write pipe is created by CreateNamedPipe() and the > read pipe is created by CreateFile() contrary to the current create() > code. Therefore, not only nt_create() but also PeekNamedPipe() become > unnecessary. > > Please see the revised patch attached. That's a great idea. I've applied your two patches to the topic/pipe branch. I also rebased it and did a forced push in order to bring in Corinna's loader script fix. So you'll have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'. Does this now fix all known problems with pipes? Corinna, do you still see any benefit to switching to PIPE_NOWAIT? AFAICT, it wouldn't decrease the code size at this point, so the only question is whether it might improve performance. If you think it's worth trying, I'd be glad to code it up on a new branch, and we could compare the two. Aside from that, I'm wondering how and when to merge the new pipe implementation to master. It obviously needs much more widespread testing than it's gotten so far. I'm a little nervous about it because I haven't thought about the details for two years, and no one other than me has tested it until a few days ago. Ken