From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2118.outbound.protection.outlook.com [40.107.220.118]) by sourceware.org (Postfix) with ESMTPS id 16FDE3858C3B for ; Sun, 29 Aug 2021 21:09:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 16FDE3858C3B ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lSYBP2L8m6EMOZ56sZvkyhG/3bevFOdIhBqeJD63OqB1SKGCXR0zyfROfRObRhNkou/ot/1DEccbsad6xZez7TMjbrLyl2xBPa3Ez67WNX0InA7jR37sfCD2drB7G1ajugI5n2d1hQTNXLI3KJd6R70ji/1OgLx6vZJ6519dfTioa6TW/r1UoaJhWD3eD/8ufg1d0FUOZoX9Va08Q803oMDRdhJyKe5Yu98NyXuYkRtlBE+w5YjPGVPxBNt1IsIUDbhidk/V6kZbBbr1R5sHJQXNpecYWdBG8fhZpGfZUewXo/kUngkc2ZvoHbL3TxMTupDhpUXKzRByfiqfB3/koQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EQjJy8w4Y8voiff0QGNkTONgPh7YGIafEIAms2Gjbi0=; b=f3vZXC1WxDrnU6WVM2ACuEjJgA4VLS7quHVOtf0w9ZAnNlmBZpXd0/7Hw6li/GFUY+bz86x9E0bkdZ5PoZQg5HytxMKeussatm1Ywi0IiUODql3jr/jJlmhZJIeVdCmtDM03oQic/Vgh16XKTfU56s8JXv08LlzkMkwAXE4GbO/SQ+rGF1p020lFXlXEJ5NkCns6Iky6s6rHejiBkurfJVlTHDlrjgyaAEfeNA6Cis7R3BQLmtbLaHLx4EeDodhZg1iDwLUmbjRoyr4NST/vIS6dVnQZbiqzClYe590NRSs+Te1ixBJXngThMHhot5kXDuG51Ps1HYT1FKV1WeM6Gg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN7PR04MB4050.namprd04.prod.outlook.com (2603:10b6:406:ca::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23; Sun, 29 Aug 2021 21:09:42 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::38bd:b608:234f:9ec6]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::38bd:b608:234f:9ec6%7]) with mapi id 15.20.4457.024; Sun, 29 Aug 2021 21:09:42 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin@cygwin.com References: <41A583E1-C8E7-42AB-9F24-EEC33A41EC60@house.org> <20210825201845.07b6400b79dc5558a7761efe@nifty.ne.jp> <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp> <3b560051-ab27-f392-ca4b-d1fd9b5733b0@cornell.edu> <20210827202440.47706fc2fc07c5e9a1bc0047@nifty.ne.jp> <4f2cb5f3-ce9c-c617-f65f-841a5eca096e@cornell.edu> <20210828022111.91ef5b4ff24f6da9fadb489e@nifty.ne.jp> <20210828184102.f2206a8a9e5fe5cf24bf5e45@nifty.ne.jp> <20210829180729.48b4e877f773cb3980c5766d@nifty.ne.jp> <789f056a-f164-d71d-1dc9-230f5a41846d@cornell.edu> <20210830043756.8aa0ada77db0bfbbe3889f62@nifty.ne.jp> From: Ken Brown Message-ID: <47e5dd74-b940-f305-fd5a-c6c9d8f41305@cornell.edu> Date: Sun, 29 Aug 2021 17:09:41 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <20210830043756.8aa0ada77db0bfbbe3889f62@nifty.ne.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0PR03CA0253.namprd03.prod.outlook.com (2603:10b6:610:e5::18) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.1.211] (74.69.128.111) by CH0PR03CA0253.namprd03.prod.outlook.com (2603:10b6:610:e5::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23 via Frontend Transport; Sun, 29 Aug 2021 21:09:42 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cc693d17-bc6b-41b8-1257-08d96b315201 X-MS-TrafficTypeDiagnostic: BN7PR04MB4050: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: H5GWjIeDBdKx5JbuLyyA/xOHvrsbcaagnM1sCQuXzOSSGAnAP7lgpTMOkHTWnLkgrS42GnCAD41kdzqGNs7jDm0nLWAFec9oCcjkgyH3gr68om0+P6rF/MMTkTj2i+ZicKRc7BJ0F1Ux+aPxfmSb8V6LKYFpNYhCRrM7zHYP04Jw4+DBmNYtFh+ihJONY5hL6x35yA27Mi7BpJKVH4ETHz8gb/y8w1GX0KQPtifXQ2UFBpqCzEKWDWrUN6MXuwg8tDSz3oTgVoLbELnckievtf/bjl0AxS4jdySng4EwBaKSZPBEpZEit107dzt/c8W94tOtiSSWvn0IGpV9TWAzb4ou8OJtlX5hlAVW2I6Z4YIdNVSfRKIE2rU7YCP6jP8ORiymfAPVgTO6TinoyUvOutBPYbSVQqVPUOYy4XwzL+ZvKxKxGZPSBLYb1JdKtm82/K/ODlXuoC0ShAcVz1c+4Re4g2V1W81ghBbOzbSbTg7QvAoTglUgxW4pJvoju0PcXVZ1zQg6ms9jjS4hxQWi6zSnlFrCH62aebua+bD+dJVXnin1ug1ez0msCCb9t5aCp7f1q0pQ0NPDh58hJU9lYwPQ0DR3byUSw9Q7b1ESczyRwj3cHMmOz6zc8m6npIbFaSxbKCRiSucYfJtKf6Qua73y7DmF3JZ1qSXquNtv8ES6Il4h84x2c9t3LE35hBkN/cefHVrJ1Zn5BlDIf/vgZsYd7b8I9NKcuL3Jv0MbE4O4lLUVA0FoZB+KsnpSNMgQ7h663XJfVQLuJsxwMTMEIYsZVnX1Voxqo59fhDhvdl0m6iabacWY9Jt3bO72PwkE X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(366004)(396003)(136003)(376002)(346002)(6916009)(5660300002)(75432002)(8936002)(6486002)(53546011)(26005)(86362001)(316002)(786003)(31696002)(2616005)(966005)(36756003)(186003)(8676002)(956004)(66476007)(66556008)(478600001)(83380400001)(2906002)(38100700002)(66946007)(31686004)(16576012)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?AGvZaW1aZN6JwHQ8eaSpIk8M2MrVgQpJdf/ACTJYRTXi6d1yawd/HXOk?= =?Windows-1252?Q?Eez2EKZnLFMhSbO/qvd2y9Vvnyi8VN0o83ljeSUuOj0VUkZ6oLFvSA94?= =?Windows-1252?Q?YjhmYfiA2aO9pq70PppXBf7uX8OK+Tq6zsuC/pZGv3TjlNXbXPO341iw?= =?Windows-1252?Q?NroXexZDV4/OVD/l56aeKxvUNCH/dGvgDdHvXdXWPZBLiDu5QJ7ivNkz?= =?Windows-1252?Q?NSAb7IJlMHJF1j7Ur93lPhpcvyls7ZfKftIe9N8zTGQR7yYNpPT9SXkR?= =?Windows-1252?Q?jbKaLKdf13WKeRBrQIEyYfxEhjspkCe1wrxUvnVkYMl1dQqvGV7W9JHE?= =?Windows-1252?Q?8yvXyU0k4fTkYTiN+bBrEqLW/QbGcDR7h+FixqQIIF37kDsgLXd6RHm3?= =?Windows-1252?Q?VaPrrfCaA4nlpeDA9qUlvJ5Ih7RcXpsZdegwY0fzU1CGdR5E/NEjV/ta?= =?Windows-1252?Q?A0g0cnWXRfoMVOzJonBlVz0FzIJ8Px8DDiPY3jKF4nr1ccT01wsSd5nQ?= =?Windows-1252?Q?Z3VFjJrAiLHG0mBIp6x/dT5T3rZ+2WgPObnR1ztoRXQVQ0bETjQegfLa?= =?Windows-1252?Q?OJEl7Ur9yE+JZ3e0jygUXv18DCJQgo3wJdEoY8twTwhwV1vCd/aelTwx?= =?Windows-1252?Q?rXErohouo2lV+9CpknARSVrBU8vZRMy0OaEAM+Y3PwxK2m55jeMjf2V5?= =?Windows-1252?Q?fPAin0E4mgHu2NFovdMUtKSc7Te7SVTftnOnlORJZCyFxpE9XTqorU6r?= =?Windows-1252?Q?G5jscYGasqZD0gOvIqzS97oPMKbPI9KBqowuOkvcaM1MD2BOZ3JkpOf4?= =?Windows-1252?Q?a/ph3zWn0JBHAwdig9kjICsKiJghBkkHl5Nd5v3+GcqIskY2WXwpVgEI?= =?Windows-1252?Q?6fzasFtT0GH4NWketXU0x0W2kw2sdkKbSp9bD42yB/ZnOS/DWeG/8Ud5?= =?Windows-1252?Q?6XK/YPVqGsCPMRe+4P7urXf9FnkW7dhrovulxwXULEaVKvgStxreFald?= =?Windows-1252?Q?yTeepr2bJL4TROJJjjMLCDks12rjA26YF1BiLlWTw89NP/mat3V8TXv2?= =?Windows-1252?Q?T+rmLl47K2TPrh0e6r/x9plFtgUaRWJSne/8nFtdFJDqaSfe1FNlwnqD?= =?Windows-1252?Q?tu3sboFxR7kjfBhpx1gu3lqoNCe14LygQ5EPvyv3N4LgGTTD9UxAoq4+?= =?Windows-1252?Q?2Aw7Oeoi0FnyozT3Ki/9C5YX2+MSu9dQc9Q4CHtIhUdiep74LE/sXTMM?= =?Windows-1252?Q?FnAdHVr8x0WqqMYGYG9fz7robF6LgHA50irWLvrrx+SvJc2+45Xmk7Rm?= =?Windows-1252?Q?lCO86CqQx3WDWwscr80UPN+hbG8SHCYfMljKCQLyJvl3Ui+8Tj3JqQFs?= =?Windows-1252?Q?n5MwTY+TAoLTG8GcVsBD3MTpIUNREvaIt++d++FmQLyniSQxuvtGmjUi?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: cc693d17-bc6b-41b8-1257-08d96b315201 X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Aug 2021 21:09:42.4106 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aC9G7HySdsCarp7LsvFysoOyk3wFJvsAL7YrHo2V5WtARSmH5ZjDX8kBxhHfyFsrVsKCplZx4U5SrLfO2GvmNg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR04MB4050 X-Spam-Status: No, score=-3.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_PASS, TXREP, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Aug 2021 21:09:54 -0000 On 8/29/2021 3:37 PM, Takashi Yano wrote: > On Sun, 29 Aug 2021 11:57:04 -0400 > Ken Brown wrote: >> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote: >>> On Sat, 28 Aug 2021 18:41:02 +0900 >>> Takashi Yano wrote: >>>> On Sat, 28 Aug 2021 10:43:27 +0200 >>>> Corinna Vinschen wrote: >>>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote: >>>>>> On Fri, 27 Aug 2021 12:00:50 -0400 >>>>>> Ken Brown wrote: >>>>>>> Two years ago I thought I needed nt_create to avoid problems when calling >>>>>>> set_pipe_non_blocking. Are you saying that's not an issue? Is >>>>>>> set_pipe_non_blocking unnecessary? Is that the point of your modification to >>>>>>> raw_read? >>>>>> >>>>>> Yes. Instead of making windows read function itself non-blocking, >>>>>> it is possible to check if the pipe can be read before read using >>>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is >>>>>> returned. >>>>> >>>>> The problem is this: >>>>> >>>>> if (PeekNamedPipe()) >>>>> ReadFile(blocking); >>>>> >>>>> is not atomic. I. e., if PeekNamedPipe succeeds, nothing keeps another >>>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile >>>>> call. And as soon as ReadFile runs, it hangs indefinitely and we can't >>>>> stop it via a signal. >>>> >>>> Hmm, you are right. Mutex guard seems to be necessary like pty code >>>> if we go this way. >>> >>> I have found that set_pipe_non_blocking() succeeds for both read and >>> write pipes if the write pipe is created by CreateNamedPipe() and the >>> read pipe is created by CreateFile() contrary to the current create() >>> code. Therefore, not only nt_create() but also PeekNamedPipe() become >>> unnecessary. >>> >>> Please see the revised patch attached. >> >> That's a great idea. >> >> I've applied your two patches to the topic/pipe branch. I also rebased it and >> did a forced push in order to bring in Corinna's loader script fix. So you'll >> have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'. >> >> Does this now fix all known problems with pipes? > > Only the small thing remaining is pipe mode. If the pipe mode > is changed to byte mode, the following issue will be solved. > https://cygwin.com/pipermail/cygwin/2021-March/247987.html > > How about the simple patch attached? > > The comment in pipe code says: > Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE. > This *seems* to more closely mimic Linux pipe behavior and is > definitely required for pty handling since fhandler_pty_master > writes to the pipe in chunks, terminated by newline when CANON mode > is specified. > > This mentions about pty behaiviour in canonical mode, however, the > pty pipe is created as message mode even with this patch. Are there > any other reasons that message mode is preferred for pipe? No idea. All I remember is that there was a lot of discussion around the time that it was decided to use PIPE_TYPE_MESSAGE by default. Corinna probably remembers the reasons. Ken