From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2094.outbound.protection.outlook.com [40.107.223.94]) by sourceware.org (Postfix) with ESMTPS id E622F3858400 for ; Sat, 11 Sep 2021 13:12:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org E622F3858400 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J8J04wpsA1VGtdSp+its50zovTD8rraU5xTAXmzBepwUoqGT9hHc13YsVBfPi+giQTsncMd16W9jnemA+Ln+gArQyq75TA4FwyHh2IZMnG4zoyZIRABn1/lIpMtgDyJsiJO+x1mTWT0431MCm6EDAnQosNlcqlZZ4XLlr3NeEVyaokyrlGUVrhFvGuzfo5+cr7a1ef42is/QqbZV0jyLFjPsN0w6djl8ZwTG6H2Mxw5frsg9HQWDnSY5B/whvWeB3/BdGiNpX/yhFwdL+5F79lY/2Zl8M6rUB17Du6InFwt2vbRyyNOxNxVFaFv3vn/aXn/uC6g4r+xMPgDN5kKNDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=qqimIMCtPfHlOFxA34DT4lMj1cIFcCcD6C7SWcr9KE4=; b=MeTC3ZfZSfsh0jea+JcXLm9LuTyhcOeT9F1eVcI+mMbLK+OlOTMBolfkFCXXokyMb4c87kx/1N65yiHlzhufOmVnBANSlrzpeiyRCSP3KW1Tk/X5ywuMehBKQ9JSAbzu79+hz5EQZwPYV4ah7XE9HwQZYfsMeXBDhbWzAYgMhu7Eg7e42lv9Tbl12D0OQto9CUDlrkFQXBlrs8iqDTHLk+TIYbOEjSkrIPLvF9ICzW1rsQqpRCxZvHvuGeCEkeeLXr/kCMlVMqUAb3zQ4Cy4bootZhPlj03KedX1TyYagxPY44IGR1e7l5jmszcQuAkRY7OF2ifzAsr89R4czs4MEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qqimIMCtPfHlOFxA34DT4lMj1cIFcCcD6C7SWcr9KE4=; b=Dx8CE3bX4DyMoETPnCJDcaYCU11hFH7x/VoJzr4TufmiH3JH1LjjIJRJ9GVD/hzgCabbdWzVOBzCyDnYtj303DOVTHnPJg8vMq4h+il79MFUpqhvQ7Uo9/0tp01+dHH3Rlb8aMP1+o34+9Lje5QcOOmzs+bpkO1Fi8gcz4Nh7/g= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN8PR04MB5825.namprd04.prod.outlook.com (2603:10b6:408:a5::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.17; Sat, 11 Sep 2021 13:12:04 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66%6]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 13:12:03 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <41A583E1-C8E7-42AB-9F24-EEC33A41EC60@house.org> <20210825201845.07b6400b79dc5558a7761efe@nifty.ne.jp> <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp> <3b560051-ab27-f392-ca4b-d1fd9b5733b0@cornell.edu> <20210827202440.47706fc2fc07c5e9a1bc0047@nifty.ne.jp> <20210907122631.65452be8d021ec72259431d5@nifty.ne.jp> <20210909124115.555c6be15d675500617d284a@nifty.ne.jp> <20210909170549.506cc3c1f6029d904fece6dd@nifty.ne.jp> <20210909211940.51ef391e27d43f0421962cb8@nifty.ne.jp> <20210909214246.cd1ff1a3062fea27e51ad4ae@nifty.ne.jp> <33386baf-3b2d-d57f-2ad3-1bd328ed7935@cornell.edu> <20210911075734.aaf37697ba7db2ad14d911a3@nifty.ne.jp> <20210911113517.f74fc3ac1971bbf04c7a9bd1@nifty.ne.jp> From: Ken Brown Message-ID: <695ce1f4-4f7d-f3f3-6dd3-087467d67b28@cornell.edu> Date: Sat, 11 Sep 2021 09:12:02 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <20210911113517.f74fc3ac1971bbf04c7a9bd1@nifty.ne.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH2PR11CA0016.namprd11.prod.outlook.com (2603:10b6:610:54::26) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [192.168.1.211] (74.69.128.111) by CH2PR11CA0016.namprd11.prod.outlook.com (2603:10b6:610:54::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.15 via Frontend Transport; Sat, 11 Sep 2021 13:12:02 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 30f4cb16-cbde-4656-410f-08d97525bf1f X-MS-TrafficTypeDiagnostic: BN8PR04MB5825: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YtH+qJGB3poltD6so97prTS/atjEdsqRBKzZZ3fgScBgAbNJhG7u6hAqxVwtAe3jPTOnecaiFF31VTy2tXTeRsZ5KBMkqHjs/7sfZWWPmczzHLnzKr5xRhruUuARvl5wWwVbvhgziW2E6JgetlHC19KsWB6NRJjEP6QSCXu+IWDwQO3H2M8qKju0E0Z776+C/UCcRRQ2cpquARKVf7bpswM6JG3deE9yvH3OKG6UUE+Mp1nOl41Hm07jeQXETPcJTh5rWoJNREdNKmJKRsoYyXIYv3DZPLm/AzOJPjMDIej5anQKVFv0xAk5O7m7MEJSWgigGQaRmHN9gqsamR4xYi3rg2z6drh1oxQBBMrGXoi+J+vMwjYk8uk1wv7XcTPRcT7kqhKjp2S+lWDgHQNRR5pE4FT14Ulm0Wp43OW4f3T4xL2N1ve/N+hxrfcxqkdVAmlYHzmMtSM2A6khCMZpvJswTT2Xujvq4wuwQQJQy22buBxEWUsJKdVPEiqJu0dJH6WLKR51wUzTXNKhS3HUz+zPyiJ1hCDeH6JVrFeCjdvpYBkjyVEJuk7jqRTH33xj5bwMivdTNBAXMV3iZ5e0ijRVS4HxmMPEHnf6k8LM8e5CrddX+dD90F34p55dydOOQqcF+BZTe9yAZxeYOytx7kdZqznL9hb5UmcP2d0pzu+v7dF0A4MsEk8eEX/zZ0/eazsVWT8PW5fahKcZWTrv7C12iJ61cUx7m4JUV4cB6HOYV484VOHdpEWGIOMUmq35 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(346002)(396003)(366004)(376002)(136003)(36756003)(6916009)(83380400001)(31696002)(38100700002)(26005)(75432002)(186003)(8676002)(5660300002)(86362001)(478600001)(16576012)(316002)(53546011)(8936002)(66946007)(66556008)(66476007)(6486002)(2906002)(31686004)(956004)(2616005)(45980500001)(43740500002)(460985005); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?gnWgcEAnj48FcsQSVdcNN5XzjYoy3e7da1FxgkcKpz8VPvarwjrFSoR9?= =?Windows-1252?Q?MrMJD5tpbb4k3l6JoI6vxnNbor/W/9T6pZGWtnKaJVFQJWPjgeAnjqiO?= =?Windows-1252?Q?rgcQ4qQOjGZReK60u+S1kHaywfX6QUXGAfyBsws6zJbqsoZyiTPgeC8A?= =?Windows-1252?Q?8A/ItCmYILdj2PjXVXfTlr5CQ9hfSacumfgboZrQdi3P5eozol62o5hI?= =?Windows-1252?Q?1oKz1lxSWlyCccBHU4uvsnpjR4UqEMGsR2FO43sT9PXnpN1L/zPyDHMU?= =?Windows-1252?Q?du4PxkfXu4J4BzVgiQprbxgtqtchCIJOw7Ok5PTvb+OZaUUzq+W2Yiuk?= =?Windows-1252?Q?s1p5jkCQ4PqhBmGFm6sWrOCkg7aCUB5XLA3aGfY1ndPV+mSfMz5jJ8Pb?= =?Windows-1252?Q?Hsu8h5AuNSyAUSMMcJhV1+Mqzj8cJM/q5CiwixMZPhQjF5snPM7Jqbyq?= =?Windows-1252?Q?O1CaQPH0csieUoT8zW/XI6M0FJgdd/p5ylfyu72sJz59OpKp7j8LJqpc?= =?Windows-1252?Q?k1x3DQeTLKVf/eLngeNwyrFQm6y6ecQmlA62I8m+hN0i+gnVCDv7ZDG2?= =?Windows-1252?Q?CSXS+jvEyT2REi+byfabT6Q0PgEp3M42hg7QIzgdID2r0T48mnB4w4Ry?= =?Windows-1252?Q?YbDAYkbEbvToCO01tbC8a7EYAKcy69RVR5hT3lvYY+et5UqOJVpyakkw?= =?Windows-1252?Q?bXEeFUYNrJq93c01qB5LWhghgt+mQSYi5RWecLafBT+xkXMvVkvbp31m?= =?Windows-1252?Q?ycwGn8uN0aXftqbTbfWcFZSVUgzS9aCk2piyrkim6fauBuIoCYZqmi2i?= =?Windows-1252?Q?9VX2B8l6evRU5AA9UeAVfgcVSdLeHNY8jqqN7RENKLaChhsLkVwOOHqB?= =?Windows-1252?Q?aTqtFVyDt9jXeaT7J3y00EOmpriDYAEcxN54Ko2iAHHteLArY1GcPT6r?= =?Windows-1252?Q?5jnPr/zBy/nGCJD40XeWnYOQSim1u2I7JOAvno7JGdxyejVZMR9Ryfol?= =?Windows-1252?Q?6jn46m/IcXON4RQ/sew6EEd16qdHQFHHtMGKwCxM2cD2F7rC1FU0/oJN?= =?Windows-1252?Q?plSzQ/9SXNVNuwzb3MHDb6HMx2uVyLnd7v0+5WUBS0kMUIATAks9IJZ2?= =?Windows-1252?Q?fwep8VlvH6dVtDa47uyMuPZKSvUhg6cDFE0Oe0miiDqzm3OSR/QbLG7Z?= =?Windows-1252?Q?/E3vFTGGmAoapcCG3FvdYB8bcS3N5WuZJ0GlTpbfRWLtvoCGypmq4pNR?= =?Windows-1252?Q?OcYT1X25fpzJch8wgbSX06naYETy9oMkz3jVMVzFg5oSvDC3NPxojS0i?= =?Windows-1252?Q?fmb9CBh8E+zQGAM6KNKjsAbvrRiA8hwHIXMH3HDczYlCpWogTNJHGPzE?= =?Windows-1252?Q?xkRAt+OEKxCII6brRX+ejrRL/I7ZoqFthLJIEZJYs4Jdfe/N/h0fkLTr?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 30f4cb16-cbde-4656-410f-08d97525bf1f X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Sep 2021 13:12:03.2950 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4VTkB3PDO288nZVgtoONvW754EF1Gqfh+hl0oGiHMRXkD5Xqw9wfiUZztfSnl6C4T3K5y0yg5ocIu3fKv0JL9A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR04MB5825 X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2021 13:12:10 -0000 On 9/10/2021 10:35 PM, Takashi Yano wrote: > On Fri, 10 Sep 2021 22:17:21 -0400 > Ken Brown wrote: >> On 9/10/2021 6:57 PM, Takashi Yano wrote: >>> On Fri, 10 Sep 2021 11:17:58 -0400 >>> Ken Brown wrote: >>>> I've rerun your test with the latest version, and the test results are similar. >>>> I've also run a suite of fifo tests that I've accumulated, and they all pass >>>> also, so I pushed your patch. >>>> >>>> I think we're in pretty good shape now. The only detail remaining, AFAIK, is >>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin >>>> process. I've proposed a timeout, but maybe there's a better idea. >>> >>> I am not pretty sure what is the problem, but is not the following >>> patch enough? >>> >>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h >>> index d309be2f7..13fba9a14 100644 >>> --- a/winsup/cygwin/fhandler.h >>> +++ b/winsup/cygwin/fhandler.h >>> @@ -1205,6 +1205,7 @@ public: >>> select_record *select_except (select_stuff *); >>> char *get_proc_fd_name (char *buf); >>> int open (int flags, mode_t mode = 0); >>> + void open_setup (int flags); >>> void fixup_after_fork (HANDLE); >>> int dup (fhandler_base *child, int); >>> int close (); >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc >>> index 6994a5dce..d84e6ad84 100644 >>> --- a/winsup/cygwin/fhandler_pipe.cc >>> +++ b/winsup/cygwin/fhandler_pipe.cc >>> @@ -191,6 +191,17 @@ out: >>> return 0; >>> } >>> >>> +void >>> +fhandler_pipe::open_setup (int flags) >>> +{ >>> + fhandler_base::open_setup (flags); >>> + if (get_dev () == FH_PIPER && !read_mtx) >>> + { >>> + SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags); >>> + read_mtx = CreateMutexW (sa, FALSE, NULL); >>> + } >>> +} >>> + >>> off_t >>> fhandler_pipe::lseek (off_t offset, int whence) >>> { >>> >>> >>> AFAIK, another problem remaining is: >>> >>> On Mon, 6 Sep 2021 14:49:55 +0200 >>> Corinna Vinschen wrote: >>>> - What about calling select for writing on pipes read by non-Cygwin >>>> processes? In that case, we still can't rely on WriteQuotaAvailable, >>>> just as before. >> >> This is the problem I was talking about. In this case the non-Cygwin process >> might have a large pending read, so that the Cygwin process calling select on >> the write side will see WriteQuotaAvailable == 0. This could lead to a deadlock >> with the Cygwin process waiting for write ready while the non-Cygwin process is >> blocked trying to read. > > Then, the above patch is for another issue. > The problem happes when: > 1) Start command prompt. > 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat > This causes hang up in cat. In this case, pipe is created by cmd.exe. > Therefore, read_mtx is not created. Confirmed, and your patch fixes it. Maybe you should check for error in the call to CreateMutexW and print a debug message in that case. >> My suggestion is that we impose a timeout in this situation, after which select >> reports write ready. > > Keeping read handle in write pipe (Corinna's query_hdl) causes problem > that write side cannot detect close on read side. > Is it possible to open read handle temporally when pipe_data_available() > is called? That would be nice, but I have no idea how you could do that. Ken