From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2136.outbound.protection.outlook.com [40.107.236.136]) by sourceware.org (Postfix) with ESMTPS id 0D1833858D34 for ; Mon, 30 Aug 2021 19:12:20 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 0D1833858D34 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PSmZc96UxG5JKAU/Dw/ZOIvYosq68V/uVwdkquEut9GYArw6SXwjFD1cMmFIV0dvaPRLGRCVlyMVCVGRj3J+CZayNa1AymJHLo12lJiaZVO4Wgnu4SDDPMJRTToT9oPVOu1exixqPGvsw0D0Cus3VfinymW8SouKXZZmX06PvOj7wq68zIHf24Knqdc4ZQGeXPNYTqFdkPLeU8ODhmlOJIQkuJXjSxc4VWPMWQ4E2wrhR64jB1Yfy0I9YgoeiTDUynSSXegC8uQ679w0bbk0rMU7dShrP5thN/NBu0l00UO6hocPxEw08Bgd1Jj4bktmTBYONl7ErgQMojgSxnBsTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EgaDiP0z8Ujjg2u9i4UmIEuzLU+TCzEYywecWgcgrvU=; b=JP5c9DjwVlpgPANKUtSm3SUPjdF8Q8i1th9nJ3BQa1K6/LdHwPtbTkPG2+yKnY8GsngX0lDdo6CxYRhTASQ4G/kgGWjU3/X6Z1IhkbdiuKVtx2kRZeqtxVm07TJpJjtNw8Oqru2H46qmb7CkVz/PT1N9kmATqycibjEGI11eGR6BvwaqwkxT64d50WyFbRBQBAufrFHmu16hcjSvCJ7mKLYzha4pg/398uNI4Ye00yNVflkqNDTtK1pjOUDDmokl3HoeXdpW2nUjlZMhROE5DZ1SShEBF1GepGOKjV3hl1djlz/dbzTuErMr27rFCmHH5pqbH78zNyrWxJI4LVbY+g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EgaDiP0z8Ujjg2u9i4UmIEuzLU+TCzEYywecWgcgrvU=; b=MSvI+MdjRQ8PHmbWIm94VUCM8m7eLb+LdUbE9uXR/ltPEHKoIhU/dFHemSKpqXYh/PMjccabTa7ek9TrBCi//z5u2ZtpSScZkH/YEobW2rnt5TPTIwXL8+MWxtoGttZEKczJOsMboJcr/wjXM2TpZzbhsPJxRN1PZyX02SLrGjA= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN8PR04MB5474.namprd04.prod.outlook.com (2603:10b6:408:5a::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.19; Mon, 30 Aug 2021 19:12:18 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::38bd:b608:234f:9ec6]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::38bd:b608:234f:9ec6%7]) with mapi id 15.20.4457.024; Mon, 30 Aug 2021 19:12:18 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <20210828184102.f2206a8a9e5fe5cf24bf5e45@nifty.ne.jp> <20210829180729.48b4e877f773cb3980c5766d@nifty.ne.jp> <20210830091314.f9a2cb71794d0f68cdb5eba7@nifty.ne.jp> <20210830092259.52f7d54fc3fa340738373af4@nifty.ne.jp> <529d7dd7-d876-ca51-cc1f-e414d3c24f71@cornell.edu> <6f4d6673-0128-cc62-3a5a-5b0897fbbf3f@cornell.edu> From: Ken Brown Message-ID: Date: Mon, 30 Aug 2021 15:12:18 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <6f4d6673-0128-cc62-3a5a-5b0897fbbf3f@cornell.edu> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-ClientProxiedBy: CH2PR05CA0064.namprd05.prod.outlook.com (2603:10b6:610:38::41) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.1.211] (74.69.128.111) by CH2PR05CA0064.namprd05.prod.outlook.com (2603:10b6:610:38::41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.10 via Frontend Transport; Mon, 30 Aug 2021 19:12:18 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1cc8c0ee-36b7-4943-8635-08d96bea15f0 X-MS-TrafficTypeDiagnostic: BN8PR04MB5474: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QP6IT6yKLvaDvLgZRXM1VjEe7Rcr+l11NemII3JQsIRCmocC42a0vcRCdTc5RZC5dxCAq1EhaxF2gfVMrpUhhQawIYc0TV/0PDXyjFzgjW8kGN2MTWdandxG4OXry0V8Zzm94JsND/bOTN4BAlBex+vWzjIPHL+WpCzvUJi5NztphVEQZeaBEFspQWKEulXtlEh6+PSznK9aee5ATBTbn/vriBHAgW9hG4KnOlRYIRSNzymgVVF4Wc9RsDpcwwPiKNoCnlM8Vboj8nGjy9JtIYKg5DYubpUbibH0pOVCEllLSGmOeTG89VOrqe3c7Eo431DlLVd6nItU7b3gJm/nsVNKLc+0gOsT3/vREPlKa8rzByJ1Fnva4pRz70jXh1lMFd8MLTpHvcgfeOY95DvjOrbUYFd9ySGQtJ3hYA8znQl4WaE67V4si08RkdifEgNnQOi/1TAfPqsLcbnXeKHysjxPFQEDYl+vPkC+AWdnOIJqKLPi+IP/h2GXJb+fNk1pFKaqks+5X3DzjLeRbgdo1hP7H6b1ccJyLxkc5+F6B0F8McuXZ2g816fCm6doBGymzeY/VspYzXA7gJs3tccSucB7FVqxvUgS7C0nrsIkow1/dIUDEQNd1Hby8yBS2cZKOt5Pji3ds5YooMCBn9IBOMeSGfvdGyQGb8d8/UNhkhf5BviUXBeH/tKsADIVI8sN1ZnNofKMiZw9Q7tAaITzj3QkJ2sgvrD6nzpdXHqm9jU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(39860400002)(376002)(346002)(136003)(366004)(478600001)(186003)(26005)(31686004)(2616005)(8936002)(83380400001)(2906002)(53546011)(36756003)(786003)(38100700002)(6916009)(86362001)(75432002)(66476007)(66556008)(31696002)(5660300002)(66946007)(6486002)(956004)(8676002)(316002)(16576012)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?ujPf/O+c3ZWxRj+0l04jVyOub5ggRobOYZHyn9ox2+XHGfQ3zMF7MrH9?= =?Windows-1252?Q?j36iHDiH2oN3avKyFxPSbgInbHVaZm8p8y8WbKyeNibJ1to5jDwa3ydo?= =?Windows-1252?Q?/Wr1eu5dwppjuIiGI+NAqyGO33miz4fiDbITjoj9FztOAP018fTBXSST?= =?Windows-1252?Q?qtBoO0XTQatKh8W6uAwTNypFo/Ry02M1GqXU8lto7qEBL8z7PihSCi5w?= =?Windows-1252?Q?BJ4g116YJ0f8L8J0Svz1LoIgAMYbDAKRIS6U7sK8ZMwyTLks9Pem++TI?= =?Windows-1252?Q?BsY8d2NEDaYbqjchxiB8b+fphhedl6Ka4SoNcyXhCZSQqLu9771Nwi2I?= =?Windows-1252?Q?/fkszYkvjHX951Jyb9h+XfmbiDAYFg9OWRrUPBfRQ/ATawn2/T0uNurr?= =?Windows-1252?Q?dgPjbC2BV5QA4R1CjHzGuDwz1YzOOF6Uxdlu21zcDCO3w1cPIf5lXOuI?= =?Windows-1252?Q?KVqEj0T08KlnGfk3BMxQD249wUwu90umoO8jv9idjJZj43qnbpxUciuz?= =?Windows-1252?Q?NneHtTl2ZfOrxSurHuggP2slyKRrxR6N0FbIS0T6NVnqHdd63MGC1WW4?= =?Windows-1252?Q?B6lLWgAkr1TD9fB2pCw3V5qJ98YUHFzVgHaaUG0FeX9xTMKBhrAqdEX4?= =?Windows-1252?Q?GNWltpfFYSMorPFo/jgpuuB+ii7/lOMeu8vocp3pj5QWH6oXFNPzp6uJ?= =?Windows-1252?Q?ojzHODTrkb0OMaCMUls/Rfcny0l1Va/mWr6EGv3Nh4J+wqg1eQYJ1LUd?= =?Windows-1252?Q?ya11BOv/RE6jJnodbA9/bnKipWA/MIXZZY/1iQevcrU0EHirrnNpZ+70?= =?Windows-1252?Q?1h989X0iqJ24fFCx7uhibhSm+ZRRKPuXQCbPJccGAHF0wHgRpMUuPUQq?= =?Windows-1252?Q?GNCyfVlbuUf4nwDEh4Xe2A9QwS51tL0G5aZtoX3WwvXw9OoSHgrx9cyW?= =?Windows-1252?Q?ZLLFglhCjoowwwqK28JOJAqABJ3qe1vmhGwtgZF3GAyWTAIq5o5z144O?= =?Windows-1252?Q?/xJSK7P2N//nJV1KSElcTmzjxsINyvrIps6WBMP3D3Ep26lhqzFpFSsy?= =?Windows-1252?Q?wmM06RK5aJgiQuJMlpb1AyYR5kjqQ9Eh/iHzTvOaq5hSyYqCbf+HXm6m?= =?Windows-1252?Q?kecYW9v0WnrynIY0+yzcElZYickup/BxlLmGYhXh412VLo2XevAVpXy5?= =?Windows-1252?Q?ZNBwlwfRgbbeeAWZJbkxIEcgF0/TCpE11vn0fnMWvNWzHTUuo0ndacyT?= =?Windows-1252?Q?A6+qvhglhixVF4MfKFElJUoD+QN+uUfoUcJrd3jINPL737MDYSatrSGl?= =?Windows-1252?Q?9Ku2JXiKh4pSYL/LGM975x+25AneXVtiUiIeXm2HG0xNqE84BxXIkf88?= =?Windows-1252?Q?DfDAEP9pkflbhIqIn9dR/AB7C5GsEoCxA9mDCH0vb+SEVKXddJ9UxZWC?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 1cc8c0ee-36b7-4943-8635-08d96bea15f0 X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2021 19:12:18.5708 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bcTSusQZocVFojsbQ8lTZ8MCJ9bXFGHA+fSWV0cbQedeXRiCy4BlHms+l/pv1wM0GaZK5teKfxnlWZ9ZE2Y4Hg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR04MB5474 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00, DKIM_INVALID, DKIM_SIGNED, KAM_DMARC_STATUS, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Aug 2021 19:12:30 -0000 On 8/30/2021 2:59 PM, Ken Brown wrote: > On 8/30/2021 1:00 PM, Corinna Vinschen wrote: >> On Aug 30 17:53, Corinna Vinschen wrote: >>> On Aug 30 16:05, Corinna Vinschen wrote: >>>> On Aug 30 09:36, Ken Brown wrote: >>>>> BTW, when I was working on the pipe approach to AF_UNIX sockets >>>>> (topic/af_unix branch), I had occasion to step through >>>>> select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota - >>>>> fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote >>>>> peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi >>>>> is suggesting now. >>>> >>>> If that's working reliable these days (keeping fingers crossed for W7), >>>> it's ok if we use that.  We may want to check if the above observation >>>> in terms on WriteQuotaAvailable on a pipe with pending read is still an >>>> issue. >>> >>> Ok, I wrote a small testcase.  It creates a named pipe, reads from the >>> pipe, then, later, writes to the pipe.  Interlaced with these calls, it >>> calls NtQueryInformationFile(FilePipeLocalInformation) on the write side >>> of the pipe.  Kind of like this: >>> >>>    CreatePipe >>>    NtQueryInformationFile >>>    ReadFile >>>    NtQueryInformationFile >>>    WriteFile >>>    NtQueryInformationFile >>> >>> Here's the result: >>> >>> Before ReadFile: >>> >>>    InboundQuota: 65536 >>>    ReadDataAvailable: 0 >>>    OutboundQuota: 65536 >>>    WriteQuotaAvailable: 65536 >>> >>> While ReadFile is running: >>> >>>    InboundQuota: 65536 >>>    ReadDataAvailable: 0 >>>    OutboundQuota: 65536 >>>    WriteQuotaAvailable: 65494    !!! >>> >>> After WriteFile and ReadFile succeeded: >>> >>>    InboundQuota: 65536 >>>    ReadDataAvailable: 0 >>>    OutboundQuota: 65536 >>>    WriteQuotaAvailable: 65536 >>> >>> That means, while a reader on the reader side is waiting for data, the >>> WriteQuotaAvailable on the write side is decremented by the amount of >>> data requested by the reader (42 bytes in my case), just as outlined in that >>> mail from 2004.  And this is on W10 now. >>> >>> What to do with this information?  TBD. >> >> Ok, let's discuss this.  I added more code to my testcase and here's >> what I see.  I dropped all data from the output which doesn't change. >> >> What I'm trying to get a grip on are the dependencies here. >> >> After creating the pipe: >> >>    read side: ReadDataAvailable: 0 >>    write side: WriteQuotaAvailable: 65536 >> >> After writing 20 bytes... >> >>    read side: ReadDataAvailable: 20 >>    write side: WriteQuotaAvailable: 65516 >> >> After writing 40 more bytes... >> >>    read side: ReadDataAvailable: 60 >>    write side: WriteQuotaAvailable: 65476 >> >> After reading 42 bytes... >> >>    read side: ReadDataAvailable: 18 >>    write side: WriteQuotaAvailable: 65518 >> >> After writing 20 bytes... >> >>    read side: ReadDataAvailable: 38 >>    write side: WriteQuotaAvailable: 65498 >> >> *While* reading 42 bytes with an empty buffer... >> >>    read side: ReadDataAvailable: 0 >>    write side: WriteQuotaAvailable: 65494 >> >> Another important fun fact:  Assuming the read and write buffer sizes >> are differently specified.  I called CreateNamedPipe with an outbuffer >> size of 32K and an inbuffer size of 64K: >> >> After creating the pipe: >> >>    read side: >>      InboundQuota: 65536 >>      ReadDataAvailable: 0 >>      OutboundQuota: 32768 >>      WriteQuotaAvailable: 32768 >>    write side: >>      InboundQuota: 65536 >>      ReadDataAvailable: 0 >>      OutboundQuota: 32768 >>      WriteQuotaAvailable: 65536    !!! >> >> This last data point shows that: >> >> - InboundQuota and OutboundQuota are always constant values and >>    do not depend on the side the information has been queried on. >>    That certainly makes sense. >> >> - WriteQuotaAvailable does not depend on the OutboundQuota, but on >>    the InboundQuota, and very likely on the InboundQuota of the read >>    side.  The OutboundQuota *probably* only makes sense when using >>    named pipes with remote clients, which we never do anyway. >> >> The preceeding output shows that ReadDataAvailable on the read side and >> WriteQuotaAvailable on the write side are connected.  If we write 20 >> bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is >> decremented by 20. >> >> So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable. >> >> Except when a ReadFile is pending on the read side.  It's as if the >> running ReadFile already reserved write quota.  So the write side >> WriteQuotaAvailable is the number of bytes we can write without blocking, >> after all pending ReadFiles have been satisfied. >> >> Unfortunately that doesn't really make sense when looked at it from the >> user space. >> >> What that means in the first place is that WriteQuotaAvailable on the >> write side is unreliable.  What we really need is InboundQuota - >> read.ReadDataAvailable.  The problem with that is that the write side >> usually has no access to the read side of the pipe. > > For the purposes of select.cc:pipe_data_available, we only need to know whether > InboundQuota - read.ReadDataAvailable is positive.  If WriteQuotaAvailable is > positive, then we're OK, even though its precise value might be too small.  But > if WriteQuotaAvailable == 0, we don't know whether the buffer is actually full > or there's a pending ReadFile.  It's only in this case that we need access to > the read side. > > What if we reverse the roles of the read and write sides of the pipe, so that > the write side is the server and the read side is the client.  We can then try > to use ImpersonateNamedPipeClient to get information about the read side when > WriteQuotaAvailable == 0.  If we succeed, then we can determine whether or not > there's space in the buffer.  If we fail, we simply report, possibly > incorrectly, that there is space.  This is no worse than the current situation > (on the master branch), in which we use OutboundQuota - ReadDataAvailable, which > is always positive. I should add that I know absolutely nothing about ImpersonateNamedPipeClient, so this might be complete nonsense. Ken