From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by sourceware.org (Postfix) with ESMTPS id 974193847725; Wed, 3 Apr 2024 17:43:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 974193847725 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 974193847725 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::62a ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712166213; cv=none; b=TB9p1EIz5V3S9qI3mT6eghFaydOb53fGVC38SSn2uR5c/O8FM9FH3Gmk/ivRj6/9bXkPRkd5f79sxqDKZp8whJjsPRK778W6Hq/RP/oiRDL5fo8gB+qECqaxqmNYglQN8Q9jI+ntD4VpDwpThkpntgJnjjo8lXJ7y1J4CVoEpfo= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712166213; c=relaxed/simple; bh=fVtyHRD9E/KMUidjJys3zVTOXdbc09/Wluw9NDepOI4=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=kVJKDVxUdyJTYMj7owaORjMmQnHs5d4cWXHnaQQz95WgMZKSm9OXvFX/Gh1TOErhgQ9DDzPSkpoqxxk6wvP3DoSNyk9TaMnh7nc7RnflDKHrMEHpBI4VE7fFBcIhfceu+sAN5IeOSFugk2L3N6qb+4o5rZHYDsdML4XpegljXfE= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-a4e79d7d21dso8514866b.1; Wed, 03 Apr 2024 10:43:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712166209; x=1712771009; darn=cygwin.com; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=S2fVG3rugi+7EaJgBoVyEOJ9p6EdMG67Q0mMv9L/cqA=; b=Y56fuvoOAQVRRiG5wYVJ4tnfdLQAiWzTpEF5ASvOuV5Uq3NL3lB33wzilWUIs74D0/ kidYbSCcSF8k85Ci3g+Nh+oN8OmS2sB0X8WPON2aXbYdI1ZucfX8giRh0HWMAQpE+D2t WOw6OMmypP9BjWAnzKk0+7gZazCd6uVgOM9UXc8zG0PpGjNF6E6YgEHqGIweCvawsc3G laNOWJvgoBBHUcjjR1BM11xViHXpnF0ZTxNrOMc5bdPWl0PeErKNZo2fE5SK4j3/qiOr KYh5liTe6aM7TzqDQJZDaJy1WLNPFpPjGpeNsLOXs/oQzVutXYvKdtEgqvgCqGV6LEQo Fz7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712166209; x=1712771009; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S2fVG3rugi+7EaJgBoVyEOJ9p6EdMG67Q0mMv9L/cqA=; b=DBizfvXsavqwtaOUN4fCWCBGMquhBMz/QZmrqBvJ/AYWtkYXZYzIL2iYjLQHCtDucL 0dsLjcu80Od6gzfVuj6sY38gdklQ5CxlqlGePANgoZot9OjkaK/Q/rSz+hTkNM1WGOcg jCqbyhZRdqzfHinY4n1/NNw3YT8fivJEaLodjn++YcT+mHHLgqLgUle5iHebf/29k9W3 xkVOPXW+3S4VfTVQ/XWYrlXAEJf1OX+d7tJQSmyS/NZVGVKLy24soTiJ857MMJ9OMZLP P/EAvOEyq2q3yINsp5MnYwFCA7YT2BCbwgyQPYmldD/LzLHv4DCy0NgAmY72Z9y3J7kd YgoA== X-Gm-Message-State: AOJu0YzmpTAJjjL4t0WSm6udrz+rmeio97hnORfzZyQXHJqH/RR94u77 Jp5Kpa2nmuXjERPXO5BAO0RXkS8P/RtkNL+7YfVF7mcGqYHiSnFtetYiDswItubcOdlyqdxTbl6 H3OYy82ZPJro/bijLtjjFuclu6G0+i9d/I6M= X-Google-Smtp-Source: AGHT+IGvPSeei9j3h/VtHTO6OF3Nj0xarGIrlI8TBG1a8YKOl3fnIxi8zB3OPODMBGtqDPLMhCyzJI4eqiMOUUHJG4Y= X-Received: by 2002:a50:9e81:0:b0:56e:ee2:7897 with SMTP id a1-20020a509e81000000b0056e0ee27897mr127657edf.5.1712166208666; Wed, 03 Apr 2024 10:43:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Cedric Blancher Date: Wed, 3 Apr 2024 19:42:51 +0200 Message-ID: Subject: Re: Cygwin&Win32 file prefetch, block sizes? To: cygwin@cygwin.com Cc: Corinna Vinschen Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Wed, 3 Apr 2024 at 10:15, Corinna Vinschen via Cygwin wrote: > > On Apr 3 00:35, Martin Wege via Cygwin wrote: > > On Tue, Apr 2, 2024 at 3:17=E2=80=AFPM Corinna Vinschen via Cygwin > > wrote: > > > > > > On Apr 2 02:04, Martin Wege via Cygwin wrote: > > > > Hello, > > > > > > > > Is there any document which describes how Cygwin and Win32 file > > > > prefetch and readahead work, and which sizes are used (e.g. always > > > > read one full page even if only 16 bytes are requested?)? > > > > > > I'm not aware of any docs, but again, keep in mind that Cygwin is a > > > usersapce DLL. We basically do what Windows does for low-level file > > > access. > > > > > > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does th= at > > > > mean the file's block size is really 64k? Is this info per filesyst= em, > > > > or hardcoded in Cygwin? > > > > > > Hardcoded in Cygwin since 2017, based on a discussion in terms of > > > file access performance, especially when using stdio.h functions: > > > > > > https://cygwin.com/cgit/newlib-cygwin/commit/?id=3D7bef7db5ccd9c > > > > OUCH. > > > > While I can understand the motivation, FAT32 on multi-GB-devices > > having 64k block size, and Win32 API on Win95/98/ME/Win7 being > > optimized to that insane block size, it is absolutely WRONG with > > today's NTFS and even more so with ReFS. This only works if you stream > > files, but as soon as you are doing random read/writes the performance > > is terrible due to cache thrashing. That could explain the many > > complaints about Cygwin's IO performance. > > The above patch *only* sets stat::st_blksize to 64K. Nothing else > happens! Yes, but applications use that information, and then make wrong choices. > > This usually means that stdio.h functions use this size for their buffer > and readahead. It doesn't affect direct calls to read(2)/write(2) and > fread(3)/fwrite(3) at all! > > > So, what can be done? I'm not a benchmarking guru, so I'd like to > > propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to > > No. > > We have two ways to handle this *iff* there's really a reason to > handle this. > > - Either we just lower PREFERRED_IO_BLKSIZE to 4K or 8K, but that's > kind of bad in terms of pipes, the clipboard, etc. I think the env variable was strictly meant for benchmarking **ONLY**, so someone with good benchmarking expertise can grab the Cygwin 3.6 alpha binaries and does benchmarking with them. > > - So we keep PREFERRED_IO_BLKSIZE at 64K but don't use it for disk > files. Rather, we read this info from the filesystem: > > https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/ns= -ntddk-_file_fs_sector_size_information @Corinna Vinschen Could you please do me a favour and check whether SMB supports FILE_FS_SECTOR_SIZE_INFORMATION? > > If the filesystem is local and SSINFO_FLAGS_NO_SEEK_PENALTY is set, we > could stick to 64K. That still means that buffers will be very large, and associated memcpy(), read(), write() are large too. Also harmful is that such buffers might be allocated with alloca(), because the original authors didn't anticipate 64k buffer sizes... > > Otherwise the PhysicalBytesPerSectorForPerformance member might be > helpful I guess. Needs checking, of course. > > If this isn't any good, we can still fallback to > FILE_FS_FULL_SIZE_INFORMATION as in fhandler_base::fstatvfs_by_handle, > https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/fhandler/disk_= file.cc#n661 You mean when FILE_FS_SECTOR_SIZE_INFORMATION is not supported you use FILE_FS_FULL_SIZE_INFORMATION instead? Ced --=20 Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur