From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by sourceware.org (Postfix) with ESMTPS id 4AEBE3858CDA for ; Wed, 3 Apr 2024 05:21:50 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4AEBE3858CDA Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4AEBE3858CDA Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::52d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712121713; cv=none; b=KtKC0dplkv/SZB/VZfVma/pn4RA9b26NYZgqCW8/VD6Rgl09liGyvjQ6+TDm5OYaBIpdHP6dqYplo80lMl+rOVgt5WDmWPgD7v72EJO3mvS6Kh6zCmd7Agchhml90LRH25bX//xiz9VI7SvTZiOOAxp+unbCyJleZqDVQkRNu1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712121713; c=relaxed/simple; bh=gviTmeKr+gdphgx8x8KlV1ulJRuyv/BM+Jgv8P4dHko=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=KnLGm4OICuLbs+RqmCd85qcp6FeUnePIR1qqn2oKctL3njPIe4hX43125UHyu27Av+64IUo0Lk4fnWQB3hvnalwrDlDxShiPQDX7eWNID8cs1JLSIyUFRhEQzDETEepW0YV5w4whJjsN6aQeeJa6YfpgQLHmCPZp7XrVIwn1ql4= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-ed1-x52d.google.com with SMTP id 4fb4d7f45d1cf-56e030624d1so790340a12.2 for ; Tue, 02 Apr 2024 22:21:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712121708; x=1712726508; darn=cygwin.com; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=pUnY6GhsqNWt0gZLO3pzlxVkV9EPtJcTIQlCVaBFxz4=; b=bmEoaJtjOv7Gm2SevYLkhDNT70j/8dtbypFA6Q9i3+1KBGujZctzjttRKib91DyvH7 IzAKRbAp4i7jP2PTZs7nZ0Gmr07VvLwnlfxyIoFKRvBGK9A5Q0fFRxQFsnrOHVtPK3uE a9TmkXVmjglugnwJESrxmKyDRQVD115cqDBII64cGPa993ag9wrXYK6+hmQ0OELBGA39 jx6K2PR0G9gXGSkvh2Y40NkNkD8PVUQle7OymVsOi3jMRwnmXBT5Y4oIUJQrxERm0oXd 0L/vUjYUJqSr2ribgsvcxIPyfCW0krTqE/OE8oMVEqKLq2LLG+OOCZMKcpqToL9M3SEk y4VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712121708; x=1712726508; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pUnY6GhsqNWt0gZLO3pzlxVkV9EPtJcTIQlCVaBFxz4=; b=Bov7/P9auAoZMBzM6yr6X/ogtpmIodO4nAoiRenZLiP1eGR63p/fxSbsHMr00R+BLm zbAm+nfa+801QhYtwo2Ljpi6Ox4pe0LqCaXIK/4G9vRlqQjsAFeFxPPZrugXprFCn0gG 7hkoSt57mS4IosqRvyua0zntHJeKFDUIbcZcMSTiHU5rS1fRcpIfWmJVrlTCBTAbi1A7 S91g3rN8G/luXs0eoLJw4+Q56Zp2SBfn8XBeQ+hXg4aEYZGb1hwVf524alUMkuVDW33v Uk3GV6l0O97cXWNTdKFkhHFdflFsrRFL3jDDa+wyz7wqBxREQ8UNsZ4mQ6TKRDG4GSOG 2bCw== X-Gm-Message-State: AOJu0YxH/xfhb4K0f5tecafzMB95cdjcGXCVBACE7apqIl0OVGcaaY9z POEuGe7h/gWfgi5VBIoZ5bs82YOdpYxMxkiSCSRuM+n5QuumYLrTFtLZcR004uZpv+fgn/qM5pJ pL1nlYlYU9RhypQi/qEIbRsvFzpzskXU9 X-Google-Smtp-Source: AGHT+IGDUXWTVpS032gZw1JQjOQ1GncrsmNq59ta0zwcHuuMOqK9kTBEJtaoSipKHG0KVuZ/oYkh3RdD547MM9sLHsk= X-Received: by 2002:a50:8d41:0:b0:56c:4db:33f7 with SMTP id t1-20020a508d41000000b0056c04db33f7mr11200334edt.10.1712121708267; Tue, 02 Apr 2024 22:21:48 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Cedric Blancher Date: Wed, 3 Apr 2024 07:21:00 +0200 Message-ID: Subject: Re: Cygwin&Win32 file prefetch, block sizes? To: cygwin@cygwin.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Wed, 3 Apr 2024 at 00:36, Martin Wege via Cygwin wro= te: > > On Tue, Apr 2, 2024 at 3:17=E2=80=AFPM Corinna Vinschen via Cygwin > wrote: > > > > On Apr 2 02:04, Martin Wege via Cygwin wrote: > > > Hello, > > > > > > Is there any document which describes how Cygwin and Win32 file > > > prefetch and readahead work, and which sizes are used (e.g. always > > > read one full page even if only 16 bytes are requested?)? > > > > I'm not aware of any docs, but again, keep in mind that Cygwin is a > > usersapce DLL. We basically do what Windows does for low-level file > > access. > > > > > Quick /usr/bin/stat /etc/profile returns "IO Block: 65536". Does that > > > mean the file's block size is really 64k? Is this info per filesystem= , > > > or hardcoded in Cygwin? > > > > Hardcoded in Cygwin since 2017, based on a discussion in terms of > > file access performance, especially when using stdio.h functions: > > > > https://cygwin.com/cgit/newlib-cygwin/commit/?id=3D7bef7db5ccd9c > > OUCH. > > While I can understand the motivation, FAT32 on multi-GB-devices > having 64k block size, and Win32 API on Win95/98/ME/Win7 being > optimized to that insane block size, it is absolutely WRONG with > today's NTFS and even more so with ReFS. This only works if you stream > files, but as soon as you are doing random read/writes the performance > is terrible due to cache thrashing. That could explain the many > complaints about Cygwin's IO performance. > It can also explain why Cygwin is so slow on SMB filesystems. If it really works on 64k blocks, then this would pull all small files over the wire, even if only a bit gets touched. Thinking more about this, this basically defeats every driver/tcp/ip/ethernet/net/switch optimization like jumbo frames et al. /usr/bin/stat %o (optimum IO block size hint) also returns 65536 on SMB, which is NOT GOOD: This needs to be confirmed, and if this is really true, then it should be fixed for SMB. I need a coffee to think about this... > So, what can be done? I'm not a benchmarking guru, so I'd like to > propose to add a tunable called EXPERIMENTAL_PREFERRED_IO_BLKSIZE to > the CYGWIN env variable (marked as "experimental"), so the > benchmarking guys can do performance testing without recompiling > everything, get perf results for Cygwin 3.6, and decide what to do for > Cygwin 3.7. I would agree, but I would also clamp the minimum at page size (4096 bytes on x86) and 8M (4x 2M hugepage size) to prevent abuse. Does Cygwin partake in the GSOC programm (Google Summer Of Code)? This would IMO be a high priority item. Ced --=20 Cedric Blancher [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur