From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 114752 invoked by alias); 30 Sep 2018 19:50:42 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 114736 invoked by uid 89); 30 Sep 2018 19:50:41 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-1.6 required=5.0 tests=BAYES_00,KAM_LAZY_DOMAIN_SECURITY,RCVD_IN_DNSWL_LOW autolearn=no version=3.3.2 spammy=doubt, Problem, scanner, Did X-HELO: smtp01.udag.de Received: from smtp01.udag.de (HELO smtp01.udag.de) (62.146.106.17) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Sun, 30 Sep 2018 19:50:39 +0000 Received: from [10.0.0.40] (p5dee49c5.dip0.t-ipconnect.de [93.238.73.197]) by smtp01.udag.de (Postfix) with ESMTPA id CC21783; Sun, 30 Sep 2018 21:50:36 +0200 (CEST) Subject: Re: Filesystem enumeration performance improvement To: cygwin@cygwin.com, marco.mason@gmail.com References: From: =?UTF-8?Q?J=c3=bcrgen_Wagner?= Message-ID: Date: Sun, 30 Sep 2018 19:50:00 -0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-IsSubscribed: yes X-SW-Source: 2018-09/txt/msg00298.txt.bz2 Hi Marco, =C2=A0 as you don't use the Cygwin APIs but go to the Windows APIs directly, any changes to the way stat()/readdir() or related functions in Cygwin operate do not seem to be a plausible reason why your code is running faster. I doubt printf() can be improved to provide such a dramatic speed-up. In my experience, such effects usually have one of two reasons: - There is some caching involved, either in Windows or on the disk level. Run the benchmark tests with empty caches or caching disabled. - Your virus scanner has improved and the operation of determining the status of files no longer excessively causes checks. This is a bit harder to verify or test. Did you compare your program's performance with that of Cygwin's "find"? Did that also show such a dramatic increase in throughput? There is a free and quite fast disk space analyzer called RidNacs (ScanDisk backwards). If the magic you observe is an optimized way of caching, this program should also be affected. Cheers, --J. On 30.09.2018 20:41, Marco Mason wrote: > I recently upgraded from cygwin v2.10 to v2.11.1 and noticed that one of = my > programs got a tremendous speed boost. It's a custom filesystem > enumeration program whose output I feed to frcode to update the > /var/locatedb database. It used to take quite a bit of time (15-20 > minutes?), and now runs in about a minute. Since the program seems to wo= rk > well, just many times faster, I'm rather happy with the changes. > > The reason I'm writing is that I don't see *why* I should have any timing > changes at all! The reason I have my own file enumerator for locatedb is > that the original went through the POSIX layer and was pretty slow, > especially for remote-mounts. As I only needed enough for locate, I wrote > my own enumerator against the Windows API for speed. Since my loop is > essentially just using FindFirstFile/FindNextFile and printf(), I don't > know why file gathering would be any faster. > > So either printf() has gotten remarkably faster, or there are some > interactions between Cygwin and windows in the file enumeration area that > are surprising me. Can someone please clue me in to what might be causing > the speed increases? > > Looking at the git log and mailing list history, my best guess would be > that it's related to the EMail threads "Why does readdir() open files ?" > (Ben Rubson 2018-03-28) and "Why does (stat() ?) open files ?" (Ben Rubson > 2018-04-09). However, I can't seem to pin down which git commits are > relevent to those threads. If anyone can provide a little insight, I'd > really appreciate it. > > --marco > > -- > Problem reports: http://cygwin.com/problems.html > FAQ: http://cygwin.com/faq/ > Documentation: http://cygwin.com/docs.html > Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple > > -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple