From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Larry Hall (RFK Partners, Inc)" To: Jonathan Kamens , cygwin@cygwin.com Subject: Re: Optimizing away "ReadFile" calls when Make calls stat() Date: Tue, 13 Feb 2001 10:56:00 -0000 Message-id: <4.3.1.2.20010213134821.019a7130@pop.ma.ultranet.com> References: <20010213183634.1435.qmail@lizard.curl.com> X-SW-Source: 2001-02/msg00672.html At 01:36 PM 2/13/2001, Jonathan Kamens wrote: >We use Cygwin to develop a large product (running a build and the test >suites takes about two hours on a very fast machine); our builds are >driven by GNU Make. We compile and test the same product under Linux. >We've found that builds under Cygwin run several times slower than >builds under Linux, even on machines of comparable speed, RAM, etc. >The slowness is seriously impacting the productivity of our developers >who work on Windows, so we're searching for any way we can to speed up >Cygwin builds. > >We've found that one of the biggest culprits in slowing down the >Cygwin builds is Make. The problem is that every time Make does >stat() to find out the modification time on a dependency to determine >whether or not its dependents need to be rebuilt, Cygwin calls >ReadFile on the file twice -- once to determine whether it's a >symbolic link, and a second time to determine whether it should appear >to be executable according to stat(). We have thousands of >dependencies in our Makefiles, and many of those dependencies >frequently live on network drives, so these calls to ReadFile >seriously slow things down. > >We don't use symbolic links anywhere in our source tree or build >tree, and Make doesn't really care whether a file is executable when >deciding whether it is newer than one of its dependents, so both of >these calls to ReadFile are totally unnecessary to us. As an >experiment, I added code to the Cygwin DLL to allow these ReadFile >calls to be temporarily disabled, and then I compiled a modified >version of Make which disables the ReadFile calls before calling >stat() and then turns them back on. > >To measure the effect of these changes, I ran "make all" in a build >tree tha was already completely built, so that I would be timing only >the work Make does to check dependencies, rather than timing actual >build work. With the unmodified Make, "make all" takes around six >minutes; with the modified Make, it takes around three. We consider >this a significant improvement. (However, note that on Linux, "make >all" when nothing needs to be done takes only 17 seconds, so clearly >there's still a lot of room for improvement under Cygwin.) > >I'm wondering if the maintainers of Cygwin would be willing to >consider incorporating these changes, if I submit them, into the >Cygwin DLL and the Cygwin version of Make. I'm thinking that the DLL >changes would actually need to be split into two flags -- one to say, >"Don't call ReadFile to find out whether a file is executable, because >I don't care about that," and the other to say, "Don't call ReadFile >to find out if a file is a symbolic link, because I know I'm not using >any symbolic links." Then, GNU Make on Cygwin could always set the >first flag, and it could set the second flag if the user specified >"--nosymlinks" or something like that. > >I realize that this is a bit gross. However, (a) surely it isn't much >more gross than storing symbolic links inside files and reading files >to determine whether they should look executable :-), and (b) it >really does give a drastic performance improvement for the small price >of not using symbolic links in your source or build tree. > >Please comment. I know this would only address half the problem but I wonder if it would make sense to cache the results of ReadFile() so that separate checks for symbolic links and executables would result in only 1 ReadFile() call. This seems like a nice general optimization which wouldn't be so "gross"... Larry Hall lhall@rfk.com RFK Partners, Inc. http://www.rfk.com 118 Washington Street (508) 893-9779 - RFK Office Holliston, MA 01746 (508) 893-9889 - FAX -- Want to unsubscribe from this list? Check out: http://cygwin.com/ml/#unsubscribe-simple