From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 23631 invoked by alias); 5 Dec 2015 13:07:59 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 23618 invoked by uid 89); 5 Dec 2015 13:07:58 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=0.3 required=5.0 tests=BAYES_50,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-wm0-f42.google.com Received: from mail-wm0-f42.google.com (HELO mail-wm0-f42.google.com) (74.125.82.42) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Sat, 05 Dec 2015 13:07:57 +0000 Received: by wmec201 with SMTP id c201so97722645wme.1 for ; Sat, 05 Dec 2015 05:07:54 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.194.175.194 with SMTP id cc2mr23145866wjc.121.1449320874700; Sat, 05 Dec 2015 05:07:54 -0800 (PST) Received: by 10.27.87.195 with HTTP; Sat, 5 Dec 2015 05:07:54 -0800 (PST) In-Reply-To: <5662C199.7040906@maxrnd.com> References: <564E3017.90205@maxrnd.com> <5650379B.4030405@maxrnd.com> <20151121105301.GE2755@calimero.vinschen.de> <5652C402.7040006@maxrnd.com> <24780-1448274431-7444@sneakemail.com> <5653B52B.5000804@maxrnd.com> <20151126093427.GJ2755@calimero.vinschen.de> <5656DDEF.9070603@maxrnd.com> <5662C199.7040906@maxrnd.com> Date: Sat, 05 Dec 2015 13:07:00 -0000 Message-ID: Subject: Re: Cygwin multithreading performance From: Kacper Michajlow To: cygwin@cygwin.com Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes X-SW-Source: 2015-12/txt/msg00053.txt.bz2 2015-12-05 11:51 GMT+01:00 Mark Geisert : > Mark Geisert wrote: >> >> Corinna Vinschen wrote: >>> >>> On Nov 23 16:54, Mark Geisert wrote: >>>> >>>> John Hein wrote: >>>>> >>>>> Mark Geisert wrote at 23:45 -0800 on Nov 22, 2015: >>>>> > Corinna Vinschen wrote: >>>>> > > On Nov 21 01:21, Mark Geisert wrote: >>>>> > [...] so I wonder if there's >>>>> > >> some unintentional serialization going on somewhere, but I >>>>> don't know yet >>>>> > >> how I could verify that theory. >>>>> > > >>>>> > > If I'm allowed to make an educated guess, the big serializer >>>>> in Cygwin >>>>> > > are probably the calls to malloc, calloc, realloc, free. We >>>>> desperately >>>>> > > need a new malloc implementation better suited to >>>>> multi-threading. >> >> [...] >>>>> >>>>> >>>>> Someone recently mentioned on this list they were working on porting >>>>> jemalloc. That would be a good choice. >>>> >>>> >>>> Indeed; thanks for the reminder. Somehow I hadn't followed that thread. >>> >>> >>> Indeed^2. Did you look into the locking any further to see if there's >>> more than one culprit? I guess we've a rather long way to a "lock-less >>> kernel"... > > [...] >> >> But that is just groundwork to identifying which locks are suffering the >> most contention. To identify them at source level I think I'll also >> need to record the caller's RIP when they are being locked. > > > In the OP's very good testcase the most heavily contended locks, by far, are > those internal to git's builtin/pack-objects.c. I plan to show actual stats > after some more cleanup, but I did notice something in that git source file > that might explain the difference between Cygwin and MinGW when running this > testcase... > > #ifndef NO_PTHREADS > > static pthread_mutex_t read_mutex; > #define read_lock() pthread_mutex_lock(&read_mutex) > #define read_unlock() pthread_mutex_unlock(&read_mutex) > > static pthread_mutex_t cache_mutex; > #define cache_lock() pthread_mutex_lock(&cache_mutex) > #define cache_unlock() pthread_mutex_unlock(&cache_mutex) > > static pthread_mutex_t progress_mutex; > #define progress_lock() pthread_mutex_lock(&progress_mutex) > #define progress_unlock() pthread_mutex_unlock(&progress_mutex) > > #else > > #define read_lock() (void)0 > #define read_unlock() (void)0 > #define cache_lock() (void)0 > #define cache_unlock() (void)0 > #define progress_lock() (void)0 > #define progress_unlock() (void)0 > > #endif > > Is it possible the MinGW version of git is compiled with NO_PTHREADS > #defined? If so, it would mean there's no locking being done at all and > would explain the faster execution and near 100% CPU utilization when > running under MinGW. Nah, there is no threading enabled when there is no pthreads. How would that work? :D See thread-utils.h #ifndef NO_PTHREADS #include extern int online_cpus(void); extern int init_recursive_mutex(pthread_mutex_t*); #else #define online_cpus() 1 #endif Looks like there is indeed a bug in git code when passing "--threads" explicitly to "git pack-objects", because they show warning about threads being unsupported, but doesn't overwrite delta_search_threads value. I will go to git's ML about it. This is completely not related to our issue. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple