From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 2356 invoked by alias); 5 Dec 2015 13:59:49 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 2341 invoked by uid 89); 5 Dec 2015 13:59:48 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.0 required=5.0 tests=AWL,BAYES_00,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-wm0-f43.google.com Received: from mail-wm0-f43.google.com (HELO mail-wm0-f43.google.com) (74.125.82.43) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Sat, 05 Dec 2015 13:59:47 +0000 Received: by wmuu63 with SMTP id u63so92955939wmu.0 for ; Sat, 05 Dec 2015 05:59:44 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.194.175.194 with SMTP id cc2mr23351672wjc.121.1449323984089; Sat, 05 Dec 2015 05:59:44 -0800 (PST) Received: by 10.27.87.195 with HTTP; Sat, 5 Dec 2015 05:59:44 -0800 (PST) In-Reply-To: References: <564E3017.90205@maxrnd.com> <5650379B.4030405@maxrnd.com> <20151121105301.GE2755@calimero.vinschen.de> <5652C402.7040006@maxrnd.com> <24780-1448274431-7444@sneakemail.com> <5653B52B.5000804@maxrnd.com> <20151126093427.GJ2755@calimero.vinschen.de> <5656DDEF.9070603@maxrnd.com> <5662C199.7040906@maxrnd.com> Date: Sat, 05 Dec 2015 13:59:00 -0000 Message-ID: Subject: Re: Cygwin multithreading performance From: Kacper Michajlow To: cygwin@cygwin.com Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes X-SW-Source: 2015-12/txt/msg00054.txt.bz2 2015-12-05 14:07 GMT+01:00 Kacper Michajlow : > 2015-12-05 11:51 GMT+01:00 Mark Geisert : >> Mark Geisert wrote: >>> >>> Corinna Vinschen wrote: >>>> >>>> On Nov 23 16:54, Mark Geisert wrote: >>>>> >>>>> John Hein wrote: >>>>>> >>>>>> Mark Geisert wrote at 23:45 -0800 on Nov 22, 2015: >>>>>> > Corinna Vinschen wrote: >>>>>> > > On Nov 21 01:21, Mark Geisert wrote: >>>>>> > [...] so I wonder if there's >>>>>> > >> some unintentional serialization going on somewhere, but I >>>>>> don't know yet >>>>>> > >> how I could verify that theory. >>>>>> > > >>>>>> > > If I'm allowed to make an educated guess, the big serializer >>>>>> in Cygwin >>>>>> > > are probably the calls to malloc, calloc, realloc, free. We >>>>>> desperately >>>>>> > > need a new malloc implementation better suited to >>>>>> multi-threading. >>> >>> [...] >>>>>> >>>>>> >>>>>> Someone recently mentioned on this list they were working on porting >>>>>> jemalloc. That would be a good choice. >>>>> >>>>> >>>>> Indeed; thanks for the reminder. Somehow I hadn't followed that thread. >>>> >>>> >>>> Indeed^2. Did you look into the locking any further to see if there's >>>> more than one culprit? I guess we've a rather long way to a "lock-less >>>> kernel"... >> >> [...] >>> >>> But that is just groundwork to identifying which locks are suffering the >>> most contention. To identify them at source level I think I'll also >>> need to record the caller's RIP when they are being locked. >> >> >> In the OP's very good testcase the most heavily contended locks, by far, are >> those internal to git's builtin/pack-objects.c. I plan to show actual stats >> after some more cleanup, but I did notice something in that git source file >> that might explain the difference between Cygwin and MinGW when running this >> testcase... >> >> #ifndef NO_PTHREADS >> >> static pthread_mutex_t read_mutex; >> #define read_lock() pthread_mutex_lock(&read_mutex) >> #define read_unlock() pthread_mutex_unlock(&read_mutex) >> >> static pthread_mutex_t cache_mutex; >> #define cache_lock() pthread_mutex_lock(&cache_mutex) >> #define cache_unlock() pthread_mutex_unlock(&cache_mutex) >> >> static pthread_mutex_t progress_mutex; >> #define progress_lock() pthread_mutex_lock(&progress_mutex) >> #define progress_unlock() pthread_mutex_unlock(&progress_mutex) >> >> #else >> >> #define read_lock() (void)0 >> #define read_unlock() (void)0 >> #define cache_lock() (void)0 >> #define cache_unlock() (void)0 >> #define progress_lock() (void)0 >> #define progress_unlock() (void)0 >> >> #endif >> >> Is it possible the MinGW version of git is compiled with NO_PTHREADS >> #defined? If so, it would mean there's no locking being done at all and >> would explain the faster execution and near 100% CPU utilization when >> running under MinGW. > > Nah, there is no threading enabled when there is no pthreads. How > would that work? :D See thread-utils.h > > #ifndef NO_PTHREADS > #include > > extern int online_cpus(void); > extern int init_recursive_mutex(pthread_mutex_t*); > > #else > > #define online_cpus() 1 > > #endif > > > Looks like there is indeed a bug in git code when passing "--threads" > explicitly to "git pack-objects", because they show warning about > threads being unsupported, but doesn't overwrite delta_search_threads > value. I will go to git's ML about it. This is completely not related > to our issue. Obviously I was wrong. There is #define ll_find_deltas(l, s, w, d, p) find_deltas(l, &s, w, d, p) So 'delta_search_threads' value is never used. Still not related to cygwin issue tho ;) -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple