Jemalloc under CYGWIN

Jemalloc under CYGWIN

[Yucong Sun]

Hi there,

I'm trying to make jemalloc work with CYGWIN. and I've been meeting
with a mysterious deadlock issue on startup (from CYGWIN's
malloc-wrapper to jemalloc and pthread_mutex_lock get deadlock).

Has anyone else tried this?

Thanks

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Jemalloc under CYGWIN

[Yucong Sun]

Hi,

Throught some frustrating and furious debugging I now understand the
core issue here.  CYGWIN calls malloc provided by jemalloc during
initializations,  which in turn calls pthreads functions, which in
turn uses malloc, which also uses pthreads, causing a deadlock.

Now,  is there anyway to workaround this issue?

On Wed, Oct 7, 2015 at 7:17 PM, Yucong Sun <sunyucong@gmail.com> wrote:
> Hi there,
>
> I'm trying to make jemalloc work with CYGWIN. and I've been meeting
> with a mysterious deadlock issue on startup (from CYGWIN's
> malloc-wrapper to jemalloc and pthread_mutex_lock get deadlock).
>
> Has anyone else tried this?
>
> Thanks

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Jemalloc under CYGWIN

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On Oct 16 23:56, Yucong Sun wrote:
> Hi,
> 
> Throught some frustrating and furious debugging I now understand the
> core issue here.  CYGWIN calls malloc provided by jemalloc during
> initializations,  which in turn calls pthreads functions, which in
> turn uses malloc, which also uses pthreads, causing a deadlock.
> 
> Now,  is there anyway to workaround this issue?

Not quickly, at least.  I looked into thread.cc and pthread.cc and
only found two uses of malloc, one of them should only occur when
pthread_exit'ing the main thread, the other in pthread_getattr_np.
The first shouldn't matter to jemalloc (or, does it?), the second
one I just eliminated.

What exactly is the malloc problem you're seeing?

In the long run it would be very nice if we could get jemalloc working
as the default malloc implementation in Cygwin, replacing dlmalloc.
This would allow to get rid of the slow global locking in malloc_wrapper.cc.

Historical note: At one point I tried to integrate ptmalloc3, but it
always had some unwanted side-effects and fixing them would have been
very complicated at the time.


So, if you're looking into getting jemalloc into Cygwin, feel free
to discuss this further on the cygwin-developers mailing list.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Jemalloc under CYGWIN

[Yucong Sun]

> What exactly is the malloc problem you're seeing?

The specific problem I'm having is that jemalloc's malloc_init() calls
needs to use pthread_mutex_init()  or even pthread_mutex with a
initializer.   Both in-turn uses malloc, triggering this issue.

A quick fix would be somehow make pthread always use system
malloc/free, which shouldn't be that bad.

Another issue I saw is that jemalloc will use readlink()  for
"/etc/jemalloc.conf" during malloc_init(), which on cygwin, this
function uses "new" to do some path manipulating work, which also have
the same issue.  However, we can probably just disable that .

So, the quickest fix would be to make cygwin's thread implementations
always use system malloc.

some small workarounds in jemalloc would be needed afterwards, but it
should be a nice starting point.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Jemalloc under CYGWIN

[Yucong Sun]

> What exactly is the malloc problem you're seeing?

The specific problem I'm having is that jemalloc's malloc_init() calls
needs to use pthread_mutex_init()  or even pthread_mutex with a
initializer.   Both in-turn uses malloc, triggering this issue.

A quick fix would be somehow make pthread always use system
malloc/free, which shouldn't be that bad.

Another issue I saw is that jemalloc will use readlink()  for
"/etc/jemalloc.conf" during malloc_init(), which on cygwin, this
function uses "new" to do some path manipulating work, which also have
the same issue.  However, we can probably just disable that .

So, the quickest fix would be to make cygwin's thread implementations
always use system malloc.

some small workarounds in jemalloc would be needed afterwards, but it
should be a nice starting point.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Jemalloc under CYGWIN

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

On Oct 21 21:49, Yucong Sun wrote:
> > What exactly is the malloc problem you're seeing?
> 
> The specific problem I'm having is that jemalloc's malloc_init() calls
> needs to use pthread_mutex_init()  or even pthread_mutex with a
> initializer.   Both in-turn uses malloc, triggering this issue.
> 
> A quick fix would be somehow make pthread always use system
> malloc/free, which shouldn't be that bad.

What about using a native critical section instead?  It shouldn't be too
tricky to conditionalize this in jemalloc.

> Another issue I saw is that jemalloc will use readlink()  for
> "/etc/jemalloc.conf" during malloc_init(), which on cygwin, this
> function uses "new" to do some path manipulating work, which also have
> the same issue.  However, we can probably just disable that .

Or alternatively the file can be reasd like the other files in /etc
Cygwin needs early access to.  See the NT_readline class and an
example usage in, e.g, mount_info::from_fstab.

> So, the quickest fix would be to make cygwin's thread implementations
> always use system malloc.
> 
> some small workarounds in jemalloc would be needed afterwards, but it
> should be a nice starting point.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Jemalloc under CYGWIN

[Yucong Sun]

On Wed, Oct 21, 2015 at 10:15 PM, Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
> On Oct 21 21:49, Yucong Sun wrote:
>> > What exactly is the malloc problem you're seeing?
>>
>> The specific problem I'm having is that jemalloc's malloc_init() calls
>> needs to use pthread_mutex_init()  or even pthread_mutex with a
>> initializer.   Both in-turn uses malloc, triggering this issue.
>>
>> A quick fix would be somehow make pthread always use system
>> malloc/free, which shouldn't be that bad.
>
> What about using a native critical section instead?  It shouldn't be too
> tricky to conditionalize this in jemalloc.

Possibly, jemalloc already have support of this.  However I wasn't so
sure that this was possible before,

see https://github.com/jemalloc/jemalloc/blob/dev/include/jemalloc/internal/mutex.h#L80

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Jemalloc under CYGWIN

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

On Oct 21 22:43, Yucong Sun wrote:
> On Wed, Oct 21, 2015 at 10:15 PM, Corinna Vinschen
> <corinna-cygwin@cygwin.com> wrote:
> > On Oct 21 21:49, Yucong Sun wrote:
> >> > What exactly is the malloc problem you're seeing?
> >>
> >> The specific problem I'm having is that jemalloc's malloc_init() calls
> >> needs to use pthread_mutex_init()  or even pthread_mutex with a
> >> initializer.   Both in-turn uses malloc, triggering this issue.
> >>
> >> A quick fix would be somehow make pthread always use system
> >> malloc/free, which shouldn't be that bad.
> >
> > What about using a native critical section instead?  It shouldn't be too
> > tricky to conditionalize this in jemalloc.
> 
> Possibly, jemalloc already have support of this.  However I wasn't so
> sure that this was possible before,
> 
> see https://github.com/jemalloc/jemalloc/blob/dev/include/jemalloc/internal/mutex.h#L80

Yes, that looks good.  Please use critical sections for the time being.
We're going to drop XP support end of this year, but up to that point we
can't use SRW locks.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Jemalloc under CYGWIN

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

Hi Yucong Sun,

On Oct 21 17:40, Corinna Vinschen wrote:
> On Oct 21 22:43, Yucong Sun wrote:
> > On Wed, Oct 21, 2015 at 10:15 PM, Corinna Vinschen
> > <corinna-cygwin@cygwin.com> wrote:
> > > On Oct 21 21:49, Yucong Sun wrote:
> > >> > What exactly is the malloc problem you're seeing?
> > >>
> > >> The specific problem I'm having is that jemalloc's malloc_init() calls
> > >> needs to use pthread_mutex_init()  or even pthread_mutex with a
> > >> initializer.   Both in-turn uses malloc, triggering this issue.
> > >>
> > >> A quick fix would be somehow make pthread always use system
> > >> malloc/free, which shouldn't be that bad.
> > >
> > > What about using a native critical section instead?  It shouldn't be too
> > > tricky to conditionalize this in jemalloc.
> > 
> > Possibly, jemalloc already have support of this.  However I wasn't so
> > sure that this was possible before,
> > 
> > see https://github.com/jemalloc/jemalloc/blob/dev/include/jemalloc/internal/mutex.h#L80
> 
> Yes, that looks good.  Please use critical sections for the time being.
> We're going to drop XP support end of this year, but up to that point we
> can't use SRW locks.

did you manage to include jemalloc into Cygwin in the meantime?  It
would be very cool to get a patch to replace dlmalloc with jemalloc :}


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Jemalloc under CYGWIN

[Yucong Sun]

I'm sorry, I got distracted from this issue and hadn't make any progress.

One thing specifically I need to be able to work on this issue is some
information describing how to build/test cygwin lib (rather than using
it). I couldn't find any information on the website, (or I havn't look
deeply enough).

Cheers.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Jemalloc under CYGWIN

[Andrey Repin]

Greetings, Yucong Sun!

> I'm sorry, I got distracted from this issue and hadn't make any progress.

> One thing specifically I need to be able to work on this issue is some
> information describing how to build/test cygwin lib (rather than using
> it). I couldn't find any information on the website, (or I havn't look
> deeply enough).

Probably the latter. :) It isn't obvious, but it is there.
The starting page is http://cygwin.com/contrib.html

I've managed to build it through trial and error, given my near zero knowledge
in bird family languages, it still wasn't all that hard.
For someone familiar with it, it should be much easier.


-- 
With best regards,
Andrey Repin
Saturday, December 5, 2015 19:59:32

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Jemalloc under CYGWIN

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

On Dec  5 22:32, Yucong Sun wrote:
> I'm sorry, I got distracted from this issue and hadn't make any progress.

No worries.

> One thing specifically I need to be able to work on this issue is some
> information describing how to build/test cygwin lib (rather than using
> it). I couldn't find any information on the website, (or I havn't look
> deeply enough).

It's not complicated, really.

1. Install gcc and mingw-gcc.

2. Prepare build dir:

    $ mkdir cygwin
    $ cd cygwin
    $ git clone git://sourceware.org/git/newlib-cygwin.git
    [...]
    $ mkdir build
    $ cd build

3. Build:

    $ export PREFIX=/usr
    $ ../newlib-cygwin/configure \
      --prefix=${PREFIX} \
      --exec-prefix=${PREFIX} \
      --bindir=${PREFIX}/bin \
      --sbindir=${PREFIX}/sbin \
      --libexecdir=${PREFIX}/lib \
      --datadir=${PREFIX}/share \
      --localstatedir=/var \
      --sysconfdir=/etc \
      --infodir=${PREFIX}/share/info \
      --mandir=${PREFIX}/share/man \
      --docdir=${PREFIX}/share/doc/cygwin \
      --htmldir=${PREFIX}/share/doc/cygwin/html
  $ make
  # Or for debugging:
  $ make CFLAGS='-gdwarf-2' CFLAGS_FOR_TARGET='-gdwarf-2'

  Add -j<NUM> as desired.


HTH,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]