Re: malloc crash

[Corinna Vinschen <corinna-cygwin@cygwin.com>]

On Oct 26 10:32, Ken Brown wrote:
> On 10/26/2021 5:24 AM, Corinna Vinschen wrote:
> > On Oct 25 18:02, Ken Brown wrote:
> > > Or does the fifo_reader thread call a malloc function before the main thread
> > > has called malloc_init()?  This would presumably cause __malloc_lock() to
> > > fail, but there's no error check.
> > 
> > That sounds more likely.  In theory this shouldn't have much influence,
> > though.  First of all, all fixup calls are running in a single thread,
> > so there's no serialization required(*), and the malloc_init call
> > doesn't set up the malloc arena, it just initializes the muto and checks
> > for user space provided malloc calls, which is not a problem in this
> > scenario.
> > 
> > (*) unless multiple threads are started during fixup and some of these
> >      threads mallocate memory again...
> > 
> > Ken, is there a chance to tweak the fifo code to stop creating
> > threads from inside fixup, and to defer the thread start to the first
> > call in the process actually relying on the thread being started?
> 
> I can't think of any way to do that.  The thread is listening for various
> events that cause it to take action, so it has to always be running.  But I
> can probably tweak the code so that the thread doesn't have to call malloc
> early on.
> 
> It might take a while to get this right, and the bug has existed ever since
> I overhauled the fifo code.  So I don't think you have to hold up releasing
> 3.3.0 while I work on this.

Try the below patch instead, per Takashi's testing and subsequent discussion.

From 9e53881e81bc6d2d072a0d625a9eac8ffc7cc698 Mon Sep 17 00:00:00 2001
From: Corinna Vinschen <corinna@vinschen.de>
Date: Tue, 26 Oct 2021 17:53:08 +0200
Subject: [PATCH] Cygwin: split malloc_init

Per https://cygwin.com/pipermail/cygwin-developers/2021-October/012429.html,
we may encounter a crash when starting multiple threads during process
startup (here: fhandler_fifo::fixup_after_{fork,exec}) which in turn
allocate memory via malloc.

The problem is concurrent usage of malloc before the malloc muto has
been initialized.

To fix this issue, split malloc_init into malloc_init_0, called from
dll_crt0_0, and malloc_init_1, called from dll_crt_0_1.  malloc_init_0
just initializes the muto, malloc_init_1 checks for user space provided
malloc.

Signed-off-by: Corinna Vinschen <corinna@vinschen.de>
---
 winsup/cygwin/dcrt0.cc          | 4 +++-
 winsup/cygwin/heap.cc           | 1 -
 winsup/cygwin/heap.h            | 3 ++-
 winsup/cygwin/malloc_wrapper.cc | 6 +++++-
 4 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/winsup/cygwin/dcrt0.cc b/winsup/cygwin/dcrt0.cc
index 6f4723bb059d..7c460274fb86 100644
--- a/winsup/cygwin/dcrt0.cc
+++ b/winsup/cygwin/dcrt0.cc
@@ -769,6 +769,8 @@ dll_crt0_0 ()
   NtOpenProcessToken (NtCurrentProcess (), MAXIMUM_ALLOWED, &hProcToken);
   set_cygwin_privileges (hProcToken);
 
+  malloc_init_0 ();
+
   device::init ();
   do_global_ctors (&__CTOR_LIST__, 1);
   cygthread::init ();
@@ -857,7 +859,7 @@ dll_crt0_1 (void *)
      on a functioning malloc and it's possible that the user's program may
      have overridden malloc.  We only know about that at this stage,
      unfortunately. */
-  malloc_init ();
+  malloc_init_1 ();
   user_shared->initialize ();
 
 #ifdef CYGHEAP_DEBUG
diff --git a/winsup/cygwin/heap.cc b/winsup/cygwin/heap.cc
index b839c8cd48ee..f27f81bc4b59 100644
--- a/winsup/cygwin/heap.cc
+++ b/winsup/cygwin/heap.cc
@@ -230,7 +230,6 @@ user_heap_info::init ()
   debug_printf ("heap base %p, heap top %p, heap size %ly (%lu)",
 		base, top, chunk, chunk);
   page_const--;
-  // malloc_init ();
 }
 
 #define pround(n) (((size_t)(n) + page_const) & ~page_const)
diff --git a/winsup/cygwin/heap.h b/winsup/cygwin/heap.h
index 565758e4872c..42099051f619 100644
--- a/winsup/cygwin/heap.h
+++ b/winsup/cygwin/heap.h
@@ -10,7 +10,8 @@ details. */
 
 /* Heap management. */
 void heap_init ();
-void malloc_init ();
+void malloc_init_0 ();
+void malloc_init_1 ();
 
 #define inheap(s) \
   (cygheap->user_heap.ptr && s \
diff --git a/winsup/cygwin/malloc_wrapper.cc b/winsup/cygwin/malloc_wrapper.cc
index 3b245800abec..85c411a3e258 100644
--- a/winsup/cygwin/malloc_wrapper.cc
+++ b/winsup/cygwin/malloc_wrapper.cc
@@ -272,10 +272,14 @@ strdup (const char *s)
 muto NO_COPY mallock;
 
 void
-malloc_init ()
+malloc_init_0 ()
 {
   mallock.init ("mallock");
+}
 
+void
+malloc_init_1 ()
+{
   /* Check if malloc is provided by application. If so, redirect all
      calls to malloc/free/realloc to application provided. This may
      happen if some other dll calls cygwin's malloc, but main code provides
-- 
2.31.1