Error: child_copy: cygheap read copy failed.

Error: child_copy: cygheap read copy failed.

[Takashi Yano]

Hi Corinna,

I am now working on porting ffmpeg for cygwin, and noticed that
the following error sometimes happens.

$ while true; do ffplay 2>&1 |grep cygheap; done
      0 [main] ffplay (10172) child_copy: cygheap read copy failed, 0x0..0x800028FB8, done 0, windows pid 10172, Win32 error 299
      0 [main] ffplay (13052) child_copy: cygheap read copy failed, 0x0..0x800028FB8, done 0, windows pid 13052, Win32 error 299
      0 [main] ffplay (9092) child_copy: cygheap read copy failed, 0x0..0x800028FB8, done 0, windows pid 9092, Win32 error 299
      0 [main] ffplay (8988) child_copy: cygheap read copy failed, 0x0..0x800028FB8, done 0, windows pid 8988, Win32 error 299
      0 [main] ffplay (11800) child_copy: cygheap read copy failed, 0x0..0x800028FB8, done 0, windows pid 11800, Win32 error 299
      0 [main] ffplay (6088) child_copy: cygheap read copy failed, 0x0..0x800028FB8, done 0, windows pid 6088, Win32 error 299

This seems to happen after the commit:

commit 2f9b8ff00cce0d0ceeb1091b6b9aecae914ddb89
Author: Corinna Vinschen <corinna@vinschen.de>
Date:   Wed Oct 26 21:16:35 2022 +0200

    Cygwin: decouple cygheap from Cygwin DLL

    One reason that ASLR is tricky is the fact that the cygheap
    is placed at the end of the DLL and especially that it's expected
    to be growable.  To support ASLR, this construct must go.

    Define dedicated cygheap memory region and reserve entire region.
    Commit 3 Megs, as was the default size of the cygheap before.

    Fix linker script accordingly, drop a now useless version check
    in get_cygwin_startup_info().

I looked into this problem a bit and found that VirtualAlloc()
for MEM_RESERVE of cygheap area in cygheap_fixup_in_child()
failed with ERROR_INVALID_ADDRESS. It seems that the address
range is already occupied for some reason.

I also noticed that the following patch seems to resolve the issue.

diff --git a/winsup/cygwin/local_includes/memory_layout.h b/winsup/cygwin/local_includes/memory_layout.h
index a3a0cae70..67164cfaf 100644
--- a/winsup/cygwin/local_includes/memory_layout.h
+++ b/winsup/cygwin/local_includes/memory_layout.h
@@ -44,15 +44,15 @@ details. */
 
 /* That's where the cygheap is located. CYGHEAP_STORAGE_INITIAL defines the
    end of the initially committed heap area. */
-#define CYGHEAP_STORAGE_LOW		0x800000000UL
-#define CYGHEAP_STORAGE_INITIAL		0x800300000UL
-#define CYGHEAP_STORAGE_HIGH		0xa00000000UL
+#define CYGHEAP_STORAGE_LOW		0x70000000000UL
+#define CYGHEAP_STORAGE_INITIAL		0x70000300000UL
+#define CYGHEAP_STORAGE_HIGH		0x70200000000UL
 
 /* This is where the user heap starts.  There's no defined end address.
    The user heap pontentially grows into the mmap arena.  However,
    the user heap grows upwards and the mmap arena grows downwards,
    so there's not much chance to meet unluckily. */
-#define USERHEAP_START			0xa00000000UL
+#define USERHEAP_START			0x00800000000UL
 
 /* The memory region used for memory maps.  Mmaps grow downwards.
    Set the lowest address to leave ~32 Gigs for heap.

Could you please have a look?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Error: child_copy: cygheap read copy failed.

[Corinna Vinschen]

On Jan 20 12:56, Takashi Yano via Cygwin wrote:
> Hi Corinna,
> 
> I am now working on porting ffmpeg for cygwin, and noticed that
> the following error sometimes happens.
> 
> $ while true; do ffplay 2>&1 |grep cygheap; done
>       0 [main] ffplay (10172) child_copy: cygheap read copy failed, 0x0..0x800028FB8, done 0, windows pid 10172, Win32 error 299
> 
> This seems to happen after the commit:
> 
> commit 2f9b8ff00cce0d0ceeb1091b6b9aecae914ddb89
> Author: Corinna Vinschen <corinna@vinschen.de>
> Date:   Wed Oct 26 21:16:35 2022 +0200
> 
>     Cygwin: decouple cygheap from Cygwin DLL
> [...]
> I looked into this problem a bit and found that VirtualAlloc()
> for MEM_RESERVE of cygheap area in cygheap_fixup_in_child()
> failed with ERROR_INVALID_ADDRESS. It seems that the address
> range is already occupied for some reason.

You should try to find out what that is.
> 
> I also noticed that the following patch seems to resolve the issue.
> 
> diff --git a/winsup/cygwin/local_includes/memory_layout.h b/winsup/cygwin/local_includes/memory_layout.h
> index a3a0cae70..67164cfaf 100644
> --- a/winsup/cygwin/local_includes/memory_layout.h
> +++ b/winsup/cygwin/local_includes/memory_layout.h
> @@ -44,15 +44,15 @@ details. */
>  
>  /* That's where the cygheap is located. CYGHEAP_STORAGE_INITIAL defines the
>     end of the initially committed heap area. */
> -#define CYGHEAP_STORAGE_LOW		0x800000000UL
> -#define CYGHEAP_STORAGE_INITIAL		0x800300000UL
> -#define CYGHEAP_STORAGE_HIGH		0xa00000000UL
> +#define CYGHEAP_STORAGE_LOW		0x70000000000UL
> +#define CYGHEAP_STORAGE_INITIAL		0x70000300000UL
> +#define CYGHEAP_STORAGE_HIGH		0x70200000000UL
>  
>  /* This is where the user heap starts.  There's no defined end address.
>     The user heap pontentially grows into the mmap arena.  However,
>     the user heap grows upwards and the mmap arena grows downwards,
>     so there's not much chance to meet unluckily. */
> -#define USERHEAP_START			0xa00000000UL
> +#define USERHEAP_START			0x800000000UL

That doesn't make sense.  So we now have the user heap in place of the
cygheap, and that works?  So why should anything occuping the 0x800000000
- 0xa00000000 area if the cygheap needs it, suddenly not occupy it when
the user heap needs it?

Of course, this could be a result of the user heap not being reserved in
full size like the cygheap is.  The fact that it works with your patch
above points to an incidental success, with the user heap keeping most
of the area from 0x800000000 - 0xa00000000 free.  It would then fail
only later, if the application tries to raise the user heap beyond a
certain size.

Right now this looks like a very special case.  I don't know what I
should do from my side.  Please add debug output and/or use vmmap
from sysinternals to find out what's actually occupying the area
from 0x800000000 - 0xa00000000 and where exactly.

Also, your ffplay executable isn't high-entropy-VA enabled, by any
chance?


Corinna

Re: Error: child_copy: cygheap read copy failed.

[Takashi Yano]

On Fri, 20 Jan 2023 12:30:29 +0100
Corinna Vinschen wrote:
> Also, your ffplay executable isn't high-entropy-VA enabled, by any
> chance?

I'm sorry, but the high-entropy-va is ebabled.

$ peflags -v /usr/bin/ff{mpeg,probe,play}
/usr/bin/ffmpeg: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
/usr/bin/ffprobe: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
/usr/bin/ffplay: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])

peflags -e0 /usr/bin/ff{mpeg,probe,play} solves the issue.
Thansk!

Shoud I remove dynamicbase flag as well?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Error: child_copy: cygheap read copy failed.

[Corinna Vinschen]

On Jan 20 21:18, Takashi Yano via Cygwin wrote:
> On Fri, 20 Jan 2023 12:30:29 +0100
> Corinna Vinschen wrote:
> > Also, your ffplay executable isn't high-entropy-VA enabled, by any
> > chance?
> 
> I'm sorry, but the high-entropy-va is ebabled.
> 
> $ peflags -v /usr/bin/ff{mpeg,probe,play}
> /usr/bin/ffmpeg: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> /usr/bin/ffprobe: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> /usr/bin/ffplay: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> 
> peflags -e0 /usr/bin/ff{mpeg,probe,play} solves the issue.
> Thansk!

I'm glad to read that, actually!

> Shoud I remove dynamicbase flag as well?

I did a lot of local testing with ASLR locally, and it's fine for DLLs,
but not so much for the application executable itself.

You can keep dynamicbase for testing, but currently it's not advised
to do so, and you should definitely remove it for distro executables.


Thanks,
Corinna

Re: Error: child_copy: cygheap read copy failed.

[Corinna Vinschen]

On Jan 20 21:18, Takashi Yano via Cygwin wrote:
> On Fri, 20 Jan 2023 12:30:29 +0100
> Corinna Vinschen wrote:
> > Also, your ffplay executable isn't high-entropy-VA enabled, by any
> > chance?
> 
> I'm sorry, but the high-entropy-va is ebabled.
> 
> $ peflags -v /usr/bin/ff{mpeg,probe,play}
> /usr/bin/ffmpeg: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> /usr/bin/ffprobe: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> /usr/bin/ffplay: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])

Btw., how did this happen?  The native Cygwin compiler/linker combo
usually doesn't set the dynamicbase and high-entropy-VA flags by
default...


Corinna

Re: Error: child_copy: cygheap read copy failed.

[Takashi Yano]

On Fri, 20 Jan 2023 13:26:20 +0100
Corinna Vinschen wrote:
> On Jan 20 21:18, Takashi Yano via Cygwin wrote:
> > On Fri, 20 Jan 2023 12:30:29 +0100
> > Corinna Vinschen wrote:
> > > Also, your ffplay executable isn't high-entropy-VA enabled, by any
> > > chance?
> > 
> > I'm sorry, but the high-entropy-va is ebabled.
> > 
> > $ peflags -v /usr/bin/ff{mpeg,probe,play}
> > /usr/bin/ffmpeg: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> > /usr/bin/ffprobe: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> > /usr/bin/ffplay: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> 
> Btw., how did this happen?  The native Cygwin compiler/linker combo
> usually doesn't set the dynamicbase and high-entropy-VA flags by
> default...

Perhaps this is because configure of ffmpeg explicitly
has the following lines:

check_ldflags -Wl,--nxcompat,--dynamicbase
check_ldflags -Wl,--high-entropy-va # binutils 2.25

Removing these lines makes ff{mpeg,probe,play} disable
both high-entropy-va and dynamicbase flags.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Error: child_copy: cygheap read copy failed.

[Corinna Vinschen]

On Jan 20 22:30, Takashi Yano via Cygwin wrote:
> On Fri, 20 Jan 2023 13:26:20 +0100
> Corinna Vinschen wrote:
> > On Jan 20 21:18, Takashi Yano via Cygwin wrote:
> > > On Fri, 20 Jan 2023 12:30:29 +0100
> > > Corinna Vinschen wrote:
> > > > Also, your ffplay executable isn't high-entropy-VA enabled, by any
> > > > chance?
> > > 
> > > I'm sorry, but the high-entropy-va is ebabled.
> > > 
> > > $ peflags -v /usr/bin/ff{mpeg,probe,play}
> > > /usr/bin/ffmpeg: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> > > /usr/bin/ffprobe: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> > > /usr/bin/ffplay: coff(0x022e[+executable_image,+line_nums_stripped,+local_syms_stripped,+bigaddr,+sepdbg]) pe(0x8160[+high-entropy-va,+dynamicbase,+nxcompat,+tsaware])
> > 
> > Btw., how did this happen?  The native Cygwin compiler/linker combo
> > usually doesn't set the dynamicbase and high-entropy-VA flags by
> > default...
> 
> Perhaps this is because configure of ffmpeg explicitly
> has the following lines:
> 
> check_ldflags -Wl,--nxcompat,--dynamicbase
> check_ldflags -Wl,--high-entropy-va # binutils 2.25

Oh, ok. The configurey should only do that on native Windows builds.

> Removing these lines makes ff{mpeg,probe,play} disable
> both high-entropy-va and dynamicbase flags.

Great!


Thanks,
Corinna