Extend faq.using to discuss fork failures

Extend faq.using to discuss fork failures

[Ryan Johnson]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

Hi all,

Based on the feedback on cygwin-dev, I've put together a revised pair of 
faq.using entries: one listing briefly the symptoms of fork failures and 
what to do about it, and the other giving some details about why fork 
fails (sometimes in spite of everything we do to compensate).

         * faq-using.xml (faq.using.fixing-fork-failures): Add.
         (faq.using.why-fork-fails): Add.

Thoughts?
Ryan

[-- Attachment #2: faq-fork-patch.txt --]
[-- Type: text/plain, Size: 5353 bytes --]

Index: winsup/doc/faq-using.xml
===================================================================
RCS file: /cvs/src/src/winsup/doc/faq-using.xml,v
retrieving revision 1.35
diff -u -r1.35 faq-using.xml
--- winsup/doc/faq-using.xml	4 Aug 2011 18:25:41 -0000	1.35
+++ winsup/doc/faq-using.xml	26 Aug 2011 01:58:44 -0000
@@ -1199,3 +1199,92 @@
 </listitem>
 </itemizedlist></para>
 </answer></qandaentry>
+<qandaentry id='faq.using.fixing-fork-failures'>
+  <question><para>Calls to <literal>fork</literal> fail a lot. How can
+  I fix the problem?</para></question>
+  <answer>
+
+  <para>Unix-like applications make extensive use of
+  <literal>fork</literal>, a function which spawns an exact copy of
+  the running process. Notable fork-using applications include bash
+  (and bash scripts), emacs, gcc, make, perl, python, and
+  ruby. Unfortunately, the Windows ecosystem is quite hostile to a
+  reliable fork implementation, leading to error messages such as:</para>
+  <para><itemizedlist>
+    <listitem>unable to remap <emphasis>$dll</emphasis> to same address as parent</listitem>
+    <listitem>couldn't allocate heap </listitem>
+    <listitem>died waiting for dll loading </listitem>
+    <listitem>child -1 - died waiting for longjmp before initialization</listitem>
+    <listitem>STATUS_ACCESS_VIOLATION </listitem>
+    <listitem>resource temporarily unavailable </listitem>
+  </itemizedlist></para>
+  <para>If you find that frequent fork failures interfere with normal
+  use of cygwin, please try the following: </para>
+  <para><itemizedlist>
+    <listitem>Restart whatever process is trying (and failing) to use
+    <literal>fork</literal>. Sometimes Windows sets up a process
+    environment that is even more hostile to fork than usual.</listitem>
+    <listitem>Ensure that you have eliminated (not just disabled) all
+    software on the BLODA (see <ulink
+    url="http://cygwin.com/faq/faq.using.html#faq.using.bloda"
+    />)</listitem>
+    <listitem>Install the 'rebase' package, read its README in
+    <literal>/usr/share/doc/Cygwin</literal>, and follow the
+    instructions there to run 'rebaseall'.</listitem>
+    </itemizedlist></para>
+  <para>Please note that installing new packages or updating existing
+  ones often undoes the effects of rebaseall and cause fork failures
+  to reappear. If so, just run rebaseall again.
+  </para></answer>
+</qandaentry>
+<qandaentry id='faq.using.why-fork-fails'>
+  <question><para>Why does <literal>fork</literal> fail so much,
+  anyway? (or: Why does <literal>fork</literal> still fail even though
+  I ran rebaseall?)</para></question>
+  <answer>
+  <para>The semantics of <literal>fork</literal> require that a forked
+  child process have <emphasis>exactly</emphasis> the same address
+  space layout as its parent. However, Windows provides no native
+  support for cloning address space between processes and several
+  features actively undermine a reliable <literal>fork</literal>
+  implementation. Three issues are especially prevalent:</para>
+  <para><itemizedlist>
+    <listitem>DLL base address collisions. Unlike *nix shared
+    libraries, which use "position-independent code", Windows shared
+    libraries assume a fixed base address. Whenever the hard-wired
+    address ranges of two DLLs collide (which occurs quite often), the
+    Windows loader must "rebase" one of them to a different
+    address. However, it does not resolve collisions consistently, and
+    may rebase a different dll and/or move it to a different address
+    every time. Cygwin can usually compensate for this effect when it
+    involves libraries opened dynamically, but collisions among
+    statically-linked dlls (dependencies known at compile time) are
+    resolved before <literal>cygwin1.dll</literal> initializes and
+    cannot be fixed afterward. This problem can only be solved by
+    removing the base address conflicts which cause the problem,
+    usually using the <literal>rebaseall</literal> package.</listitem>
+
+    <listitem>Address space layout randomization (ASLR). Starting with
+    Vista, Windows implements ASLR, which means that thread stacks,
+    heap, memory-mapped files, and statically-linked dlls are placed
+    at different (random) locations in each process. This behavior
+    interferes with a proper <literal>fork</literal>, and if an
+    unmovable object (process heap or system dll) ends up at the wrong
+    location, Cygwin can do nothing to compensate (though it will
+    retry a few times automatically). In a 64-bit system, marking
+    executables as large address-ware and rebasing dlls to high
+    addresses has been reported to help, as ASLR affects only the
+    lower 2GB of address space.</listitem>
+
+    <listitem>DLL injection by BLODA. Badly-behaved applications which
+    inject dlls into other processes often manage to clobber important
+    sections of the child's address space, leading to base address
+    collisions which rebasing cannot fix. The only way to resolve this
+    problem is to remove (usually uninstall) the offending
+    app.</listitem></itemizedlist></para>
+    <para>In summary, current Windows implementations make it
+    impossible to implement a perfectly reliable fork, and occasional
+    fork failures are inevitable. PTC.
+    </para>
+  </answer>
+</qandaentry>

Re: Extend faq.using to discuss fork failures

[Ryan Johnson]

Ooops. Mailer hiccup. Please ignore this one.

On 25/08/2011 10:08 PM, Ryan Johnson wrote:
> Hi all,
>
> Based on the feedback on cygwin-dev, I've put together a revised pair 
> of faq.using entries: one listing briefly the symptoms of fork 
> failures and what to do about it, and the other giving some details 
> about why fork fails (sometimes in spite of everything we do to 
> compensate).
>
>         * faq-using.xml (faq.using.fixing-fork-failures): Add.
>         (faq.using.why-fork-fails): Add.
>
> Thoughts?
> Ryan

Re: Extend faq.using to discuss fork failures

[Eric Blake]

On 08/25/2011 08:08 PM, Ryan Johnson wrote:

> Based on the feedback on cygwin-dev, I've put together a revised pair of faq.using entries: one listing briefly the symptoms of fork failures and what to do about it, and the other giving some details about why fork fails (sometimes in spite of everything we do to compensate).

> +
> +<listitem>DLL injection by BLODA. Badly-behaved applications which
> +    inject dlls into other processes often manage to clobber important
> +    sections of the child's address space, leading to base address
> +    collisions which rebasing cannot fix.

I would suggest adding an <a href=> tag around BLODA, to point users to 
the list of known bad software.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Re: Extend faq.using to discuss fork failures

[Jon TURNEY]

On 04/11/2011 16:44, Christopher Faylor wrote:
> On Fri, Nov 04, 2011 at 12:22:13PM -0400, Christopher Faylor wrote:
>> On Fri, Nov 04, 2011 at 01:34:09PM +0000, Jon TURNEY wrote:
>>> On 03/11/2011 21:05, Christopher Faylor wrote:
>>>> I would still prefer eschewing actively negative words like "hostile" and just
>>>> neutrally stating that Windows does not use a fork/exec model and does not offer
>>>> any easy way to implement fork.
>>>
>>> Hmm, yes, I'll fix that.
>>
>> Thanks.
>>
>>>> I'd also like to see specific errors mentioned so that when people are searching for
>>>> a solution to the problem they will be able to find it in the FAQ.
>>>
>>> Is there something wrong with the itemized list which follows that sentence?
>>
>> No, sorry.  I'm email challenged at the moment so I missed it.
>
> Btw, since this is such a glaring omission from the FAQ I think you
> should make the edits that Corinna and I suggested and just check it
> in.  We can tweak it as needed when people express confusion.

Ok, checked in with the suggested edits.

Re: Extend faq.using to discuss fork failures

[Christopher Faylor]

On Fri, Nov 04, 2011 at 12:22:13PM -0400, Christopher Faylor wrote:
>On Fri, Nov 04, 2011 at 01:34:09PM +0000, Jon TURNEY wrote:
>>On 03/11/2011 21:05, Christopher Faylor wrote:
>>> I would still prefer eschewing actively negative words like "hostile" and just
>>> neutrally stating that Windows does not use a fork/exec model and does not offer
>>> any easy way to implement fork.
>>
>>Hmm, yes, I'll fix that.
>
>Thanks.
>
>>> I'd also like to see specific errors mentioned so that when people are searching for
>>> a solution to the problem they will be able to find it in the FAQ.
>>
>>Is there something wrong with the itemized list which follows that sentence?
>
>No, sorry.  I'm email challenged at the moment so I missed it.

Btw, since this is such a glaring omission from the FAQ I think you
should make the edits that Corinna and I suggested and just check it
in.  We can tweak it as needed when people express confusion.

Thanks to you and Ryan for doing this.  We really needed it.

cgf

Re: Extend faq.using to discuss fork failures

[Christopher Faylor]

On Fri, Nov 04, 2011 at 01:34:09PM +0000, Jon TURNEY wrote:
>On 03/11/2011 21:05, Christopher Faylor wrote:
>> I would still prefer eschewing actively negative words like "hostile" and just
>> neutrally stating that Windows does not use a fork/exec model and does not offer
>> any easy way to implement fork.
>
>Hmm, yes, I'll fix that.

Thanks.

>> I'd also like to see specific errors mentioned so that when people are searching for
>> a solution to the problem they will be able to find it in the FAQ.
>
>Is there something wrong with the itemized list which follows that sentence?

No, sorry.  I'm email challenged at the moment so I missed it.

cgf

Re: Extend faq.using to discuss fork failures

[Jon TURNEY]

On 03/11/2011 17:17, Corinna Vinschen wrote:
> Thanks for doing that.  I looks good to me, with just one exception.
>
>> +<listitem>Address space layout randomization (ASLR). Starting with
>> +Vista, Windows implements ASLR, which means that thread stacks,
>> +heap, memory-mapped files, and statically-linked dlls are placed
>> +at different (random) locations in each process. This behaviour
>> +interferes with a proper<literal>fork</literal>, and if an
>> +unmovable object (process heap or system dll) ends up at the wrong
>> +location, Cygwin can do nothing to compensate (though it will
>> +retry a few times automatically). In a 64-bit system, marking
>> +executables as large address-ware and rebasing dlls to high
>> +addresses has been reported to help, as ASLR affects only the
>> +lower 2GB of address space.</listitem>
>
> Starting with "In a 64-bit system" it's getting a bit weird:
>
> - Starting with 4.5.3, gcc marks executables as large address aware
>    automatically, so this is going to be a lesser problem over time.  Is
>    it worth to mention this at all?  I suppose so, but the user should be
>    pointed to peflags to tests for this property first for the given
>    reason.
>
> - Starting with Cygwin 1.7.10, the high address area will be used for
>    the application heap on 64 bit systems and large address aware
>    executables.  Mmaps are located there, too.  This in turn leaves more
>    room for DLLs in the normal 2 Gigs memory area.  Therefore I would not
>    like to suggest rebasing DLLs into the high address area at all.  This
>    should only be done by people who know what they are doing.  Usually
>    there should be enough space in the lower 2 Gigs, especially when heap
>    and mmaps are out of the way, and given that the more recent rebaseall
>    will not create an arbitrary 64K hole between DLLs anymore when
>    rebasing.

I think it would be simplest just to drop the sentence starting "In a 64-bit 
system".

Re: Extend faq.using to discuss fork failures

[Jon TURNEY]

On 03/11/2011 21:05, Christopher Faylor wrote:
> I would still prefer eschewing actively negative words like "hostile" and just
> neutrally stating that Windows does not use a fork/exec model and does not offer
> any easy way to implement fork.

Hmm, yes, I'll fix that.

> I'd also like to see specific errors mentioned so that when people are searching for
> a solution to the problem they will be able to find it in the FAQ.

Is there something wrong with the itemized list which follows that sentence?

Re: Extend faq.using to discuss fork failures

[Christopher Faylor]

On Thu, Nov 03, 2011 at 04:35:25PM +0000, Jon TURNEY wrote:
>On 30/08/2011 14:41, Ryan Johnson wrote:
>> That sounds reasonable, though I suspect we'd want want to keep the concluding
>> bits in the FAQ as well. Unfortunately, summertime free time has come to an
>> end so I don't know when I'll get to this next. Perhaps a good compromise for
>> now would be for you to post only the first FAQ question? That would at least
>> cut traffic to the cygwin ML a bit.
>
>I've updated Ryan's patch to hopefully address the comments made, polished the 
>language a bit in places, and split it into a patch for the FAQ which just 
>says how to fix problems and a patch for the UG which contains the technical 
>details.

>Index: doc/faq-using.xml
>===================================================================
>RCS file: /cvs/src/src/winsup/doc/faq-using.xml,v
>retrieving revision 1.35
>diff -u -p -r1.35 faq-using.xml
>--- doc/faq-using.xml	4 Aug 2011 18:25:41 -0000	1.35
>+++ doc/faq-using.xml	3 Nov 2011 16:26:56 -0000
>@@ -1099,7 +1099,7 @@ it.</para>
> IPv6 stack, see the <ulink url="http://www.microsoft.com/technet/network/ipv6/ipv6faq.mspx">Microsoft TechNet IPv6 FAQ article</ulink>
> </para></answer></qandaentry>
> 
>-<qandaentry id="faq.using.bloda">
>+<qandaentry id="faq.using.bloda" xreflabel="BLODA">
> <question><para>What applications have been found to interfere with Cygwin?</para></question>
> <answer>
> 
>@@ -1199,3 +1199,38 @@ such as virtual memory paging and file c
> </listitem>
> </itemizedlist></para>
> </answer></qandaentry>
>+
>+<qandaentry id='faq.using.fixing-fork-failures'>
>+  <question><para>How do I fix <literal>fork()</literal> failures?</para></question>
>+  <answer>
>+  <para>Unfortunately, Windows can be quite hostile to a

I would still prefer eschewing actively negative words like "hostile" and just
neutrally stating that Windows does not use a fork/exec model and does not offer
any easy way to implement fork.

I'd also like to see specific errors mentioned so that when people are searching for
a solution to the problem they will be able to find it in the FAQ.

cgf

Re: Extend faq.using to discuss fork failures

[Corinna Vinschen]

Hi Jon,

On Nov  3 16:35, Jon TURNEY wrote:
> On 30/08/2011 14:41, Ryan Johnson wrote:
> >That sounds reasonable, though I suspect we'd want want to keep the concluding
> >bits in the FAQ as well. Unfortunately, summertime free time has come to an
> >end so I don't know when I'll get to this next. Perhaps a good compromise for
> >now would be for you to post only the first FAQ question? That would at least
> >cut traffic to the cygwin ML a bit.
> 
> I've updated Ryan's patch to hopefully address the comments made,
> polished the language a bit in places, and split it into a patch for
> the FAQ which just says how to fix problems and a patch for the UG
> which contains the technical details.

Thanks for doing that.  I looks good to me, with just one exception.

> +<listitem>Address space layout randomization (ASLR). Starting with
> +Vista, Windows implements ASLR, which means that thread stacks,
> +heap, memory-mapped files, and statically-linked dlls are placed
> +at different (random) locations in each process. This behaviour
> +interferes with a proper <literal>fork</literal>, and if an
> +unmovable object (process heap or system dll) ends up at the wrong
> +location, Cygwin can do nothing to compensate (though it will
> +retry a few times automatically). In a 64-bit system, marking
> +executables as large address-ware and rebasing dlls to high
> +addresses has been reported to help, as ASLR affects only the
> +lower 2GB of address space.</listitem>

Starting with "In a 64-bit system" it's getting a bit weird:

- Starting with 4.5.3, gcc marks executables as large address aware
  automatically, so this is going to be a lesser problem over time.  Is
  it worth to mention this at all?  I suppose so, but the user should be
  pointed to peflags to tests for this property first for the given
  reason.

- Starting with Cygwin 1.7.10, the high address area will be used for
  the application heap on 64 bit systems and large address aware
  executables.  Mmaps are located there, too.  This in turn leaves more
  room for DLLs in the normal 2 Gigs memory area.  Therefore I would not
  like to suggest rebasing DLLs into the high address area at all.  This
  should only be done by people who know what they are doing.  Usually
  there should be enough space in the lower 2 Gigs, especially when heap
  and mmaps are out of the way, and given that the more recent rebaseall
  will not create an arbitrary 64K hole between DLLs anymore when
  rebasing.

With these changes, feel free to check in the patch.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Re: Extend faq.using to discuss fork failures

[Jon TURNEY]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

On 30/08/2011 14:41, Ryan Johnson wrote:
> That sounds reasonable, though I suspect we'd want want to keep the concluding
> bits in the FAQ as well. Unfortunately, summertime free time has come to an
> end so I don't know when I'll get to this next. Perhaps a good compromise for
> now would be for you to post only the first FAQ question? That would at least
> cut traffic to the cygwin ML a bit.

I've updated Ryan's patch to hopefully address the comments made, polished the 
language a bit in places, and split it into a patch for the FAQ which just 
says how to fix problems and a patch for the UG which contains the technical 
details.

[-- Attachment #2: faq-fork-failures.patch --]
[-- Type: text/plain, Size: 2673 bytes --]

Index: doc/faq-using.xml
===================================================================
RCS file: /cvs/src/src/winsup/doc/faq-using.xml,v
retrieving revision 1.35
diff -u -p -r1.35 faq-using.xml
--- doc/faq-using.xml	4 Aug 2011 18:25:41 -0000	1.35
+++ doc/faq-using.xml	3 Nov 2011 16:26:56 -0000
@@ -1099,7 +1099,7 @@ it.</para>
 IPv6 stack, see the <ulink url="http://www.microsoft.com/technet/network/ipv6/ipv6faq.mspx">Microsoft TechNet IPv6 FAQ article</ulink>
 </para></answer></qandaentry>
 
-<qandaentry id="faq.using.bloda">
+<qandaentry id="faq.using.bloda" xreflabel="BLODA">
 <question><para>What applications have been found to interfere with Cygwin?</para></question>
 <answer>
 
@@ -1199,3 +1199,38 @@ such as virtual memory paging and file c
 </listitem>
 </itemizedlist></para>
 </answer></qandaentry>
+
+<qandaentry id='faq.using.fixing-fork-failures'>
+  <question><para>How do I fix <literal>fork()</literal> failures?</para></question>
+  <answer>
+  <para>Unfortunately, Windows can be quite hostile to a
+  reliable fork() implementation, leading to error messages such as:</para>
+  <para><itemizedlist>
+    <listitem>unable to remap <emphasis>somedll</emphasis> to same address as parent</listitem>
+    <listitem>couldn't allocate heap</listitem>
+    <listitem>died waiting for dll loading</listitem>
+    <listitem>child -1 - died waiting for longjmp before initialization</listitem>
+    <listitem>STATUS_ACCESS_VIOLATION</listitem>
+    <listitem>resource temporarily unavailable</listitem>
+  </itemizedlist></para>
+  <para>Potential solutions for the above errors:</para>
+  <para><itemizedlist>
+    <listitem>Restart whatever process is trying (and failing) to use
+    <literal>fork()</literal>. Sometimes Windows sets up a process
+    environment that is even more hostile to fork() than usual.</listitem>
+    <listitem>Ensure that you have eliminated (not just disabled) all
+    software on the <xref linkend="faq.using.bloda"/>.
+    </listitem>
+    <listitem>Read the 'rebase' package README in
+    <literal>/usr/share/doc/rebase/</literal>, and follow the
+    instructions there to run 'rebaseall'.</listitem>
+    </itemizedlist></para>
+  <para>Please note that installing new packages or updating existing
+  ones undoes the effects of rebaseall and often causes fork() failures
+  to reappear. If so, just run rebaseall again.
+  </para>
+  <para>See the <ulink url="http://cygwin.com/cygwin-ug-net/highlights.html#ov-hi-process">
+  process creation</ulink> section of the User's Guide for the technical reasons it is so
+  difficult to make <literal>fork()</literal> work reliably.</para>
+</answer>
+</qandaentry>

[-- Attachment #3: ug-fork-problems.patch --]
[-- Type: text/plain, Size: 3262 bytes --]

Index: doc/overview2.sgml
===================================================================
RCS file: /cvs/src/src/winsup/doc/overview2.sgml,v
retrieving revision 1.20
diff -u -p -r1.20 overview2.sgml
--- doc/overview2.sgml	18 Sep 2010 15:58:46 -0000	1.20
+++ doc/overview2.sgml	3 Nov 2011 16:27:36 -0000
@@ -346,6 +346,60 @@ cases, stubs of each of these Win32 proc
 their exec'd Cygwin process to exit.</para>
 </sect2>
 
+<sect3 id='ov-hi-process-problems'>
+<title>Problems with process creation</title>
+
+<para>The semantics of <literal>fork</literal> require that a forked
+child process have <emphasis>exactly</emphasis> the same address
+space layout as its parent. However, Windows provides no native
+support for cloning address space between processes and several
+features actively undermine a reliable <literal>fork</literal>
+implementation. Three issues are especially prevalent:</para>
+
+<para><itemizedlist>
+<listitem>DLL base address collisions. Unlike *nix shared
+libraries, which use "position-independent code", Windows shared
+libraries assume a fixed base address. Whenever the hard-wired
+address ranges of two DLLs collide (which occurs quite often), the
+Windows loader must "rebase" one of them to a different
+address. However, it may not resolve collisions consistently, and
+may rebase a different dll and/or move it to a different address
+every time. Cygwin can usually compensate for this effect when it
+involves libraries opened dynamically, but collisions among
+statically-linked dlls (dependencies known at compile time) are
+resolved before <literal>cygwin1.dll</literal> initializes and
+cannot be fixed afterward. This problem can only be solved by
+removing the base address conflicts which cause the problem,
+usually using the <literal>rebaseall</literal> tool.</listitem>
+
+<listitem>Address space layout randomization (ASLR). Starting with
+Vista, Windows implements ASLR, which means that thread stacks,
+heap, memory-mapped files, and statically-linked dlls are placed
+at different (random) locations in each process. This behaviour
+interferes with a proper <literal>fork</literal>, and if an
+unmovable object (process heap or system dll) ends up at the wrong
+location, Cygwin can do nothing to compensate (though it will
+retry a few times automatically). In a 64-bit system, marking
+executables as large address-ware and rebasing dlls to high
+addresses has been reported to help, as ASLR affects only the
+lower 2GB of address space.</listitem>
+
+<listitem>DLL injection by
+<ulink url="http://cygwin.com/faq/faq.using.html#faq.using.bloda">
+BLODA</ulink>. Badly-behaved applications which
+inject dlls into other processes often manage to clobber important
+sections of the child's address space, leading to base address
+collisions which rebasing cannot fix. The only way to resolve this
+problem is to remove (usually uninstall) the offending
+app.</listitem></itemizedlist></para>
+
+<para>In summary, current Windows implementations make it
+impossible to implement a perfectly reliable fork, and occasional
+fork failures are inevitable.
+</para>
+
+</sect3>
+
 <sect2 id="ov-hi-signals"><title>Signals</title>
 <para>When
 a Cygwin process starts, the library starts a secondary thread for

Re: Extend faq.using to discuss fork failures

[Christopher Faylor]

On Tue, Aug 30, 2011 at 11:00:20AM +0200, Corinna Vinschen wrote:
>Hi Ryan,
>
>Thanks for the FAQ entry.  I had a look now, finally.  Two nits:
>
>On Aug 25 22:08, Ryan Johnson wrote:
>> Index: winsup/doc/faq-using.xml
>> ===================================================================
>> RCS file: /cvs/src/src/winsup/doc/faq-using.xml,v
>> retrieving revision 1.35
>> diff -u -r1.35 faq-using.xml
>> --- winsup/doc/faq-using.xml	4 Aug 2011 18:25:41 -0000	1.35
>> +++ winsup/doc/faq-using.xml	26 Aug 2011 01:58:44 -0000
>> @@ -1199,3 +1199,92 @@
>>  </listitem>
>>  </itemizedlist></para>
>>  </answer></qandaentry>
>> +<qandaentry id='faq.using.fixing-fork-failures'>
>> +  <question><para>Calls to <literal>fork</literal> fail a lot. How can
>> +  I fix the problem?</para></question>

I'd prefer a simple "How do I fix fork failures?"

>> +  <answer>
>> +
>> +  <para>Unix-like applications make extensive use of
>> +  <literal>fork</literal>, a function which spawns an exact copy of
>> +  the running process. Notable fork-using applications include bash
>> +  (and bash scripts), emacs, gcc, make, perl, python, and
>> +  ruby. Unfortunately, the Windows ecosystem is quite hostile to a
>> +  reliable fork implementation, leading to error messages such as:</para>

I'd prefer just going straight to the solution without worrying about
the theory or the ecosystem.  If more discussion is needed, as Corinna
says, point elsewhere.

So, a listing of potential errors followed by a succint explanation of
how to fix them is what I'd like to see.  You can move the "Windows ecosystem"
comments to the next section.

>> +  <para><itemizedlist>
>> +    <listitem>unable to remap <emphasis>$dll</emphasis> to same address as parent</listitem>
>> +    <listitem>couldn't allocate heap </listitem>
>> +    <listitem>died waiting for dll loading </listitem>
>> +    <listitem>child -1 - died waiting for longjmp before initialization</listitem>
>> +    <listitem>STATUS_ACCESS_VIOLATION </listitem>
>> +    <listitem>resource temporarily unavailable </listitem>
>> +  </itemizedlist></para>
>> +  <para>If you find that frequent fork failures interfere with normal
>> +  use of cygwin, please try the following: </para>

Please just use something like "Potential solutions for the above errors:"

>> +  <para><itemizedlist>
>> +    <listitem>Restart whatever process is trying (and failing) to use
>> +    <literal>fork</literal>. Sometimes Windows sets up a process
>> +    environment that is even more hostile to fork than usual.</listitem>
>> +    <listitem>Ensure that you have eliminated (not just disabled) all
>> +    software on the BLODA (see <ulink
>> +    url="http://cygwin.com/faq/faq.using.html#faq.using.bloda"
>> +    />)</listitem>
>> +    <listitem>Install the 'rebase' package, read its README in
>> +    <literal>/usr/share/doc/Cygwin</literal>, and follow the
>> +    instructions there to run 'rebaseall'.</listitem>
>
>The rebase package is always installed since it's part of the Base
>category.  This entry should be rephrased accordingly.
>
>> +    </itemizedlist></para>
>> +  <para>Please note that installing new packages or updating existing
>> +  ones often undoes the effects of rebaseall and cause fork failures
>> +  to reappear. If so, just run rebaseall again.
>> +  </para></answer>
>> +</qandaentry>
>> +<qandaentry id='faq.using.why-fork-fails'>
>> +  <question><para>Why does <literal>fork</literal> fail so much,
>> +  anyway? (or: Why does <literal>fork</literal> still fail even though
>> +  I ran rebaseall?)</para></question>
>> +  <answer>
>> +  <para>The semantics of <literal>fork</literal> require that a forked
>> +  child process have <emphasis>exactly</emphasis> the same address
>> +  space layout as its parent. However, Windows provides no native
>> +  support for cloning address space between processes and several
>> +  features actively undermine a reliable <literal>fork</literal>
>> +  implementation.
>
>Everything else which follows from here is a good description of the
>inner workings, but that shouldn't go into the FAQ.  What about creating
>a link to the user's guide "Process Creation" section here.  The technical
>details could then go into the "Process Creation" section.

I agree.  I should have thought of that when I mentioned making two
sections.

cgf

Re: Extend faq.using to discuss fork failures

[Ryan Johnson]

Hi Corinna,

That sounds reasonable, though I suspect we'd want want to keep the 
concluding bits in the FAQ as well. Unfortunately, summertime free time 
has come to an end so I don't know when I'll get to this next. Perhaps a 
good compromise for now would be for you to post only the first FAQ 
question? That would at least cut traffic to the cygwin ML a bit.

Ryan

On 30/08/2011 2:00 AM, Corinna Vinschen wrote:
> Hi Ryan,
>
> Thanks for the FAQ entry.  I had a look now, finally.  Two nits:
>
> On Aug 25 22:08, Ryan Johnson wrote:
>> Index: winsup/doc/faq-using.xml
>> ===================================================================
>> RCS file: /cvs/src/src/winsup/doc/faq-using.xml,v
>> retrieving revision 1.35
>> diff -u -r1.35 faq-using.xml
>> --- winsup/doc/faq-using.xml	4 Aug 2011 18:25:41 -0000	1.35
>> +++ winsup/doc/faq-using.xml	26 Aug 2011 01:58:44 -0000
>> @@ -1199,3 +1199,92 @@
>>   </listitem>
>>   </itemizedlist></para>
>>   </answer></qandaentry>
>> +<qandaentry id='faq.using.fixing-fork-failures'>
>> +<question><para>Calls to<literal>fork</literal>  fail a lot. How can
>> +  I fix the problem?</para></question>
>> +<answer>
>> +
>> +<para>Unix-like applications make extensive use of
>> +<literal>fork</literal>, a function which spawns an exact copy of
>> +  the running process. Notable fork-using applications include bash
>> +  (and bash scripts), emacs, gcc, make, perl, python, and
>> +  ruby. Unfortunately, the Windows ecosystem is quite hostile to a
>> +  reliable fork implementation, leading to error messages such as:</para>
>> +<para><itemizedlist>
>> +<listitem>unable to remap<emphasis>$dll</emphasis>  to same address as parent</listitem>
>> +<listitem>couldn't allocate heap</listitem>
>> +<listitem>died waiting for dll loading</listitem>
>> +<listitem>child -1 - died waiting for longjmp before initialization</listitem>
>> +<listitem>STATUS_ACCESS_VIOLATION</listitem>
>> +<listitem>resource temporarily unavailable</listitem>
>> +</itemizedlist></para>
>> +<para>If you find that frequent fork failures interfere with normal
>> +  use of cygwin, please try the following:</para>
>> +<para><itemizedlist>
>> +<listitem>Restart whatever process is trying (and failing) to use
>> +<literal>fork</literal>. Sometimes Windows sets up a process
>> +    environment that is even more hostile to fork than usual.</listitem>
>> +<listitem>Ensure that you have eliminated (not just disabled) all
>> +    software on the BLODA (see<ulink
>> +    url="http://cygwin.com/faq/faq.using.html#faq.using.bloda"
>> +    />)</listitem>
>> +<listitem>Install the 'rebase' package, read its README in
>> +<literal>/usr/share/doc/Cygwin</literal>, and follow the
>> +    instructions there to run 'rebaseall'.</listitem>
> The rebase package is always installed since it's part of the Base
> category.  This entry should be rephrased accordingly.
>
>> +</itemizedlist></para>
>> +<para>Please note that installing new packages or updating existing
>> +  ones often undoes the effects of rebaseall and cause fork failures
>> +  to reappear. If so, just run rebaseall again.
>> +</para></answer>
>> +</qandaentry>
>> +<qandaentry id='faq.using.why-fork-fails'>
>> +<question><para>Why does<literal>fork</literal>  fail so much,
>> +  anyway? (or: Why does<literal>fork</literal>  still fail even though
>> +  I ran rebaseall?)</para></question>
>> +<answer>
>> +<para>The semantics of<literal>fork</literal>  require that a forked
>> +  child process have<emphasis>exactly</emphasis>  the same address
>> +  space layout as its parent. However, Windows provides no native
>> +  support for cloning address space between processes and several
>> +  features actively undermine a reliable<literal>fork</literal>
>> +  implementation.
> Everything else which follows from here is a good description of the
> inner workings, but that shouldn't go into the FAQ.  What about creating
> a link to the user's guide "Process Creation" section here.  The technical
> details could then go into the "Process Creation" section.
>
>> +  Three issues are especially prevalent:</para>
>> +<para><itemizedlist>
>> +<listitem>DLL base address collisions. Unlike *nix shared
>> +    libraries, which use "position-independent code", Windows shared
>> +    libraries assume a fixed base address. Whenever the hard-wired
>> +    address ranges of two DLLs collide (which occurs quite often), the
>> +    Windows loader must "rebase" one of them to a different
>> +    address. However, it does not resolve collisions consistently, and
>> +    may rebase a different dll and/or move it to a different address
>> +    every time. Cygwin can usually compensate for this effect when it
>> +    involves libraries opened dynamically, but collisions among
>> +    statically-linked dlls (dependencies known at compile time) are
>> +    resolved before<literal>cygwin1.dll</literal>  initializes and
>> +    cannot be fixed afterward. This problem can only be solved by
>> +    removing the base address conflicts which cause the problem,
>> +    usually using the<literal>rebaseall</literal>  package.</listitem>
>> +
>> +<listitem>Address space layout randomization (ASLR). Starting with
>> +    Vista, Windows implements ASLR, which means that thread stacks,
>> +    heap, memory-mapped files, and statically-linked dlls are placed
>> +    at different (random) locations in each process. This behavior
>> +    interferes with a proper<literal>fork</literal>, and if an
>> +    unmovable object (process heap or system dll) ends up at the wrong
>> +    location, Cygwin can do nothing to compensate (though it will
>> +    retry a few times automatically). In a 64-bit system, marking
>> +    executables as large address-ware and rebasing dlls to high
>> +    addresses has been reported to help, as ASLR affects only the
>> +    lower 2GB of address space.</listitem>
>> +
>> +<listitem>DLL injection by BLODA. Badly-behaved applications which
>> +    inject dlls into other processes often manage to clobber important
>> +    sections of the child's address space, leading to base address
>> +    collisions which rebasing cannot fix. The only way to resolve this
>> +    problem is to remove (usually uninstall) the offending
>> +    app.</listitem></itemizedlist></para>
>> +<para>In summary, current Windows implementations make it
>> +    impossible to implement a perfectly reliable fork, and occasional
>> +    fork failures are inevitable. PTC.
>> +</para>
>> +</answer>
>> +</qandaentry>
>
> Corinna
>

Re: Extend faq.using to discuss fork failures

[Corinna Vinschen]

Hi Ryan,

Thanks for the FAQ entry.  I had a look now, finally.  Two nits:

On Aug 25 22:08, Ryan Johnson wrote:
> Index: winsup/doc/faq-using.xml
> ===================================================================
> RCS file: /cvs/src/src/winsup/doc/faq-using.xml,v
> retrieving revision 1.35
> diff -u -r1.35 faq-using.xml
> --- winsup/doc/faq-using.xml	4 Aug 2011 18:25:41 -0000	1.35
> +++ winsup/doc/faq-using.xml	26 Aug 2011 01:58:44 -0000
> @@ -1199,3 +1199,92 @@
>  </listitem>
>  </itemizedlist></para>
>  </answer></qandaentry>
> +<qandaentry id='faq.using.fixing-fork-failures'>
> +  <question><para>Calls to <literal>fork</literal> fail a lot. How can
> +  I fix the problem?</para></question>
> +  <answer>
> +
> +  <para>Unix-like applications make extensive use of
> +  <literal>fork</literal>, a function which spawns an exact copy of
> +  the running process. Notable fork-using applications include bash
> +  (and bash scripts), emacs, gcc, make, perl, python, and
> +  ruby. Unfortunately, the Windows ecosystem is quite hostile to a
> +  reliable fork implementation, leading to error messages such as:</para>
> +  <para><itemizedlist>
> +    <listitem>unable to remap <emphasis>$dll</emphasis> to same address as parent</listitem>
> +    <listitem>couldn't allocate heap </listitem>
> +    <listitem>died waiting for dll loading </listitem>
> +    <listitem>child -1 - died waiting for longjmp before initialization</listitem>
> +    <listitem>STATUS_ACCESS_VIOLATION </listitem>
> +    <listitem>resource temporarily unavailable </listitem>
> +  </itemizedlist></para>
> +  <para>If you find that frequent fork failures interfere with normal
> +  use of cygwin, please try the following: </para>
> +  <para><itemizedlist>
> +    <listitem>Restart whatever process is trying (and failing) to use
> +    <literal>fork</literal>. Sometimes Windows sets up a process
> +    environment that is even more hostile to fork than usual.</listitem>
> +    <listitem>Ensure that you have eliminated (not just disabled) all
> +    software on the BLODA (see <ulink
> +    url="http://cygwin.com/faq/faq.using.html#faq.using.bloda"
> +    />)</listitem>
> +    <listitem>Install the 'rebase' package, read its README in
> +    <literal>/usr/share/doc/Cygwin</literal>, and follow the
> +    instructions there to run 'rebaseall'.</listitem>

The rebase package is always installed since it's part of the Base
category.  This entry should be rephrased accordingly.

> +    </itemizedlist></para>
> +  <para>Please note that installing new packages or updating existing
> +  ones often undoes the effects of rebaseall and cause fork failures
> +  to reappear. If so, just run rebaseall again.
> +  </para></answer>
> +</qandaentry>
> +<qandaentry id='faq.using.why-fork-fails'>
> +  <question><para>Why does <literal>fork</literal> fail so much,
> +  anyway? (or: Why does <literal>fork</literal> still fail even though
> +  I ran rebaseall?)</para></question>
> +  <answer>
> +  <para>The semantics of <literal>fork</literal> require that a forked
> +  child process have <emphasis>exactly</emphasis> the same address
> +  space layout as its parent. However, Windows provides no native
> +  support for cloning address space between processes and several
> +  features actively undermine a reliable <literal>fork</literal>
> +  implementation.

Everything else which follows from here is a good description of the
inner workings, but that shouldn't go into the FAQ.  What about creating
a link to the user's guide "Process Creation" section here.  The technical
details could then go into the "Process Creation" section.

> +  Three issues are especially prevalent:</para>
> +  <para><itemizedlist>
> +    <listitem>DLL base address collisions. Unlike *nix shared
> +    libraries, which use "position-independent code", Windows shared
> +    libraries assume a fixed base address. Whenever the hard-wired
> +    address ranges of two DLLs collide (which occurs quite often), the
> +    Windows loader must "rebase" one of them to a different
> +    address. However, it does not resolve collisions consistently, and
> +    may rebase a different dll and/or move it to a different address
> +    every time. Cygwin can usually compensate for this effect when it
> +    involves libraries opened dynamically, but collisions among
> +    statically-linked dlls (dependencies known at compile time) are
> +    resolved before <literal>cygwin1.dll</literal> initializes and
> +    cannot be fixed afterward. This problem can only be solved by
> +    removing the base address conflicts which cause the problem,
> +    usually using the <literal>rebaseall</literal> package.</listitem>
> +
> +    <listitem>Address space layout randomization (ASLR). Starting with
> +    Vista, Windows implements ASLR, which means that thread stacks,
> +    heap, memory-mapped files, and statically-linked dlls are placed
> +    at different (random) locations in each process. This behavior
> +    interferes with a proper <literal>fork</literal>, and if an
> +    unmovable object (process heap or system dll) ends up at the wrong
> +    location, Cygwin can do nothing to compensate (though it will
> +    retry a few times automatically). In a 64-bit system, marking
> +    executables as large address-ware and rebasing dlls to high
> +    addresses has been reported to help, as ASLR affects only the
> +    lower 2GB of address space.</listitem>
> +
> +    <listitem>DLL injection by BLODA. Badly-behaved applications which
> +    inject dlls into other processes often manage to clobber important
> +    sections of the child's address space, leading to base address
> +    collisions which rebasing cannot fix. The only way to resolve this
> +    problem is to remove (usually uninstall) the offending
> +    app.</listitem></itemizedlist></para>
> +    <para>In summary, current Windows implementations make it
> +    impossible to implement a perfectly reliable fork, and occasional
> +    fork failures are inevitable. PTC.
> +    </para>
> +  </answer>
> +</qandaentry>


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Extend faq.using to discuss fork failures

[Ryan Johnson]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

Hi all,

Based on the feedback from cygwin-dev, I've put together a revised pair 
of faq.using entries: one listing briefly the symptoms of fork failures 
and what to do about it, and the other giving some details about why 
fork fails (sometimes in spite of everything we do to compensate).

         * faq-using.xml (faq.using.fixing-fork-failures): Add.
         (faq.using.why-fork-fails): Add.

Thoughts?
Ryan

[-- Attachment #2: faq-fork-patch.txt --]
[-- Type: text/plain, Size: 5353 bytes --]

Index: winsup/doc/faq-using.xml
===================================================================
RCS file: /cvs/src/src/winsup/doc/faq-using.xml,v
retrieving revision 1.35
diff -u -r1.35 faq-using.xml
--- winsup/doc/faq-using.xml	4 Aug 2011 18:25:41 -0000	1.35
+++ winsup/doc/faq-using.xml	26 Aug 2011 01:58:44 -0000
@@ -1199,3 +1199,92 @@
 </listitem>
 </itemizedlist></para>
 </answer></qandaentry>
+<qandaentry id='faq.using.fixing-fork-failures'>
+  <question><para>Calls to <literal>fork</literal> fail a lot. How can
+  I fix the problem?</para></question>
+  <answer>
+
+  <para>Unix-like applications make extensive use of
+  <literal>fork</literal>, a function which spawns an exact copy of
+  the running process. Notable fork-using applications include bash
+  (and bash scripts), emacs, gcc, make, perl, python, and
+  ruby. Unfortunately, the Windows ecosystem is quite hostile to a
+  reliable fork implementation, leading to error messages such as:</para>
+  <para><itemizedlist>
+    <listitem>unable to remap <emphasis>$dll</emphasis> to same address as parent</listitem>
+    <listitem>couldn't allocate heap </listitem>
+    <listitem>died waiting for dll loading </listitem>
+    <listitem>child -1 - died waiting for longjmp before initialization</listitem>
+    <listitem>STATUS_ACCESS_VIOLATION </listitem>
+    <listitem>resource temporarily unavailable </listitem>
+  </itemizedlist></para>
+  <para>If you find that frequent fork failures interfere with normal
+  use of cygwin, please try the following: </para>
+  <para><itemizedlist>
+    <listitem>Restart whatever process is trying (and failing) to use
+    <literal>fork</literal>. Sometimes Windows sets up a process
+    environment that is even more hostile to fork than usual.</listitem>
+    <listitem>Ensure that you have eliminated (not just disabled) all
+    software on the BLODA (see <ulink
+    url="http://cygwin.com/faq/faq.using.html#faq.using.bloda"
+    />)</listitem>
+    <listitem>Install the 'rebase' package, read its README in
+    <literal>/usr/share/doc/Cygwin</literal>, and follow the
+    instructions there to run 'rebaseall'.</listitem>
+    </itemizedlist></para>
+  <para>Please note that installing new packages or updating existing
+  ones often undoes the effects of rebaseall and cause fork failures
+  to reappear. If so, just run rebaseall again.
+  </para></answer>
+</qandaentry>
+<qandaentry id='faq.using.why-fork-fails'>
+  <question><para>Why does <literal>fork</literal> fail so much,
+  anyway? (or: Why does <literal>fork</literal> still fail even though
+  I ran rebaseall?)</para></question>
+  <answer>
+  <para>The semantics of <literal>fork</literal> require that a forked
+  child process have <emphasis>exactly</emphasis> the same address
+  space layout as its parent. However, Windows provides no native
+  support for cloning address space between processes and several
+  features actively undermine a reliable <literal>fork</literal>
+  implementation. Three issues are especially prevalent:</para>
+  <para><itemizedlist>
+    <listitem>DLL base address collisions. Unlike *nix shared
+    libraries, which use "position-independent code", Windows shared
+    libraries assume a fixed base address. Whenever the hard-wired
+    address ranges of two DLLs collide (which occurs quite often), the
+    Windows loader must "rebase" one of them to a different
+    address. However, it does not resolve collisions consistently, and
+    may rebase a different dll and/or move it to a different address
+    every time. Cygwin can usually compensate for this effect when it
+    involves libraries opened dynamically, but collisions among
+    statically-linked dlls (dependencies known at compile time) are
+    resolved before <literal>cygwin1.dll</literal> initializes and
+    cannot be fixed afterward. This problem can only be solved by
+    removing the base address conflicts which cause the problem,
+    usually using the <literal>rebaseall</literal> package.</listitem>
+
+    <listitem>Address space layout randomization (ASLR). Starting with
+    Vista, Windows implements ASLR, which means that thread stacks,
+    heap, memory-mapped files, and statically-linked dlls are placed
+    at different (random) locations in each process. This behavior
+    interferes with a proper <literal>fork</literal>, and if an
+    unmovable object (process heap or system dll) ends up at the wrong
+    location, Cygwin can do nothing to compensate (though it will
+    retry a few times automatically). In a 64-bit system, marking
+    executables as large address-ware and rebasing dlls to high
+    addresses has been reported to help, as ASLR affects only the
+    lower 2GB of address space.</listitem>
+
+    <listitem>DLL injection by BLODA. Badly-behaved applications which
+    inject dlls into other processes often manage to clobber important
+    sections of the child's address space, leading to base address
+    collisions which rebasing cannot fix. The only way to resolve this
+    problem is to remove (usually uninstall) the offending
+    app.</listitem></itemizedlist></para>
+    <para>In summary, current Windows implementations make it
+    impossible to implement a perfectly reliable fork, and occasional
+    fork failures are inevitable. PTC.
+    </para>
+  </answer>
+</qandaentry>