Intermittent fork failures

Intermittent fork failures

[Igor Pechtchanski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1254 bytes --]

Hi,

I've been getting intermittent "fork: Resource Temporarily Unavailable"
errors with 1.5.18 and snapshots.  AFAICT, the resources are there.

I was able to reproduce both the succeeding and the failing case under
strace, but, to my untrained eye, it doesn't contain much relevant
information.  The differing sections for the failure and success cases (29
and 77 lines, respectively) are attached (the PIDs and some handle values
are masked to allow easier diffing -- I tried to be careful).  About 1000
lines leading to these sections are identical in the two cases.  Full
strace output is available on request.  Both attached straces were
produced under "uname -r" -> "1.5.19s(0.138/4/2)".

I cannot reproduce this under gdb.  Any pointers on how to debug this
further are appreciated.  What information would be relevant to put in the
strace?
	Igor
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_		pechtcha@cs.nyu.edu
ZZZzz /,`.-'`'    -.  ;-;;,_		igor@watson.ibm.com
     |,4-  ) )-,_. ,\ (  `'-'		Igor Pechtchanski, Ph.D.
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

If there's any real truth it's that the entire multidimensional infinity
of the Universe is almost certainly being run by a bunch of maniacs. /DA

[-- Attachment #2: Type: TEXT/PLAIN, Size: 2787 bytes --]

XXXXX XXXXXXXX [main] bash HHHHH stack_base: bottom 0x230000, top 0x30000, stack 0x22E5C0, size 6720, reserve 2097152
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: CreateProcess (C:\cygwin\bin\bash.exe, C:\cygwin\bin\bash.exe, 0, 0, 1, 0x20, 0, 0, 0x22E5E0, 0x22E630)
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: pinfo failed
XXXXX XXXXXXXX [main] bash HHHHH __set_errno: int fork_parent(void*&, dll*&, bool&, void*, child_info_fork&):378 val 11
XXXXX XXXXXXXX [main] bash HHHHH sig_send: sendsig 0x728, pid HHHHH, signal -41, its_me 1
XXXXX XXXXXXXX [main] bash HHHHH sig_send: wakeup 0x6B4
XXXXX XXXXXXXX [main] bash HHHHH sig_send: Waiting for pack.wakeup 0x6B4
XXXXX XXXXXXXX [main] bash HHHHH sig_send: returning 0x0 from sending signal -41
XXXXX XXXXXXXX [main] bash HHHHH fork: -1 = fork()

[-- Attachment #3: Type: TEXT/PLAIN, Size: 6413 bytes --]

XXXXX XXXXXXXX [main] bash HHHHH stack_base: bottom 0x230000, top 0x30000, stack 0x22E5C0, size 6720, reserve 2097152
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: CreateProcess (C:\cygwin\bin\bash.exe, C:\cygwin\bin\bash.exe, 0, 0, 1, 0x20, 0, 0, 0x22E5E0, 0x22E630)
XXXXX XXXXXXXX [main] bash HHHHH open_shared: name cygpid, shared 0x11C0000 (wanted 0x0), h 0x6AC
XXXXX XXXXXXXX [main] bash HHHHH time: 1128400280 = time (0)
XXXXX XXXXXXXX [main] bash HHHHH proc_subproc: args: 1, 2286976
XXXXX XXXXXXXX [main] bash HHHHH _pinfo::dup_proc_pipe: closed wr_proc_pipe 0x7FC for pid 12676(12676)
XXXXX XXXXXXXX [main] bash HHHHH pinfo::wait: created tracking thread for pid 12676, winpid 0x3184, rd_pipe 0x6B0
XXXXX XXXXXXXX [main] bash HHHHH proc_subproc: added pid 12676 to proc table, slot 0
XXXXX XXXXXXXX [main] bash HHHHH proc_subproc: returning 1
XXXXX XXXXXXXX [main] bash HHHHH child_info::sync: waiting for subproc_ready(0xHDD) and child process(0x680)
XXXXX XXXXXXXX [proc_waiter] bash HHHHH cygthread::stub: thread 'proc_waiter', id 0x3720, stack_ptr 0x13CEFC0
**********************************************
Program name: C:\cygwin\bin\bash.exe (pid 12676, ppid HHHHH)
App version:  1005.19, api: 0.138
DLL version:  1005.19, api: 0.138
DLL build:    20050920 00:31:28SNP
OS version:   Windows NT-5.1
Heap size:    1073741824
Date/Time:    2005-10-04 XX:XX:XX
**********************************************
XXXXX XXXXXXXX [main] bash 12676 set_myself: myself->dwProcessId 12676
XXXXX XXXXXXXX [main] bash 12676 events_init: windows_system_directory 'C:\WINDOWS\System32\', windows_system_directory_length 20
XXXXX XXXXXXXX [main] bash 12676 fork_child: child is running.  pid 12676, ppid HHHHH, stack here 0x22E788
XXXXX XXXXXXXX [main] bash 12676 sync_with_parent: signalling parent: after longjmp
XXXXX XXXXXXXX [main] bash 12676 child_info::ready: signalled 0xHDD that I was ready
XXXXX XXXXXXXX [main] bash HHHHH child_info::sync: got subproc_ready for pid 12676
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: child is alive (but stopped)
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x456000, high 0x458290, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x46C000, high 0x46F070, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x10010000, high 0x10019000, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x22E510, high 0x230000, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x610F6000, high 0x610FAB60, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x61134000, high 0x6113D820, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: done
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: copying data/bss of a linked dll
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x5BD000, high 0x5BD020, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x5BE000, high 0x5BE3F0, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: done
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: copying data/bss of a linked dll
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x10008000, high 0x10008040, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x10009000, high 0x100094F0, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: done
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: copying data/bss of a linked dll
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x5F5000, high 0x5F56A0, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x604000, high 0x604960, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: done
XXXXX XXXXXXXX [main] bash HHHHH fork_parent: copying data/bss of a linked dll
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x703DA000, high 0x703DD170, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: child handle 0x680, low 0x703E0000, high 0x703E1390, res 1
XXXXX XXXXXXXX [main] bash HHHHH fork_copy: done
XXXXX XXXXXXXX [main] bash 12676 sync_with_parent: awake
XXXXX XXXXXXXX [main] bash 12676 sync_with_parent: no problems
XXXXX XXXXXXXX [main] bash 12676 fork_child: hParent 0x6B8, child 1 first_dll 0x610000, load_dlls 0
XXXXX XXXXXXXX [main] bash 12676 set_privilege: 0 = set_privilege ((token 798) SeRestorePrivilege, 1)
XXXXX XXXXXXXX [main] bash 12676 set_privilege: 1 = set_privilege ((token 798) SeChangeNotifyPrivilege, 1)
XXXXX XXXXXXXX [main] bash 12676 set_file_api_mode: File APIs set to ANSI
XXXXX XXXXXXXX [main] bash 12676 fixup_mmaps_after_fork: recreate_mmaps_after_fork, mmapped_areas 0x0
XXXXX XXXXXXXX [main] bash 12676 dtable::fixup_after_fork: fd 3 ()
XXXXX XXXXXXXX [main] bash 12676 fhandler_pipe::fixup_after_fork: here
XXXXX XXXXXXXX [main] bash 12676 fhandler_base::fixup_after_fork: inheriting '' from parent
XXXXX XXXXXXXX [main] bash HHHHH resume_child: signalled child
XXXXX XXXXXXXX [main] bash HHHHH child_info::sync: waiting for subproc_ready(0xHDD) and child process(0x680)
XXXXX XXXXXXXX [main] bash 12676 fhandler_base::fork_fixup: handle 0x6E0 already opened
XXXXX XXXXXXXX [main] bash 12676 fhandler_base::fork_fixup: handle 0x6C4 already opened
XXXXX XXXXXXXX [main] bash 12676 dtable::fixup_after_fork: fd 255 (/usr/src/wrappers-cvs/java/java)
XXXXX XXXXXXXX [main] bash 12676 fhandler_base::fixup_after_fork: inheriting '/usr/src/wrappers-cvs/java/java' from parent
XXXXX XXXXXXXX [main] bash 12676 sync_with_parent: signalling parent: performed fork fixup
XXXXX XXXXXXXX [main] bash HHHHH child_info::sync: got subproc_ready for pid 12676
XXXXX XXXXXXXX [main] bash 12676 child_info::ready: signalled 0xHDD that I was ready
XXXXX XXXXXXXX [main] bash 12676 sigproc_init: process/signal handling enabled(801)
XXXXX XXXXXXXX [main] bash 12676 pthread_mutex::_fixup_after_fork: mutex 10010060 in _fixup_after_fork
XXXXX XXXXXXXX [main] bash 12676 pthread_mutex::_fixup_after_fork: mutex 6113D158 in _fixup_after_fork
XXXXX XXXXXXXX [main] bash HHHHH sig_send: sendsig 0x728, pid HHHHH, signal -41, its_me 1
XXXXX XXXXXXXX [main] bash HHHHH sig_send: wakeup 0x6B4
XXXXX XXXXXXXX [main] bash HHHHH sig_send: Waiting for pack.wakeup 0x6B4
XXXXX XXXXXXXX [main] bash HHHHH sig_send: returning 0x0 from sending signal -41
XXXXX XXXXXXXX [main] bash HHHHH fork: 12676 = fork()

[-- Attachment #4: Type: text/plain, Size: 218 bytes --]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Intermittent fork failures

[Christopher Faylor]

On Tue, Oct 04, 2005 at 01:02:50PM -0400, Igor Pechtchanski wrote:
>I've been getting intermittent "fork: Resource Temporarily Unavailable"
>errors with 1.5.18 and snapshots.  AFAICT, the resources are there.

http://sources.redhat.com/ml/cygwin/2005-09/msg00945.html

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Intermittent fork failures

[Corinna Vinschen]

On Oct  4 13:02, Igor Pechtchanski wrote:
> Hi,
> 
> I've been getting intermittent "fork: Resource Temporarily Unavailable"
> errors with 1.5.18 and snapshots.  AFAICT, the resources are there.
> 
> I was able to reproduce both the succeeding and the failing case under
> strace, but, to my untrained eye, it doesn't contain much relevant
> information.  The differing sections for the failure and success cases (29
> and 77 lines, respectively) are attached (the PIDs and some handle values
> are masked to allow easier diffing -- I tried to be careful).  About 1000
> lines leading to these sections are identical in the two cases.  Full
> strace output is available on request.  Both attached straces were
> produced under "uname -r" -> "1.5.19s(0.138/4/2)".
> 
> I cannot reproduce this under gdb.  Any pointers on how to debug this
> further are appreciated.  What information would be relevant to put in the
> strace?

Did you try http://sources.redhat.com/ml/cygwin/2005-09/msg00945.html ?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat, Inc.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Intermittent fork failures

[Igor Pechtchanski]

On Tue, 4 Oct 2005, Christopher Faylor wrote:

> On Tue, Oct 04, 2005 at 01:02:50PM -0400, Igor Pechtchanski wrote:
> >I've been getting intermittent "fork: Resource Temporarily Unavailable"
> >errors with 1.5.18 and snapshots.  AFAICT, the resources are there.
>
> http://sources.redhat.com/ml/cygwin/2005-09/msg00945.html

Oops, I was trying very hard to provide all relevant information, but,
sure enough, I forgot to mention that I did see that message, and that the
problem manifested after I changed the limits.  I have:

$ cat /proc/registry/HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Session\ Manager/SubSystems/Windows
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=2048,4096,1024 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16

compared to the previous value of:

$ cat /proc/registry/HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Session\ Manager/SubSystems/Windows-prev
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16

I didn't quite double all the values -- perhaps doubling wasn't overkill
after all...
	Igor
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_		pechtcha@cs.nyu.edu
ZZZzz /,`.-'`'    -.  ;-;;,_		igor@watson.ibm.com
     |,4-  ) )-,_. ,\ (  `'-'		Igor Pechtchanski, Ph.D.
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

If there's any real truth it's that the entire multidimensional infinity
of the Universe is almost certainly being run by a bunch of maniacs. /DA

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/