frequent hangs running ldd

frequent hangs running ldd

[Jeremy Drake]

Seen on msys2, but doesn't seem specific to it.

Frequently, when running ldd in a loop, it will hang.  I managed to get a
backtrace from gdb with symbols:

(gdb) bt
#0  0x00007ffecea0fa34 in ntdll!ZwDeviceIoControlFile ()
   from /c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ffecbead7a9 in KERNELBASE!GetConsoleScreenBufferInfoEx ()
   from /c/Windows/System32/KERNELBASE.dll
#2  0x00007ffecbef58dc in KERNELBASE!GetConsoleTitleW ()
   from /c/Windows/System32/KERNELBASE.dll
#3  0x00007ffecbfb8195 in KERNELBASE!GetConsoleProcessList ()
   from /c/Windows/System32/KERNELBASE.dll
#4  0x000000018015851f in fhandler_pty_common::get_console_process_id (
    pid=19348, match=match@entry=true, cygwin=cygwin@entry=false,
    stub_only=stub_only@entry=false, nat=nat@entry=false)
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/pty.cc:95
#5  0x0000000180101e3b in fhandler_console::attach_console (
    owner=<optimized out>, err=err@entry=0x0)
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:76
#6  0x0000000180102d40 in fhandler_console::set_output_mode (m=tty::native,
    t=0x1a0030028, p=0x800021d68)
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:853
#7  0x000000018010d6cc in fhandler_console::set_console_mode_to_native ()
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:4189
#8  0x000000018010d71d in ContinueDebugEvent_Hooked (p=26628, t=26656, s=65538)
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:4238
#9  0x0000000100401bd7 in ?? ()
#10 0x000000010040279f in ?? ()
#11 0x00000001800483a7 in dll_crt0_1 ()
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/dcrt0.cc:1015
#12 0x0000000180045f63 in _cygtls::call2 (this=0x7ffffce00,
    func=0x180047240 <dll_crt0_1(void*)>, arg=0x0,
buf=buf@entry=0x7ffffcdf0)
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:41
#13 0x0000000180046014 in _cygtls::call (func=<optimized out>,
    arg=<optimized out>)
    at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:28
#14 0x0000000000000000 in ?? ()

Other attempts without symbols showed the same Windows APIs at least.

Re: frequent hangs running ldd

[Takashi Yano]

On Fri, 24 May 2024 11:50:35 -0700 (PDT)
Jeremy Drake wrote:
> Seen on msys2, but doesn't seem specific to it.
> 
> Frequently, when running ldd in a loop, it will hang.  I managed to get a
> backtrace from gdb with symbols:
> 
> (gdb) bt
> #0  0x00007ffecea0fa34 in ntdll!ZwDeviceIoControlFile ()
>    from /c/Windows/SYSTEM32/ntdll.dll
> #1  0x00007ffecbead7a9 in KERNELBASE!GetConsoleScreenBufferInfoEx ()
>    from /c/Windows/System32/KERNELBASE.dll
> #2  0x00007ffecbef58dc in KERNELBASE!GetConsoleTitleW ()
>    from /c/Windows/System32/KERNELBASE.dll
> #3  0x00007ffecbfb8195 in KERNELBASE!GetConsoleProcessList ()
>    from /c/Windows/System32/KERNELBASE.dll
> #4  0x000000018015851f in fhandler_pty_common::get_console_process_id (
>     pid=19348, match=match@entry=true, cygwin=cygwin@entry=false,
>     stub_only=stub_only@entry=false, nat=nat@entry=false)
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/pty.cc:95
> #5  0x0000000180101e3b in fhandler_console::attach_console (
>     owner=<optimized out>, err=err@entry=0x0)
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:76
> #6  0x0000000180102d40 in fhandler_console::set_output_mode (m=tty::native,
>     t=0x1a0030028, p=0x800021d68)
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:853
> #7  0x000000018010d6cc in fhandler_console::set_console_mode_to_native ()
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:4189
> #8  0x000000018010d71d in ContinueDebugEvent_Hooked (p=26628, t=26656, s=65538)
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:4238
> #9  0x0000000100401bd7 in ?? ()
> #10 0x000000010040279f in ?? ()
> #11 0x00000001800483a7 in dll_crt0_1 ()
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/dcrt0.cc:1015
> #12 0x0000000180045f63 in _cygtls::call2 (this=0x7ffffce00,
>     func=0x180047240 <dll_crt0_1(void*)>, arg=0x0,
> buf=buf@entry=0x7ffffcdf0)
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:41
> #13 0x0000000180046014 in _cygtls::call (func=<optimized out>,
>     arg=<optimized out>)
>     at /C/_/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:28
> #14 0x0000000000000000 in ?? ()
> 
> Other attempts without symbols showed the same Windows APIs at least.

Thanks for the report. However, I cannot reproduce the issue.
If it always hangs in GetConsoleProcessList (), I doubt it is not a cygwin
bug but a windows bug.

By any chance, is the number of processes that attach to the same pty more
than 32768 in your environment?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: frequent hangs running ldd

[Takashi Yano]

On Sat, 25 May 2024 04:54:24 +0900
Takashi Yano wrote:
> By any chance, is the number of processes that attach to the same pty more
> than 32768 in your environment?

s/32768/8192/

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: frequent hangs running ldd

[Jeremy Drake]

On Sat, 25 May 2024, Takashi Yano wrote:

> Thanks for the report. However, I cannot reproduce the issue.
> If it always hangs in GetConsoleProcessList (), I doubt it is not a cygwin
> bug but a windows bug.
>
> By any chance, is the number of processes that attach to the same pty more
> than 32768 in your environment?
>

I doubt it, I was running a shell with this command:
find /usr/bin -name \*.dll -printf '%p:\n' -exec ldd '{}' \;

This was in Windows Terminal, msys2-runtime 3.5.3, on Windows 11
10.0.22631.3593.  (I realized I forgot to include these details).

I will attempt to reproduce this in Windows Sandbox with Cygwin next.

Re: frequent hangs running ldd

[Takashi Yano]

On Fri, 24 May 2024 14:46:40 -0700 (PDT)
Jeremy Drake wrote:
> > Thanks for the report. However, I cannot reproduce the issue.
> > If it always hangs in GetConsoleProcessList (), I doubt it is not a cygwin
> > bug but a windows bug.
> >
> > By any chance, is the number of processes that attach to the same pty more
> > than 32768 in your environment?
> >
> 
> I doubt it, I was running a shell with this command:
> find /usr/bin -name \*.dll -printf '%p:\n' -exec ldd '{}' \;

Thanks for the details. I could reproduce the issue.
It seems that ldh.exe (which is called from ldd?) falls into infinite loop.
However, gdb cannot attach to ldh.exe...

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: frequent hangs running ldd

[Mark Geisert]

On 5/24/2024 3:17 PM, Takashi Yano via Cygwin wrote:
> On Fri, 24 May 2024 14:46:40 -0700 (PDT)
> Jeremy Drake wrote:
>>> Thanks for the report. However, I cannot reproduce the issue.
>>> If it always hangs in GetConsoleProcessList (), I doubt it is not a cygwin
>>> bug but a windows bug.
>>>
>>> By any chance, is the number of processes that attach to the same pty more
>>> than 32768 in your environment?
>>>
>>
>> I doubt it, I was running a shell with this command:
>> find /usr/bin -name \*.dll -printf '%p:\n' -exec ldd '{}' \;
> 
> Thanks for the details. I could reproduce the issue.
> It seems that ldh.exe (which is called from ldd?) falls into infinite loop.
> However, gdb cannot attach to ldh.exe...

I could reproduce this too, even to not being able to attach gdb. If you 
exit that gdb session, I think you'll find the target process still 
stuck. You might be able to attach gdb again and it'll work. Here's what 
I found:

~ gdb -q /usr/bin/ldd
Reading symbols from /usr/bin/ldd...
Reading symbols from /usr/lib/debug//usr/bin/ldd.exe.dbg...
(gdb) att 6807
Attaching to program: /usr/bin/ldd, process 9732
[New Thread 9732.0x36a4]
[New Thread 9732.0x2bac]
(gdb) i thr
   Id   Target Id                Frame
   1    Thread 9732.0x31a8 "ldd" 0x00007ff8524f0b04 in 
ntdll!ZwWaitForDebugEvent
     () from /c/Windows/SYSTEM32/ntdll.dll
   2    Thread 9732.0x36a4 "sig" 0x00007ff8524ed174 in ntdll!ZwReadFile ()
    from /c/Windows/SYSTEM32/ntdll.dll
* 3    Thread 9732.0x2bac       0x00007ff8524f0be1 in ntdll!DbgBreakPoint ()
    from /c/Windows/SYSTEM32/ntdll.dll
(gdb) thr 1
[Switching to thread 1 (Thread 9732.0x31a8)]
#0  0x00007ff8524f0b04 in ntdll!ZwWaitForDebugEvent ()
    from /c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ff8524f0b04 in ntdll!ZwWaitForDebugEvent ()
    from /c/Windows/SYSTEM32/ntdll.dll
#1  0x00007ff850165796 in WaitForDebugEventEx ()
    from /c/Windows/System32/KERNELBASE.dll
#2  0x0000000100401ba1 in report (
     in_fn=0x800009200 "/usr/bin/cygdialog-14.dll",
     multiple=multiple@entry=false)
     at /usr/src/debug/cygwin-3.5.3-1/winsup/utils/ldd.cc:325
#3  0x00000001004026de in main (argc=<optimized out>, argv=0xa000004d0)
     at /usr/src/debug/cygwin-3.5.3-1/winsup/utils/ldd.cc:439
(gdb) f 2
#2  0x0000000100401ba1 in report (
     in_fn=0x800009200 "/usr/bin/cygdialog-14.dll",
     multiple=multiple@entry=false)
     at /usr/src/debug/cygwin-3.5.3-1/winsup/utils/ldd.cc:325
325    	      if (!WaitForDebugEvent (&ev, INFINITE))
(gdb) list
320
321    	  while (1)
322    	    {
323    	      bool exitnow = false;
324    	      DWORD cont = DBG_CONTINUE;
325    	      if (!WaitForDebugEvent (&ev, INFINITE))
326    	       	break;
327    	      switch (ev.dwDebugEventCode)
328    	       	{
329    	       	case CREATE_PROCESS_DEBUG_EVENT:
(gdb)

..mark

Re: frequent hangs running ldd

[Jeremy Drake]

On Sat, 25 May 2024, Takashi Yano wrote:

> On Fri, 24 May 2024 14:46:40 -0700 (PDT)
> Jeremy Drake wrote:
> > > Thanks for the report. However, I cannot reproduce the issue.
> > > If it always hangs in GetConsoleProcessList (), I doubt it is not a cygwin
> > > bug but a windows bug.
> > >
> > > By any chance, is the number of processes that attach to the same pty more
> > > than 32768 in your environment?
> > >
> >
> > I doubt it, I was running a shell with this command:
> > find /usr/bin -name \*.dll -printf '%p:\n' -exec ldd '{}' \;
>
> Thanks for the details. I could reproduce the issue.
> It seems that ldh.exe (which is called from ldd?) falls into infinite loop.
> However, gdb cannot attach to ldh.exe...
>

Windbg reports that ldh.exe is already being debugged.  I was able to do a
"non-invasive" attach to ldh.exe in windbg, but it doesn't seem to be able
to deal with the split debug symbols (gnulink?).  I don't know if gdb can
do a non-invasive attach like that (or open a minidump assuming one could
be made from a non-invasize attach in windbg).

Re: frequent hangs running ldd

[Jeremy Drake]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Fri, 24 May 2024, Jeremy Drake wrote:

> Windbg reports that ldh.exe is already being debugged.  I was able to do a
> "non-invasive" attach to ldh.exe in windbg, but it doesn't seem to be able
> to deal with the split debug symbols (gnulink?).  I don't know if gdb can
> do a non-invasive attach like that (or open a minidump assuming one could
> be made from a non-invasize attach in windbg).

Seems it can, and at least lldb can load a minidump (unfortunately it's
not showing source file/line info like gdb does):
(lldb) bt
* thread #1, stop reason = Exception 0x80000007 encountered at address
0x000000
  * frame #0: 0x0000000180178837 msys-2.0.dll`cygheap_init()
    frame #1: 0x0000000180178b89 msys-2.0.dll`setup_cygheap()
    frame #2: 0x00000001800488bd msys-2.0.dll`dll_crt0_0()
    frame #3: 0x000000018007197c msys-2.0.dll`dll_entry(h=0x0000000180040000, reason=<unavailable>, static_load=<unavailable>)
    frame #4: 0x00007ffece99869f ntdll.dll`RtlActivateActivationContextUnsafeFast + 303
    frame #5: 0x00007ffece9dd03d ntdll.dll`RtlEnumerateEntryHashTable + 957
    frame #6: 0x00007ffece9dcdee ntdll.dll`RtlEnumerateEntryHashTable + 366
    frame #7: 0x00007ffece9dce60 ntdll.dll`RtlEnumerateEntryHashTable + 480
    frame #8: 0x00007ffece9dce60 ntdll.dll`RtlEnumerateEntryHashTable + 480
    frame #9: 0x00007ffece9dce60 ntdll.dll`RtlEnumerateEntryHashTable + 480
    frame #10: 0x00007ffece9dce60 ntdll.dll`RtlEnumerateEntryHashTable + 480
    frame #11: 0x00007ffece99d62d ntdll.dll`RtlCopyUnicodeString + 1293
    frame #12: 0x00007ffece998940 ntdll.dll`RtlImageRvaToSection + 592
    frame #13: 0x00007ffece988cac ntdll.dll`RtlUnicodeToCustomCPN + 1020
    frame #14: 0x00007ffece99a25a ntdll.dll`LdrLoadDll + 250
    frame #15: 0x00007ffecbe961e2 KERNELBASE.dll`LoadLibraryExW + 370
    frame #16: 0x00007ff6df3414ba ldh.exe
    frame #17: 0x00007ff6df3412ee ldh.exe
    frame #18: 0x00007ff6df341406 ldh.exe
    frame #19: 0x00007ffecdc6257d kernel32.dll`BaseThreadInitThunk + 29
    frame #20: 0x00007ffece9caa48 ntdll.dll`RtlUserThreadStart + 40

[-- Attachment #2: Type: text/plain, Size: 2773 bytes --]

msys-2.0.dll`cygheap_init:
    0x180178750 <+0>:   pushq  %rbx
    0x180178751 <+1>:   subq   $0x20, %rsp
    0x180178755 <+5>:   leaq   0x102de4(%rip), %rcx ; _sys_nerr + 148864
    0x18017875c <+12>:  leaq   0x153b2f(%rip), %rdx ; __infinity + 5074
    0x180178763 <+19>:  callq  0x1800cce50    ; muto::init at 32:1
    0x180178768 <+24>:  movq   0x102e01(%rip), %rcx ; _sys_nerr + 148912
    0x18017876f <+31>:  leaq   0x102e02(%rip), %rax ; _sys_nerr + 148920
    0x180178776 <+38>:  cmpq   %rax, %rcx
    0x180178779 <+41>:  je     0x1801787d0    ; <+128> at 294:47
    0x18017877b <+43>:  cmpq   $0x0, 0x4870(%rcx)
    0x180178783 <+51>:  je     0x1801787a0    ; <+80> at 311:3
    0x180178785 <+53>:  cmpq   $0x0, 0x48a0(%rcx)
    0x18017878d <+61>:  je     0x1801787b5    ; <+101> at 312:14
    0x18017878f <+63>:  addq   $0x20, %rsp
    0x180178793 <+67>:  popq   %rbx
    0x180178794 <+68>:  jmp    0x180178680    ; init_cygheap::init_tls_list at 638:1
    0x180178799 <+73>:  nopl   (%rax)
    0x1801787a0 <+80>:  cmpq   $0x0, 0x48a0(%rcx)
    0x1801787a8 <+88>:  movq   $0x3, 0x4888(%rcx)
    0x1801787b3 <+99>:  jne    0x18017878f    ; <+63> at 314:1
    0x1801787b5 <+101>: callq  0x1800c0c10    ; sigalloc at 125:1
    0x1801787ba <+106>: movq   0x102daf(%rip), %rcx ; _sys_nerr + 148912
    0x1801787c1 <+113>: addq   $0x20, %rsp
    0x1801787c5 <+117>: popq   %rbx
    0x1801787c6 <+118>: jmp    0x180178680    ; init_cygheap::init_tls_list at 638:1
    0x1801787cb <+123>: nopl   (%rax,%rax)
    0x1801787d0 <+128>: movabsq $0x800000000, %rbx ; imm = 0x800000000
    0x1801787da <+138>: movl   $0x1, %r9d
    0x1801787e0 <+144>: movl   $0x2000, %r8d ; imm = 0x2000
    0x1801787e6 <+150>: movabsq $0x200000000, %rdx ; imm = 0x200000000
    0x1801787f0 <+160>: movq   %rbx, %rcx
    0x1801787f3 <+163>: callq  0x180217090    ; _alloca + 4336
    0x1801787f8 <+168>: movq   %rbx, %rcx
    0x1801787fb <+171>: movl   $0x4, %r9d
    0x180178801 <+177>: movl   $0x1000, %r8d ; imm = 0x1000
    0x180178807 <+183>: movl   $0x300000, %edx ; imm = 0x300000
    0x18017880c <+188>: movq   %rax, 0x102d5d(%rip) ; _sys_nerr + 148912
    0x180178813 <+195>: callq  0x180217090    ; _alloca + 4336
    0x180178818 <+200>: movq   %rax, %rcx
    0x18017881b <+203>: movq   %rax, 0x102d4e(%rip) ; _sys_nerr + 148912
    0x180178822 <+210>: leaq   0x48e0(%rax), %rax
    0x180178829 <+217>: movq   %rax, 0x102d38(%rip) ; _sys_nerr + 148904
    0x180178830 <+224>: leaq   0x3d319(%rip), %rax ; __utf8_mbtowc at 534:1
->  0x180178837 <+231>: movq   %rax, (%rcx)
    0x18017883a <+234>: movl   $0x12, 0x4828(%rcx)
    0x180178844 <+244>: movq   $0x0, 0x4830(%rcx)
    0x18017884f <+255>: jmp    0x18017877b    ; <+43> at 309:3

Re: frequent hangs running ldd

[Jeremy Drake]

On Fri, 24 May 2024, Jeremy Drake wrote:

> On Fri, 24 May 2024, Jeremy Drake wrote:
>
> > Windbg reports that ldh.exe is already being debugged.  I was able to do a
> > "non-invasive" attach to ldh.exe in windbg, but it doesn't seem to be able
> > to deal with the split debug symbols (gnulink?).  I don't know if gdb can
> > do a non-invasive attach like that (or open a minidump assuming one could
> > be made from a non-invasize attach in windbg).
>
> Seems it can, and at least lldb can load a minidump (unfortunately it's
> not showing source file/line info like gdb does):
> (lldb) bt
> * thread #1, stop reason = Exception 0x80000007 encountered at address
> 0x000000
>   * frame #0: 0x0000000180178837 msys-2.0.dll`cygheap_init()

It appears that cygheap is NULL, so I'm guessing that VirtualAlloc failed.
!gle in windbg shows
0:000> !gle
LastErrorValue: (Win32) 0x1e7 (487) - Attempt to access invalid address.
LastStatusValue: (NTSTATUS) 0xc0000018 - {Conflicting Address Range}  The
specified address range conflicts with the address space.

Looking at !address, it seems Windows put the PEB, TEBs, and stacks in the
area where the cygheap should be.  Way to go, ASLR :P
        BaseAddress      EndAddress+1        RegionSize     Type       State                 Protect             Usage
--------------------------------------------------------------------------------------------------------------------------
+        5`e8181000        8`05a00000        2`1d87f000             MEM_FREE    PAGE_NOACCESS                      Free
+        8`05a00000        8`05b57000        0`00157000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
         8`05b57000        8`05b58000        0`00001000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     PEB        [4628]
         8`05b58000        8`05b5a000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     TEB        [~0; 4628.31ac]
         8`05b5a000        8`05b5c000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     TEB        [~1; 4628.4aac]
         8`05b5c000        8`05b5e000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     TEB        [~2; 4628.5840]
         8`05b5e000        8`05b60000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     TEB        [~3; 4628.6b9c]
         8`05b60000        8`05c00000        0`000a0000 MEM_PRIVATE MEM_RESERVE                                    <unknown>
+        8`05c00000        8`05df6000        0`001f6000 MEM_PRIVATE MEM_RESERVE                                    Stack      [~0; 4628.31ac]
         8`05df6000        8`05df9000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE | PAGE_GUARD        Stack      [~0; 4628.31ac]
         8`05df9000        8`05e00000        0`00007000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     Stack      [~0; 4628.31ac]
+        8`05e00000        8`05ffb000        0`001fb000 MEM_PRIVATE MEM_RESERVE                                    Stack      [~1; 4628.4aac]
         8`05ffb000        8`05ffe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE | PAGE_GUARD        Stack      [~1; 4628.4aac]
         8`05ffe000        8`06000000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     Stack      [~1; 4628.4aac]
+        8`06000000        8`061fb000        0`001fb000 MEM_PRIVATE MEM_RESERVE                                    Stack      [~2; 4628.5840]
         8`061fb000        8`061fe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE | PAGE_GUARD        Stack      [~2; 4628.5840]
         8`061fe000        8`06200000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     Stack      [~2; 4628.5840]
+        8`06200000        8`063fb000        0`001fb000 MEM_PRIVATE MEM_RESERVE                                    Stack      [~3; 4628.6b9c]
         8`063fb000        8`063fe000        0`00003000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE | PAGE_GUARD        Stack      [~3; 4628.6b9c]
         8`063fe000        8`06400000        0`00002000 MEM_PRIVATE MEM_COMMIT  PAGE_READWRITE                     Stack      [~3; 4628.6b9c]
+        8`06400000      19e`64400000      196`5e000000             MEM_FREE    PAGE_NOACCESS                      Free

Re: frequent hangs running ldd

[Jeremy Drake]

On Fri, 24 May 2024, Jeremy Drake wrote:

> Looking at !address, it seems Windows put the PEB, TEBs, and stacks in the
> area where the cygheap should be.  Way to go, ASLR :P

I think the fix for this would be to add -Wl,--disable-high-entropy-va to
ldh_LDFLAGS, as was done for strace and cygcheck at least.  I used peflags
-d0 /usr/bin/ldh.exe and I'm not seeing a hang after that.

Re: frequent hangs running ldd

[Jeremy Drake]

On Fri, 24 May 2024, Jeremy Drake wrote:

> On Fri, 24 May 2024, Jeremy Drake wrote:
>
> > Looking at !address, it seems Windows put the PEB, TEBs, and stacks in the
> > area where the cygheap should be.  Way to go, ASLR :P
>
> I think the fix for this would be to add -Wl,--disable-high-entropy-va to
> ldh_LDFLAGS, as was done for strace and cygcheck at least.  I used peflags
> -d0 /usr/bin/ldh.exe and I'm not seeing a hang after that.

Sorry, that was peflags -e0 not -d0 (dynamicbase is still on):
$ peflags -v /usr/bin/ldh.exe
/usr/bin/ldh.exe:
coff(0x0226[+executable_image,+line_nums_stripped,+bigaddr,+sepdbg])
pe(0x0140[+dynamicbase,+nxcompat])

Re: frequent hangs running ldd

[Takashi Yano]

On Fri, 24 May 2024 19:29:43 -0700 (PDT)
Jeremy Drake wrote:
> On Fri, 24 May 2024, Jeremy Drake wrote:
> 
> > On Fri, 24 May 2024, Jeremy Drake wrote:
> >
> > > Looking at !address, it seems Windows put the PEB, TEBs, and stacks in the
> > > area where the cygheap should be.  Way to go, ASLR :P
> >
> > I think the fix for this would be to add -Wl,--disable-high-entropy-va to
> > ldh_LDFLAGS, as was done for strace and cygcheck at least.  I used peflags
> > -d0 /usr/bin/ldh.exe and I'm not seeing a hang after that.
> 
> Sorry, that was peflags -e0 not -d0 (dynamicbase is still on):
> $ peflags -v /usr/bin/ldh.exe
> /usr/bin/ldh.exe:
> coff(0x0226[+executable_image,+line_nums_stripped,+bigaddr,+sepdbg])
> pe(0x0140[+dynamicbase,+nxcompat])

You are right!

It seems that VirtualAlloc() in cygheap_init() in mm/cygheap.cc
fails when the address range which cygwin uses is occupied due to
high-entropy-va in ldh.exe.

Thanks for the analysis.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: frequent hangs running ldd

[Takashi Yano]

Hi Jeremy,

On Tue, 28 May 2024 10:58:00 +0900
Takashi Yano wrote:
> On Fri, 24 May 2024 19:29:43 -0700 (PDT)
> Jeremy Drake wrote:
> > On Fri, 24 May 2024, Jeremy Drake wrote:
> > 
> > > On Fri, 24 May 2024, Jeremy Drake wrote:
> > >
> > > > Looking at !address, it seems Windows put the PEB, TEBs, and stacks in the
> > > > area where the cygheap should be.  Way to go, ASLR :P
> > >
> > > I think the fix for this would be to add -Wl,--disable-high-entropy-va to
> > > ldh_LDFLAGS, as was done for strace and cygcheck at least.  I used peflags
> > > -d0 /usr/bin/ldh.exe and I'm not seeing a hang after that.
> > 
> > Sorry, that was peflags -e0 not -d0 (dynamicbase is still on):
> > $ peflags -v /usr/bin/ldh.exe
> > /usr/bin/ldh.exe:
> > coff(0x0226[+executable_image,+line_nums_stripped,+bigaddr,+sepdbg])
> > pe(0x0140[+dynamicbase,+nxcompat])
> 
> You are right!
> 
> It seems that VirtualAlloc() in cygheap_init() in mm/cygheap.cc
> fails when the address range which cygwin uses is occupied due to
> high-entropy-va in ldh.exe.
> 
> Thanks for the analysis.

Would you make a patch for that?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: frequent hangs running ldd

[Mark Geisert]

On 5/24/2024 3:26 PM, Jeremy Drake via Cygwin wrote:
> On Sat, 25 May 2024, Takashi Yano wrote:
> 
>> On Fri, 24 May 2024 14:46:40 -0700 (PDT)
>> Jeremy Drake wrote:
>>>> Thanks for the report. However, I cannot reproduce the issue.
>>>> If it always hangs in GetConsoleProcessList (), I doubt it is not a cygwin
>>>> bug but a windows bug.
>>>>
>>>> By any chance, is the number of processes that attach to the same pty more
>>>> than 32768 in your environment?
>>>>
>>>
>>> I doubt it, I was running a shell with this command:
>>> find /usr/bin -name \*.dll -printf '%p:\n' -exec ldd '{}' \;
>>
>> Thanks for the details. I could reproduce the issue.
>> It seems that ldh.exe (which is called from ldd?) falls into infinite loop.
>> However, gdb cannot attach to ldh.exe...
>>
> 
> Windbg reports that ldh.exe is already being debugged.  I was able to do a
> "non-invasive" attach to ldh.exe in windbg, but it doesn't seem to be able
> to deal with the split debug symbols (gnulink?).  I don't know if gdb can
> do a non-invasive attach like that (or open a minidump assuming one could
> be made from a non-invasize attach in windbg).

ldd is the debugger of ldh. I found that Sysinternals Process Explorer 
can attach to ldh, show the threads, and can get stack backtraces which 
are refreshable. You have to convert addresses shown there into 
source-relevant addresses manually.

I'm bowing out for now as I think Takashi has a handle on this.
Cheers,

..mark