RE: [EXTERNAL] Re: Issues With Thread Events In User Mode GDBServer

[Bill Messmer <wmessmer@microsoft.com>]

Simon,

Thanks for the response.  I have a silly test app which spins up and then waits on 5 pthreads each of which sleep for a varying amount, compute a Fibonacci number, and then exit.  I spin up the standard gdbserver fetched from Ubuntu VM (gdbserver localhost:1234 ./thread_test) and connect to it.  With that gdbserver, I get the following communication log between my plug-in and the gdbserver (somewhat condensed -- I removed most of the memory related back and forth):

    GDBServerComposition: Command: ?
    GDBServerComposition: CommandOuptut: T0506:0000000000000000;07:10dfffffff7f0000;10:b032fef7ff7f0000;thread:262b;core:e;
    GDBServerComposition: Command: qSupported
    GDBServerComposition: CommandOuptut: PacketSize=47ff;QPassSignals+;QProgramSignals+;QStartupWithShell+;QEnvironmentHexEncoded+;QEnvironmentReset+;QEnvironmentUnset+;QSetWorkingDir+;QCatchSyscalls+;qXfer:libraries-svr4:read+;augmented-libraries-svr4-read+;qXfer:auxv:read+;qXfer:siginfo:read+;qXfer:siginfo:write+;qXfer:features:read+;QStartNoAckMode+;qXfer:osdata:read+;multiprocess+;fork-events+;vfork-events+;exec-events+;QNonStop+;QDisableRandomization+;qXfer:threads:read+;ConditionalTracepoints+;TraceStateVariables+;TracepointSource+;DisconnectedTracing+;StaticTracepoints+;InstallInTrace+;qXfer:statictrace:read+;qXfer:traceframe-info:read+;EnableDisableTracepoints+;QTBuffer:size+;tracenz+;ConditionalBreakpoints+;BreakpointCommands+;QAgent+;Qbtrace:bts+;Qbtrace-conf:bts:size+;Qbtrace:pt+;Qbtrace-conf:pt:size+;Qbtrace:off+;qXfer:btrace:read+;qXfer:btrace-conf:read+;swbreak+;hwbreak+;qXfer:exec-file:read+;vContSupported+;QThreadEvents+;no-resumed+
    GDBServerComposition: Command: qSupported:QThreadEvents+
    GDBServerComposition: CommandOuptut: PacketSize=47ff;QPassSignals+;QProgramSignals+;QStartupWithShell+;QEnvironmentHexEncoded+;QEnvironmentReset+;QEnvironmentUnset+;QSetWorkingDir+;QCatchSyscalls+;qXfer:libraries-svr4:read+;augmented-libraries-svr4-read+;qXfer:auxv:read+;qXfer:siginfo:read+;qXfer:siginfo:write+;qXfer:features:read+;QStartNoAckMode+;qXfer:osdata:read+;multiprocess+;fork-events+;vfork-events+;exec-events+;QNonStop+;QDisableRandomization+;qXfer:threads:read+;ConditionalTracepoints+;TraceStateVariables+;TracepointSource+;DisconnectedTracing+;StaticTracepoints+;InstallInTrace+;qXfer:statictrace:read+;qXfer:traceframe-info:read+;EnableDisableTracepoints+;QTBuffer:size+;tracenz+;ConditionalBreakpoints+;BreakpointCommands+;QAgent+;Qbtrace:bts+;Qbtrace-conf:bts:size+;Qbtrace:pt+;Qbtrace-conf:pt:size+;Qbtrace:off+;qXfer:btrace:read+;qXfer:btrace-conf:read+;swbreak+;hwbreak+;qXfer:exec-file:read+;vContSupported+;QThreadEvents+;no-resumed+
    GDBServerComposition: Command: QThreadEvents:1
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: QNonStop:0
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: qXfer:features:read:target.xml:0,3e8
    GDBServerComposition: CommandOuptut: l<target><architecture>i386:x86-64</architecture><osabi>GNU/Linux</osabi></target>
    GDBServerComposition: Command: qXfer:threads:read::0,3e8
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: qXfer:exec-file:read:262b:0,3e8
    GDBServerComposition: CommandOuptut: l/home/wmessmer/thread_test/thread_test
    GDBServerComposition: Command: qXfer:auxv:read::0,3e8
    GDBServerComposition: CommandOuptut: l!

    <<<< SIGNIFICANT MEMORY RELATED COMMUNICATION REMOVED HERE >>>>

    GDBServerComposition: Command: Z0,555555555100,1
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: qXfer:threads:read::0,1000
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: qXfer:libraries-svr4:read::0,1000
    GDBServerComposition: CommandOuptut: l<library-list-svr4 version="1.0"/>
    GDBServerComposition: Command: m555555554000,400
    GDBServerComposition: CommandOuptut: 7f454c46...
    GDBServerComposition: Command: m555555554400,400
    GDBServerComposition: CommandOuptut: 00000000...
    GDBServerComposition: Command: m555555554800,400
    GDBServerComposition: CommandOuptut: 00000000...
    GDBServerComposition: Command: m555555554c00,400
    GDBServerComposition: CommandOuptut: 00000000...
    GDBServerComposition: Command: m555555554000,40
    GDBServerComposition: CommandOuptut: 7f454c4602010100000000000000000003003e000100000000110000000000004000000000000000004000000000000000000000400038000d00400025002400
    GDBServerComposition: Command: m555555554000,4
    GDBServerComposition: CommandOuptut: 7f454c46
    GDBServerComposition: Command: m555555554000,4c
    GDBServerComposition: CommandOuptut: 7f454c4602010100000000000000000003003e000100000000110000000000004000000000000000004000000000000000000000400038000d00400025002400060000000400000040000000
    ModLoad: 00005555`55554000 00005555`55558018   /home/wmessmer/thread_test/thread_test
    GDBServerComposition: Command: m555555554000,40
    GDBServerComposition: CommandOuptut: 7f454c4602010100000000000000000003003e000100000000110000000000004000000000000000004000000000000000000000400038000d00400025002400
    GDBServerComposition: Command: m555555554000,4
    GDBServerComposition: CommandOuptut: 7f454c46
    GDBServerComposition: Command: m555555554000,4c
    GDBServerComposition: CommandOuptut: 7f454c4602010100000000000000000003003e000100000000110000000000004000000000000000004000000000000000000000400038000d00400025002400060000000400000040000000
    .
    GDBServerComposition: Command: m3d8,348
    GDBServerComposition: CommandOuptut: E01
    ReadVirtual() failed in GetXStateConfiguration() first read attempt (error == 0.)
    GDBServerComposition: Command: Hg262b
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: g
    GDBServerComposition: CommandOuptut: 00000000...

    <<<< SIGNIFICANT MEMORY RELATED COMMUNICATION REMOVED HERE >>>>
    <<<< INITIAL POINT AT WHICH THE GDBSERVER IS BROKEN IN >>>>

    00007fff`f7fe32b0 4889e7          mov     rdi,rsp
    0:000> g
    GDBServerComposition: Command: m7ffff7fe32b0,1
    GDBServerComposition: CommandOuptut: 48
    GDBServerComposition: Command: Hg262b
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: g
    GDBServerComposition: CommandOuptut: 00000000...
    GDBServerComposition: Command: Hg262b
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: P12=33000000
    GDBServerComposition: CommandOuptut: 
    GDBServerComposition: Command: g
    GDBServerComposition: CommandOuptut: 00000000...
    GDBServerComposition: Command: G00000000...
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: vCont;c
    GDBServerComposition: CommandOuptut: T0206:0000000000000000;07:10dfffffff7f0000;10:b032fef7ff7f0000;thread:262b;core:e;
    GDBServerComposition: Command: qXfer:threads:read::0,1000
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: qXfer:siginfo:read::0,3e8
    GDBServerComposition: CommandOuptut: l 
    
    (262b.262b): Signal SIGINT code SI_USER (Sent by kill, sigsend, raise) at 0x7ffff7fe32b0 originating from PID 262b
    First chance exceptions are reported before any exception handling.
    This exception may be expected and handled.

    GDBServerComposition: Command: Hg262b
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: g
    GDBServerComposition: CommandOuptut: 00000000...

    <<<< SIGNIFICANT MEMORY RELATED COMMUNICATION REMOVED HERE >>>>

    00007fff`f7fe32b0 4889e7          mov     rdi,rsp
    0:000> g

    GDBServerComposition: Command: m7ffff7fe32b0,1
    GDBServerComposition: CommandOuptut: 48
    GDBServerComposition: Command: vCont;c
    GDBServerComposition: CommandOuptut: T0506:0000000000000000;07:10dfffffff7f0000;10:0151555555550000;thread:262b;core:e;
    GDBServerComposition: Command: z0,555555555100,1
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: g
    GDBServerComposition: CommandOuptut: 1c000000...
    GDBServerComposition: Command: G1c000000...
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: m555555557e68,8
    GDBServerComposition: CommandOuptut: 18e1fff7ff7f0000
    GDBServerComposition: Command: m7ffff7ffe128,8
    GDBServerComposition: CommandOuptut: 0063fcf7ff7f0000
    GDBServerComposition: Command: Z0,7ffff7fc6300,1
    GDBServerComposition: CommandOuptut: OK
    GDBServerComposition: Command: qXfer:libraries-svr4:read::0,1000
    GDBServerComposition: CommandOuptut: l<library-list-svr4 version="1.0" main-lm="0x7ffff7ffe2e0"><library name="linux-vdso.so.1" lm="0x7ffff7ffe890" l_addr="0x7ffff7fc2000" l_ld="0x7ffff7fc23a0"/><library name="/lib/x86_64-linux-gnu/libc.so.6" lm="0x7ffff7fbc160" l_addr="0x7ffff7d8e000" l_ld="0x7ffff7fa6bc0"/><library name="/lib64/ld-linux-x86-64.so.2" lm="0x7ffff7ffdaf0" l_addr="0x7ffff7fc3000" l_ld="0x7ffff7ffce80"/></library-list-svr4>

    <<<< SIGNIFICANT MEMORY RELATED COMMUNICATION TRUNCATED HERE >>>>

    ModLoad: 00007fff`f7fc2000 00007fff`f7fc2000   linux-vdso.so.1
    GDBServerComposition: Command: m7ffff7d8e000,40
    GDBServerComposition: CommandOuptut: 7f454c4602010103000000000000000003003e0001000000509f0200000000004000000000000000f0c021000000000000000000400038000e00400042004100
    GDBServerComposition: Command: m7ffff7d8e000,4
    GDBServerComposition: CommandOuptut: 7f454c46
    GDBServerComposition: Command: m7ffff7d8e000,4c
    GDBServerComposition: CommandOuptut: 7f454c4602010103000000000000000003003e0001000000509f0200000000004000000000000000f0c021000000000000000000400038000e00400042004100060000000400000040000000
    ModLoad: 00007fff`f7d8e000 00007fff`f7fb5e50   /lib/x86_64-linux-gnu/libc.so.6
    GDBServerComposition: Command: m7ffff7fc3000,40
    GDBServerComposition: CommandOuptut: 7f454c4602010103000000000000000003003e0001000000b002020000000000400000000000000068a603000000000000000000400038000b0040001b001a00
    GDBServerComposition: Command: m7ffff7fc3000,4
    GDBServerComposition: CommandOuptut: 7f454c46
    GDBServerComposition: Command: m7ffff7fc3000,4c
    GDBServerComposition: CommandOuptut: 7f454c4602010103000000000000000003003e0001000000b002020000000000400000000000000068a603000000000000000000400038000b0040001b001a00010000000400000000000000
    ModLoad: 00007fff`f7fc3000 00007fff`f7ffe2d8   /lib64/ld-linux-x86-64.so.2
    GDBServerComposition: Command: vCont;c
    GDBServerComposition: CommandOuptut: T05create:;06:80ffffffffffffff;07:009fd8f7ff7f0000;10:ed49ebf7ff7f0000;thread:262c;core:10;
    GDBServerComposition: Command: qXfer:threads:read::0,1000
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    <thread id="262c" core="16" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: vCont;c
    GDBServerComposition: CommandOuptut: T05create:;06:80ffffffffffffff;07:008f58f7ff7f0000;10:ed49ebf7ff7f0000;thread:262d;core:12;
    GDBServerComposition: Command: qXfer:threads:read::0,1000
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    <thread id="262c" core="16" name="thread_test"/>
    <thread id="262d" core="18" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: vCont;c
    GDBServerComposition: CommandOuptut: T05create:;06:80ffffffffffffff;07:007fd8f6ff7f0000;10:ed49ebf7ff7f0000;thread:262e;core:0;
    GDBServerComposition: Command: qXfer:threads:read::0,1000
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    <thread id="262c" core="16" name="thread_test"/>
    <thread id="262d" core="18" name="thread_test"/>
    <thread id="262e" core="0" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: vCont;c
    GDBServerComposition: CommandOuptut: T05create:;06:80ffffffffffffff;07:006f58f6ff7f0000;10:ed49ebf7ff7f0000;thread:262f;core:5;
    GDBServerComposition: Command: qXfer:threads:read::0,1000
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    <thread id="262c" core="16" name="thread_test"/>
    <thread id="262d" core="18" name="thread_test"/>
    <thread id="262e" core="0" name="thread_test"/>
    <thread id="262f" core="5" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: vCont;c
    GDBServerComposition: CommandOuptut: T05create:;06:80ffffffffffffff;07:005fd8f5ff7f0000;10:ed49ebf7ff7f0000;thread:2630;core:a;
    GDBServerComposition: Command: qXfer:threads:read::0,1000
    GDBServerComposition: CommandOuptut: l<threads>
    <thread id="262b" core="14" name="thread_test"/>
    <thread id="262c" core="16" name="thread_test"/>
    <thread id="262d" core="18" name="thread_test"/>
    <thread id="262e" core="0" name="thread_test"/>
    <thread id="262f" core="5" name="thread_test"/>
    <thread id="2630" core="10" name="thread_test"/>
    </threads>
    GDBServerComposition: Command: vCont;c

    <<<< GDBServer Crashes Here >>>>

The GDBServer then segfaults when the first thread exits.  GDB itself shows that the gdbserver faulted at:

    Program received signal SIGSEGV, Segmentation fault.
    resume (actions=actions@entry=0x55e85605f590, num_actions=num_actions@entry=1) at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:2966
    2966    /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc: No such file or directory.
    (gdb) bt
    #0  resume (actions=actions@entry=0x55e85605f590, num_actions=num_actions@entry=1)
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:2966
    #1  0x000055e854c61020 in handle_v_cont (own_buf=0x55e85604aed0 "vCont;c")
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:2910
    #2  handle_v_requests (own_buf=0x55e85604aed0 "vCont;c", packet_len=<optimized out>,
        new_packet_len=<optimized out>) at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:3177
    #3  0x000055e854c6299e in process_serial_event ()
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:4523
    #4  handle_serial_event (err=<optimized out>, client_data=<optimized out>)
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:4555
    #5  0x000055e854c994b6 in gdb_wait_for_event (block=block@entry=1)
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbsupport/event-loop.cc:700
    #6  0x000055e854c9994b in gdb_wait_for_event (block=1)
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbsupport/event-loop.cc:596
    #7  gdb_do_one_event () at /build/gdb-wIRHdd/gdb-12.0.90/gdbsupport/event-loop.cc:237
    #8  0x000055e854c50872 in start_event_loop ()
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:3553
    #9  captured_main (argv=<optimized out>, argc=<optimized out>)
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:4033
    #10 main (argc=<optimized out>, argv=<optimized out>)
        at /build/gdb-wIRHdd/gdb-12.0.90/gdbserver/server.cc:4119

So I went into *resume* and added the "cs.last_status.kind() != TARGET_WAITKIND_THREAD_EXITED) to the below code in that function as the "current_thread->last_status" reference is the source of the segfault:

      if (cs.last_status.kind () != TARGET_WAITKIND_EXITED
          && cs.last_status.kind () != TARGET_WAITKIND_SIGNALLED
          && cs.last_status.kind () != TARGET_WAITKIND_NO_RESUMED
          && cs.last_status.kind () != TARGET_WAITKIND_THREAD_EXITED)
        current_thread->last_status = cs.last_status;

After making this change, the server no longer crashes at the first thread exit, but instead, I get a packet that is

    w0;2635

Here's the problem though.  When I receive the various "T05create;..." packets, the debuggee process is frozen.  There's a bunch of printf's in my test app...  and nothing happens until I issue the vCont back to the server.  On receipt of the w0;2635 packet, however, the process just keeps going...

I suspect that's a bug in the gdbserver (I'm no expert here in either gdbserver or its code).  That's the first question...  and the second is whether there's some other way that thread creations and exits get detected other than QThreadEvents:1 (as this doesn't seem to be well supported).

Sincerely,

Bill Messmer
wmessmer@microsoft.com

-----Original Message-----
From: Simon Marchi <simark@simark.ca> 
Sent: Sunday, September 11, 2022 11:56 AM
To: Bill Messmer <wmessmer@microsoft.com>; gdb@sourceware.org
Subject: [EXTERNAL] Re: Issues With Thread Events In User Mode GDBServer

[You don't often get email from simark@simark.ca. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

On 2022-09-09 16:04, Bill Messmer via Gdb wrote:
> Folks,
>
> Apologies if this is the wrong mailing list to ask a question regarding GDBServer / RSP and a potential bug.
>
> I have been working on new extensibility API surfaces for the Windows platform debuggers that allow folks to write plug-ins that can connect those debugging tools to a variety of new targets including ones that are not Windows based.  We've had the ability to do this for post-mortem targets for some time and are, of late, working to expand that API surface to various forms of live targets.
>
> As proof of concept for the API surface, I've been experimenting with writing such a plug-in to connect to the standard user mode GDBServer for Linux.  A few things I'll note:
>
>
>   1.  When thread events are enabled on the server via a QThreadEvents:1, GDBServer immediately crashes on any thread exit in "resume" on a NULL deref of current_thread.
>
>
>
>   1.  I tried a quick patch here (adding "cs.last_status.kind() != TARGET_WAITKIND_THREAD_EXITED") to the set of conditions that won't set "current_thread->last_status" and the wXXX thread exit packets get sent; however, regardless of whether the target is in non-stop mode or not, the process is STILL RUNNING at the time the server sends the "wXXX" packet.
>
>
> Am I missing something with GDBServer and thread events or is this just not well supported...?  The process seems to be stopped at the point that a thread creation event gets sent...  but not for a thread exit...  I assume that's a bug somewhere in GDBServer...?  Or am I misreading the docs at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceware.org%2Fgdb%2Fonlinedocs%2Fgdb%2FGeneral-Query-Packets.html&amp;data=05%7C01%7Cwmessmer%40microsoft.com%7C0880b41dbd38466bdd5a08da942739ce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637985193606221095%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=at6X74%2FAK1fkpb4aEGEZonPzTqzZ%2FYTgKeV7MEJcsUY%3D&amp;reserved=0...?  Is there some alternate means by which thread create/exit notifications come...?
>
> Sincerely,
>
> Bill Messmer
> wmessmer@microsoft.com<mailto:wmessmer@microsoft.com>

Hi Bill,

I don't quite understand the situation you are describing.  Can you maybe send a log of the communication between your tool and GDBserver?

Simon