[Bug gdb/30630] New: remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[Bug gdb/30630] New: remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[christian.prochaska@genode-labs.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

            Bug ID: 30630
           Summary: remote.c:8352: internal-error: wait: Assertion
                    `!async_event_handler_marked
                    (rs->remote_async_inferior_event_token)' failed.
           Product: gdb
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: gdb
          Assignee: unassigned at sourceware dot org
          Reporter: christian.prochaska@genode-labs.com
  Target Milestone: ---

I got this error while working on a custom GDB stub which sends two initial
stop replies and could reproduce it with gdbserver with the following test:

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>

void *thread_func(void *)
{
        sleep(1000);
}

int main()
{
        printf("pid: %u\n", getpid());
        pthread_t t;
        pthread_create(&t, 0, thread_func, 0);
        pthread_join(t, 0);
        return 0;
}

---

$ ./test 
pid: 13129
^Z
[1]+  Stopped                 ./test
$ ./gdbserver --attach :5555 13129 &
[2] 13578
$ Attached; pid = 13129
Listening on port 5555

$ ./gdb -ex "set non-stop on" -ex "target extended-remote :5555"
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Remote debugging using :5555
Remote debugging from host 127.0.0.1, port 59942
Reading /.../test from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to
access files locally instead.
Reading /.../test from remote target...
Reading symbols from target:/.../test...
../../gdb-13.2/gdb/remote.c:8352: internal-error: wait: Assertion
`!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
----- Backtrace -----
0x55e67dce9cbb gdb_internal_backtrace_1
        ../../gdb-13.2/gdb/bt-utils.c:122
0x55e67dce9cbb _Z22gdb_internal_backtracev
        ../../gdb-13.2/gdb/bt-utils.c:168
0x55e67e09ea1c internal_vproblem
        ../../gdb-13.2/gdb/utils.c:396
0x55e67e09ec3a _Z15internal_verrorPKciS0_P13__va_list_tag
        ../../gdb-13.2/gdb/utils.c:476
0x55e67e1e787e _Z18internal_error_locPKciS0_z
        ../../gdb-13.2/gdbsupport/errors.cc:58
0x55e67dfaac74
_ZN13remote_target4waitE6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
        ../../gdb-13.2/gdb/remote.c:8351
0x55e67e04cb78
_Z11target_wait6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
        ../../gdb-13.2/gdb/target.c:2579
0x55e67dfa2e18 _ZN13remote_target28process_initial_stop_repliesEi
        ../../gdb-13.2/gdb/remote.c:4579
0x55e67dfacfe6 _ZN13remote_target14start_remote_1Eii
        ../../gdb-13.2/gdb/remote.c:5020
0x55e67dfad668 _ZN13remote_target12start_remoteEii
        ../../gdb-13.2/gdb/remote.c:5072
0x55e67dfad668 _ZN13remote_target6open_1EPKcii
        ../../gdb-13.2/gdb/remote.c:5875
0x55e67e04d211 open_target
        ../../gdb-13.2/gdb/target.c:853
0x55e67dd1e0bf _Z8cmd_funcP16cmd_list_elementPKci
        ../../gdb-13.2/gdb/cli/cli-decode.c:2543
0x55e67e05cfcd _Z15execute_commandPKci
        ../../gdb-13.2/gdb/top.c:690
0x55e67defca94 catch_command_errors
        ../../gdb-13.2/gdb/main.c:513
0x55e67defcbb0 execute_cmdargs
        ../../gdb-13.2/gdb/main.c:608
0x55e67defe31b captured_main_1
        ../../gdb-13.2/gdb/main.c:1299
0x55e67defefca captured_main
        ../../gdb-13.2/gdb/main.c:1320
0x55e67defefca _Z8gdb_mainP18captured_main_args
        ../../gdb-13.2/gdb/main.c:1345
0x55e67dc2850a main
        ../../gdb-13.2/gdb/gdb.c:32
---------------------
../../gdb-13.2/gdb/remote.c:8352: internal-error: wait: Assertion
`!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) Recursive internal problem.


Fatal signal: Aborted
----- Backtrace -----
0x55e67dce9cbb gdb_internal_backtrace_1
        ../../gdb-13.2/gdb/bt-utils.c:122
0x55e67dce9cbb _Z22gdb_internal_backtracev
        ../../gdb-13.2/gdb/bt-utils.c:168
0x55e67de0d98a handle_fatal_signal
        ../../gdb-13.2/gdb/event-top.c:956
0x7f14ab8f2f0f ???
0x7f14ab8f2e87 ???
0x7f14ab8f47f0 ???
0x55e67dc2611f abort_with_message
        ../../gdb-13.2/gdb/utils.c:183
0x55e67e09eb2f internal_vproblem
        ../../gdb-13.2/gdb/utils.c:343
0x55e67e09ec3a _Z15internal_verrorPKciS0_P13__va_list_tag
        ../../gdb-13.2/gdb/utils.c:476
0x55e67e1e787e _Z18internal_error_locPKciS0_z
        ../../gdb-13.2/gdbsupport/errors.cc:58
0x55e67dfaac74
_ZN13remote_target4waitE6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
        ../../gdb-13.2/gdb/remote.c:8351
0x55e67e04cb78
_Z11target_wait6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
        ../../gdb-13.2/gdb/target.c:2579
0x55e67de9be49 do_target_wait_1
        ../../gdb-13.2/gdb/infrun.c:3736
0x55e67deae47a operator()
        ../../gdb-13.2/gdb/infrun.c:3795
0x55e67deae47a do_target_wait
        ../../gdb-13.2/gdb/infrun.c:3814
0x55e67deae47a _Z20fetch_inferior_eventv
        ../../gdb-13.2/gdb/infrun.c:4172
0x55e67dca3ac0 _Z26check_async_event_handlersv
        ../../gdb-13.2/gdb/async-event.c:337
0x55e67e1e8a9c _Z16gdb_do_one_eventi
        ../../gdb-13.2/gdbsupport/event-loop.cc:221
0x55e67e05a3c0 _Z20gdb_readline_wrapperPKc
        ../../gdb-13.2/gdb/top.c:1173
0x55e67e09de5e defaulted_query
        ../../gdb-13.2/gdb/utils.c:924
0x55e67e09e19a _Z5queryPKcz
        ../../gdb-13.2/gdb/utils.c:1016
0x55e67e09eaef internal_vproblem
        ../../gdb-13.2/gdb/utils.c:407
0x55e67e09ec3a _Z15internal_verrorPKciS0_P13__va_list_tag
        ../../gdb-13.2/gdb/utils.c:476
0x55e67e1e787e _Z18internal_error_locPKciS0_z
        ../../gdb-13.2/gdbsupport/errors.cc:58
0x55e67dfaac74
_ZN13remote_target4waitE6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
        ../../gdb-13.2/gdb/remote.c:8351
0x55e67e04cb78
_Z11target_wait6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
        ../../gdb-13.2/gdb/target.c:2579
0x55e67dfa2e18 _ZN13remote_target28process_initial_stop_repliesEi
        ../../gdb-13.2/gdb/remote.c:4579
0x55e67dfacfe6 _ZN13remote_target14start_remote_1Eii
        ../../gdb-13.2/gdb/remote.c:5020
0x55e67dfad668 _ZN13remote_target12start_remoteEii
        ../../gdb-13.2/gdb/remote.c:5072
0x55e67dfad668 _ZN13remote_target6open_1EPKcii
        ../../gdb-13.2/gdb/remote.c:5875
0x55e67e04d211 open_target
        ../../gdb-13.2/gdb/target.c:853
0x55e67dd1e0bf _Z8cmd_funcP16cmd_list_elementPKci
        ../../gdb-13.2/gdb/cli/cli-decode.c:2543
0x55e67e05cfcd _Z15execute_commandPKci
        ../../gdb-13.2/gdb/top.c:690
0x55e67defca94 catch_command_errors
        ../../gdb-13.2/gdb/main.c:513
0x55e67defcbb0 execute_cmdargs
        ../../gdb-13.2/gdb/main.c:608
0x55e67defe31b captured_main_1
        ../../gdb-13.2/gdb/main.c:1299
0x55e67defefca captured_main
        ../../gdb-13.2/gdb/main.c:1320
0x55e67defefca _Z8gdb_mainP18captured_main_args
        ../../gdb-13.2/gdb/main.c:1345
0x55e67dc2850a main
        ../../gdb-13.2/gdb/gdb.c:32
---------------------
A fatal error internal to GDB has been detected, further
debugging is not possible.  GDB will now terminate.

This is a bug, please report it.  For instructions, see:
<https://www.gnu.org/software/gdb/bugs/>.

Remote side has terminated connection.  GDBserver will reopen the connection.
Listening on port 5555
Aborted (core dumped)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[rajesh.palla at amd dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

Palla, Rajesh <rajesh.palla at amd dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rajesh.palla at amd dot com

--- Comment #1 from Palla, Rajesh <rajesh.palla at amd dot com> ---
I too have similar issue with gdb12.x version. Attaching the gdb remote stub
log

GDB -> $QNonStop:1#8d
GDB <- $OK#9a
GDB -> $qfThreadInfo#bb
GDB <- $mp1.1,p1.2#9a
GDB -> $qsThreadInfo#c8
GDB <- $l#6c
GDB -> $qAttached:1#fa
GDB <- $1#31
GDB -> $qXfer:exec-file:read:1:0,1000#4a
GDB <- $E01#a6
GDB -> $qXfer:exec-file:read:1:0,1000#4a
GDB <- $E01#a6
GDB -> $qTStatus#49
GDB <- $T0#84
GDB -> $qTfV#81
GDB <- $#00
GDB -> $?#3f
GDB <- $T05thread:p1.1;#a6
GDB -> $vStopped#55
GDB <- $T05thread:p1.2;#a7
GDB -> $vStopped#55
GDB <- $OK#9a

remote.c:8334: internal-error: wait: Assertion `!async_event_handler_marked
(rs->remote_async_inferior_event_token)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tromey at sourceware dot org
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2023-08-23
             Status|UNCONFIRMED                 |NEW

--- Comment #2 from Tom Tromey <tromey at sourceware dot org> ---
I was able to reproduce using the OP approach.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[Mikhail.Terekhov at dell dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

Mikhail Terekhov <Mikhail.Terekhov at dell dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |Mikhail.Terekhov at dell dot com

--- Comment #3 from Mikhail Terekhov <Mikhail.Terekhov at dell dot com> ---
I've got the same error in GDB 12.1 in non-stop async mode.
Here is the last several lines of remote log:

[remote] Sending packet: $vStopped#55
[remote] Packet received: T00thread:177;
[remote] Sending packet: $vStopped#55
[remote] Packet received: T00thread:178;
[remote] Sending packet: $vStopped#55
[remote] Packet received: OK
[remote] wait: enter
[remote] wait: exit
[remote] wait: enter
../../gdb/remote.c:8334: internal-error: wait: Assertion
`!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[Mikhail.Terekhov at dell dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

--- Comment #4 from Mikhail Terekhov <Mikhail.Terekhov at dell dot com> ---
BTW reverting commit 32b1f5e8d6b8ddd3be6e471c26dd85a1dac31dda fixes the issue.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[Mikhail.Terekhov at dell dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

--- Comment #5 from Mikhail Terekhov <Mikhail.Terekhov at dell dot com> ---
From `git log --grep="assert in remote_target"`

This assert was introduced in commit 32b1f5e8d6b8ddd3be6e471c26dd85a1dac31dda
Then it was reverted in commit 7719facb311657f8927c12481ccfc22fe84d5589
Then it was committed again in 8d34471f70a84e6db89be40116d1ed0f6af941c0

From GDB BUG #28627 it seems that the last commit was a mistake.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[christian.prochaska@genode-labs.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

--- Comment #6 from Christian Prochaska <christian.prochaska@genode-labs.com> ---
Replacing 'target_can_async_p()' by 'target_is_async_p()' in
'remote_target::queued_stop_reply()' solved the problem for me, but I'm not
sure if it is the correct fix. I just tried it when I saw that other calls of
'mark_async_event_handler()' have a 'target_is_async_p()' check.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[Mikhail.Terekhov at dell dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

Mikhail Terekhov <Mikhail.Terekhov at dell dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aburgess at redhat dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[Mikhail.Terekhov at dell dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

--- Comment #7 from Mikhail Terekhov <Mikhail.Terekhov at dell dot com> ---
This change eliminated assertion error for me as well.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[Mikhail.Terekhov at dell dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

--- Comment #8 from Mikhail Terekhov <Mikhail.Terekhov at dell dot com> ---
Created attachment 15151
  --> https://sourceware.org/bugzilla/attachment.cgi?id=15151&action=edit
Patch suggested in comment6

This change suggested by Christian fixes issue for me.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[simon.marchi at polymtl dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

Simon Marchi <simon.marchi at polymtl dot ca> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |simon.marchi at polymtl dot ca

--- Comment #9 from Simon Marchi <simon.marchi at polymtl dot ca> ---
Thanks for digging into the problem.  Can you please send your patch to the
gdb-patches mailing list?  Reviews happen there.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/30630] remote.c:8352: internal-error: wait: Assertion `!async_event_handler_marked (rs->remote_async_inferior_event_token)' failed.

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30630

--- Comment #10 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Simon Marchi <simark@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=635b2dd919b8c58e164b77c396041935fca1d66a

commit 635b2dd919b8c58e164b77c396041935fca1d66a
Author: Simon Marchi <simon.marchi@efficios.com>
Date:   Tue Oct 3 22:04:01 2023 -0400

    gdb: add assertion when marking the remote async flag

    As reported in bug 30630 [1], we hit a case where the remote target's
    async flag is marked while the target is not configured (yet) to work
    async.  This should not happen.  It is caught thanks to this assert in
    remote_target::wait:

        /* Start by clearing the flag that asks for our wait method to be
called,
           we'll mark it again at the end if needed.  If the target is not in
           async mode then the async token should not be marked.  */
        if (target_is_async_p ())
          rs->clear_async_event_handler ();
        else
          gdb_assert (!rs->async_event_handler_marked ());

    This is helpful, but I think that we could have caught the problem earlier
than
    that, at the moment we marked the handler.  Catching problems earlier
    makes them easier to debug.

    [1] https://sourceware.org/bugzilla/show_bug.cgi?id=30630

    Change-Id: I7e229c74b04da82bef6a817d5a676be5cf52e833
    Approved-By: Andrew Burgess <aburgess@redhat.com>

-- 
You are receiving this mail because:
You are on the CC list for the bug.