Re: [FYI/pushed v4 03/25] Step over clone syscall w/ breakpoint, TARGET_WAITKIND_THREAD_CLONED

[Guinevere Larsen <blarsen@redhat.com>]

On 13/11/2023 16:04, Pedro Alves wrote:
> (A good chunk of the problem statement in the commit log below is
> Andrew's, adjusted for a different solution, and for covering
> displaced stepping too.  The testcase is mostly Andrew's too.)
>
> This commit addresses bugs gdb/19675 and gdb/27830, which are about
> stepping over a breakpoint set at a clone syscall instruction, one is
> about displaced stepping, and the other about in-line stepping.
>
> Currently, when a new thread is created through a clone syscall, GDB
> sets the new thread running.  With 'continue' this makes sense
> (assuming no schedlock):
>
>   - all-stop mode, user issues 'continue', all threads are set running,
>     a newly created thread should also be set running.
>
>   - non-stop mode, user issues 'continue', other pre-existing threads
>     are not affected, but as the new thread is (sort-of) a child of the
>     thread the user asked to run, it makes sense that the new threads
>     should be created in the running state.
>
> Similarly, if we are stopped at the clone syscall, and there's no
> software breakpoint at this address, then the current behaviour is
> fine:
>
>   - all-stop mode, user issues 'stepi', stepping will be done in place
>     (as there's no breakpoint to step over).  While stepping the thread
>     of interest all the other threads will be allowed to continue.  A
>     newly created thread will be set running, and then stopped once the
>     thread of interest has completed its step.
>
>   - non-stop mode, user issues 'stepi', stepping will be done in place
>     (as there's no breakpoint to step over).  Other threads might be
>     running or stopped, but as with the continue case above, the new
>     thread will be created running.  The only possible issue here is
>     that the new thread will be left running after the initial thread
>     has completed its stepi.  The user would need to manually select
>     the thread and interrupt it, this might not be what the user
>     expects.  However, this is not something this commit tries to
>     change.
>
> The problem then is what happens when we try to step over a clone
> syscall if there is a breakpoint at the syscall address.
>
> - For both all-stop and non-stop modes, with in-line stepping:
>
>     + user issues 'stepi',
>     + [non-stop mode only] GDB stops all threads.  In all-stop mode all
>       threads are already stopped.
>     + GDB removes s/w breakpoint at syscall address,
>     + GDB single steps just the thread of interest, all other threads
>       are left stopped,
>     + New thread is created running,
>     + Initial thread completes its step,
>     + [non-stop mode only] GDB resumes all threads that it previously
>       stopped.
>
> There are two problems in the in-line stepping scenario above:
>
>    1. The new thread might pass through the same code that the initial
>       thread is in (i.e. the clone syscall code), in which case it will
>       fail to hit the breakpoint in clone as this was removed so the
>       first thread can single step,
>
>    2. The new thread might trigger some other stop event before the
>       initial thread reports its step completion.  If this happens we
>       end up triggering an assertion as GDB assumes that only the
>       thread being stepped should stop.  The assert looks like this:
>
>       infrun.c:5899: internal-error: int finish_step_over(execution_control_state*): Assertion `ecs->event_thread->control.trap_expected' failed.
>
> - For both all-stop and non-stop modes, with displaced stepping:
>
>     + user issues 'stepi',
>     + GDB starts the displaced step, moves thread's PC to the
>       out-of-line scratch pad, maybe adjusts registers,
>     + GDB single steps the thread of interest, [non-stop mode only] all
>       other threads are left as they were, either running or stopped.
>       In all-stop, all other threads are left stopped.
>     + New thread is created running,
>     + Initial thread completes its step, GDB re-adjusts its PC,
>       restores/releases scratchpad,
>     + [non-stop mode only] GDB resumes the thread, now past its
>       breakpoint.
>     + [all-stop mode only] GDB resumes all threads.
>
> There is one problem with the displaced stepping scenario above:
>
>    3. When the parent thread completed its step, GDB adjusted its PC,
>       but did not adjust the child's PC, thus that new child thread
>       will continue execution in the scratch pad, invoking undefined
>       behavior.  If you're lucky, you see a crash.  If unlucky, the
>       inferior gets silently corrupted.
>
> What is needed is for GDB to have more control over whether the new
> thread is created running or not.  Issue #1 above requires that the
> new thread not be allowed to run until the breakpoint has been
> reinserted.  The only way to guarantee this is if the new thread is
> held in a stopped state until the single step has completed.  Issue #3
> above requires that GDB is informed of when a thread clones itself,
> and of what is the child's ptid, so that GDB can fixup both the parent
> and the child.
>
> When looking for solutions to this problem I considered how GDB
> handles fork/vfork as these have some of the same issues.  The main
> difference between fork/vfork and clone is that the clone events are
> not reported back to core GDB.  Instead, the clone event is handled
> automatically in the target code and the child thread is immediately
> set running.
>
> Note we have support for requesting thread creation events out of the
> target (TARGET_WAITKIND_THREAD_CREATED).  However, those are reported
> for the new/child thread.  That would be sufficient to address in-line
> stepping (issue #1), but not for displaced-stepping (issue #3).  To
> handle displaced-stepping, we need an event that is reported to the
> _parent_ of the clone, as the information about the displaced step is
> associated with the clone parent.  TARGET_WAITKIND_THREAD_CREATED
> includes no indication of which thread is the parent that spawned the
> new child.  In fact, for some targets, like e.g., Windows, it would be
> impossible to know which thread that was, as thread creation there
> doesn't work by "cloning".
>
> The solution implemented here is to model clone on fork/vfork, and
> introduce a new TARGET_WAITKIND_THREAD_CLONED event.  This event is
> similar to TARGET_WAITKIND_FORKED and TARGET_WAITKIND_VFORKED, except
> that we end up with a new thread in the same process, instead of a new
> thread of a new process.  Like FORKED and VFORKED, THREAD_CLONED
> waitstatuses have a child_ptid property, and the child is held stopped
> until GDB explicitly resumes it.  This addresses the in-line stepping
> case (issues #1 and #2).
>
> The infrun code that handles displaced stepping fixup for the child
> after a fork/vfork event is thus reused for THREAD_CLONE, with some
> minimal conditions added, addressing the displaced stepping case
> (issue #3).
>
> The native Linux backend is adjusted to unconditionally report
> TARGET_WAITKIND_THREAD_CLONED events to the core.
>
> Following the follow_fork model in core GDB, we introduce a
> target_follow_clone target method, which is responsible for making the
> new clone child visible to the rest of GDB.
>
> Subsequent patches will add clone events support to the remote
> protocol and gdbserver.
>
> displaced_step_in_progress_thread becomes unused with this patch, but
> a new use will reappear later in the series.  To avoid deleting it and
> readding it back, this patch marks it with attribute unused, and the
> latter patch removes the attribute again.  We need to do this because
> the function is static, and with no callers, the compiler would warn,
> (error with -Werror), breaking the build.
>
> This adds a new gdb.threads/stepi-over-clone.exp testcase, which
> exercises stepping over a clone syscall, with displaced stepping vs
> inline stepping, and all-stop vs non-stop.  We already test stepping
> over clone syscalls with gdb.base/step-over-syscall.exp, but this test
> uses pthreads, while the other test uses raw clone, and this one is
> more thorough.  The testcase passes on native GNU/Linux, but fails
> against GDBserver.  GDBserver will be fixed by a later patch in the
> series.
>
> Co-authored-by: Andrew Burgess <aburgess@redhat.com>
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=19675
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=27830
> Change-Id: I95c06024736384ae8542a67ed9fdf6534c325c8e
> Reviewed-By: Andrew Burgess <aburgess@redhat.com>
> ---
>   gdb/infrun.c                                  | 158 +++----
>   gdb/linux-nat.c                               | 252 +++++------
>   gdb/linux-nat.h                               |   2 +
>   gdb/target-delegates.c                        |  24 ++
>   gdb/target.c                                  |   7 +
>   gdb/target.h                                  |   7 +
>   gdb/target/waitstatus.c                       |   1 +
>   gdb/target/waitstatus.h                       |  31 +-
>   gdb/testsuite/gdb.threads/stepi-over-clone.c  |  90 ++++
>   .../gdb.threads/stepi-over-clone.exp          | 395 ++++++++++++++++++
>   10 files changed, 775 insertions(+), 192 deletions(-)
>   create mode 100644 gdb/testsuite/gdb.threads/stepi-over-clone.c
>   create mode 100644 gdb/testsuite/gdb.threads/stepi-over-clone.exp
>
The test introduced by this patch does not work with clang, it adds 110 
fails, and almost all successes seem to be incorrect. I left inlined 
thoughts on the incorrect successes, and the fails are pretty self 
evident based on those thoughts, if they are correct. Unfortunately, 
something is up with my local setup and I can't run the test locally to 
double check, I'm stuck at looking at the buildbot log.

For a full log, check: 
https://builder.sourceware.org/testrun/9907246dce7004c32f7dabc8bbce8e5c9788b7ca?dgexpfile=gdb.threads%2Fstepi-over-clone.exp

> diff --git a/gdb/testsuite/gdb.threads/stepi-over-clone.exp b/gdb/testsuite/gdb.threads/stepi-over-clone.exp
> new file mode 100644
> index 00000000000..e580f2248ac
> --- /dev/null
> +++ b/gdb/testsuite/gdb.threads/stepi-over-clone.exp
> @@ -0,0 +1,395 @@
> +# Copyright 2021-2023 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +# Test performing a 'stepi' over a clone syscall instruction.
> +
> +# This test relies on us being able to spot syscall instructions in
> +# disassembly output.  For now this is only implemented for x86-64.
> +require {istarget x86_64-*-*}
> +
> +# Test only on native targets, for now.
> +proc is_native_target {} {
> +    return [expr {[target_info gdb_protocol] == ""}]
> +}
> +require is_native_target
> +
> +standard_testfile
> +
> +if { [prepare_for_testing "failed to prepare" $testfile $srcfile \
> +	  {debug pthreads additional_flags=-static}] } {
> +    return
> +}
> +
> +if {![runto_main]} {
> +    return
> +}
> +
> +# Arrange to catch the 'clone' syscall, run until we catch the
> +# syscall, and try to figure out the address of the actual syscall
> +# instruction so we can place a breakpoint at this address.
> +
> +gdb_test_multiple "catch syscall group:process" "catch process syscalls" {
> +    -re "The feature \'catch syscall\' is not supported.*\r\n$gdb_prompt $" {
> +	unsupported $gdb_test_name
> +	return
> +    }
> +    -re ".*$gdb_prompt $" {
> +	pass $gdb_test_name
> +    }
> +}
In the clang buildbot we're getting this output:

(gdb) catch syscall group:process
warning: Can not parse XML syscalls information; XML support was 
disabled at compile time.
Unknown syscall group 'process'.
(gdb) PASS: gdb.threads/stepi-over-clone.exp: catch process syscalls

This should be a failure, and should likely skip the next few tests.

I don't know

> +
> +gdb_test "continue" \
> +    "Catchpoint $decimal \\(call to syscall clone\[23\]\\), .*"
> +
> +# Return true if INSN is a syscall instruction.
> +
> +proc is_syscall_insn { insn } {
> +    if [istarget x86_64-*-* ] {
> +	return { $insn == "syscall" }
> +    } else {
> +	error "port me"
> +    }
> +}
> +
> +# A list of addresses with syscall instructions.
> +set syscall_addrs {}
> +
> +# Get list of addresses with syscall instructions.
> +gdb_test_multiple "disassemble" "" {
> +    -re "Dump of assembler code for function \[^\r\n\]+:\r\n" {
> +	exp_continue
> +    }
> +    -re "^(?:=>)?\\s+(${hex})\\s+<\\+${decimal}>:\\s+(\[^\r\n\]+)\r\n" {
> +	set addr $expect_out(1,string)
> +	set insn [string trim $expect_out(2,string)]
> +	if [is_syscall_insn $insn] {
> +	    verbose -log "Found a syscall at: $addr"
> +	    lappend syscall_addrs $addr
> +	}
> +	exp_continue
> +    }
> +    -re "^End of assembler dump\\.\r\n$gdb_prompt $" {
> +	if { [llength $syscall_addrs] == 0 } {
> +	    unsupported "no syscalls found"
> +	    return -1
> +	}
> +    }
> +}
> +
> +# The test proc.  NON_STOP and DISPLACED are either 'on' or 'off', and are
> +# used to configure how GDB starts up.  THIRD_THREAD is either true or false,
> +# and is used to configure the inferior.
> +proc test {non_stop displaced third_thread} {
> +    global binfile srcfile
> +    global syscall_addrs
> +    global GDBFLAGS
> +    global gdb_prompt hex decimal
> +
> +    for { set i 0 } { $i < 3 } { incr i } {
> +	with_test_prefix "i=$i" {
> +
> +	    # Arrange to start GDB in the correct mode.
> +	    save_vars { GDBFLAGS } {
> +		append GDBFLAGS " -ex \"set non-stop $non_stop\""
> +		append GDBFLAGS " -ex \"set displaced $displaced\""
> +		clean_restart $binfile
> +	    }
> +
> +	    runto_main
> +
> +	    # Setup breakpoints at all the syscall instructions we
> +	    # might hit.  Only issue one pass/fail to make tests more
> +	    # comparable between systems.
> +	    set test "break at syscall insns"
> +	    foreach addr $syscall_addrs {
> +		if {[gdb_test -nopass "break *$addr" \
> +			 ".*" \
> +			 $test] != 0} {
> +		    return
> +		}
> +	    }

I may be mis-interpreting this part of the test and log, but it seems 
that we set no breakpoints at all when testing in the buildbot, which 
seems to be the reason for all the failures. This is what the log says 
in this part:

(gdb) run
Starting program: 
/home/buildbot/buildbot/binutils-gdb-clang-fedrawhide-x86_64/build/gdb/testsuite/outputs/gdb.threads/stepi-over-clone/stepi-over-clone 

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, main () at 
/home/buildbot/buildbot/binutils-gdb-clang-fedrawhide-x86_64/build/gdb/testsuite/gdb.threads/stepi-over-clone.c:80
80      if (getenv ("MAKE_EXTRA_THREAD") != NULL)
(gdb) PASS: gdb.threads/stepi-over-clone.exp: third_thread=false: 
non-stop=on: displaced=off: i=0: break at syscall insns
continue
Continuing.
[New Thread 0x7ffff7ff86c0 (LWP 1302753)]
Hello from the first thread.
FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: 
displaced=off: i=0: continue (timeout)
stepi
FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: 
displaced=off: i=0: stepi (timeout)
XXX: Have completed scanning the 'stepi' output
p 1 + 2 + 3


> +	    # If we got here, all breakpoints were set successfully.
> +	    # We used -nopass above, so issue a pass now.
> +	    pass $test
> +
> +	    # Continue until we hit the syscall.
> +	    gdb_test "continue"
> +
> +	    if { $third_thread } {
> +		gdb_test_no_output "set start_third_thread=1"
> +	    }
> +
> +	    set stepi_error_count 0
> +	    set stepi_new_thread_count 0
> +	    set thread_1_stopped false
> +	    set thread_2_stopped false
> +	    set seen_prompt false
> +	    set hello_first_thread false
> +
> +	    # The program is now stopped at main, but if testing
> +	    # against GDBserver, inferior_spawn_id is GDBserver's
> +	    # spawn_id, and the GDBserver output emitted before the
> +	    # program stopped isn't flushed unless we explicitly do
> +	    # so, because it is on a different spawn_id.  We could try
> +	    # flushing it now, to avoid confusing the following tests,
> +	    # but that would have to be done under a timeout, and
> +	    # would thus slow down the testcase.  Instead, if inferior
> +	    # output goes to a different spawn id, then we don't need
> +	    # to wait for the first message from the inferior with an
> +	    # anchor, as we know consuming inferior output won't
> +	    # consume GDB output.  OTOH, if inferior output is coming
> +	    # out on GDB's terminal, then we must use an anchor,
> +	    # otherwise matching inferior output without one could
> +	    # consume GDB output that we are waiting for in regular
> +	    # expressions that are written after the inferior output
> +	    # regular expression match.
> +	    if {$::inferior_spawn_id != $::gdb_spawn_id} {
> +		set anchor ""
> +	    } else {
> +		set anchor "^"
> +	    }
> +
> +	    gdb_test_multiple "stepi" "" {
> +		-re "^stepi\r\n" {
> +		    verbose -log "XXX: Consume the initial command"
> +		    exp_continue
> +		}
> +		-re "^\\\[New Thread\[^\r\n\]+\\\]\r\n" {
> +		    verbose -log "XXX: Consume new thread line"
> +		    incr stepi_new_thread_count
> +		    exp_continue
> +		}
> +		-re "^\\\[Switching to Thread\[^\r\n\]+\\\]\r\n" {
> +		    verbose -log "XXX: Consume switching to thread line"
> +		    exp_continue
> +		}
> +		-re "^\\s*\r\n" {
> +		    verbose -log "XXX: Consume blank line"
> +		    exp_continue
> +		}
> +
> +		-i $::inferior_spawn_id
> +
> +		-re "${anchor}Hello from the first thread\\.\r\n" {
> +		    set hello_first_thread true
> +
> +		    verbose -log "XXX: Consume first worker thread message"
> +		    if { $third_thread } {
> +			# If we are going to start a third thread then GDB
> +			# should hit the breakpoint in clone before printing
> +			# this message.
> +			incr stepi_error_count
> +		    }
> +		    if { !$seen_prompt } {
> +			exp_continue
> +		    }
> +		}
> +		-re "^Hello from the third thread\\.\r\n" {
> +		    # We should never see this message.
> +		    verbose -log "XXX: Consume third worker thread message"
> +		    incr stepi_error_count
> +		    if { !$seen_prompt } {
> +			exp_continue
> +		    }
> +		}
> +
> +		-i $::gdb_spawn_id
> +
> +		-re "^$hex in clone\[23\]? \\(\\)\r\n" {
> +		    verbose -log "XXX: Consume stop location line"
> +		    set thread_1_stopped true
> +		    if { !$seen_prompt } {
> +			verbose -log "XXX: Continuing to look for the prompt"
> +			exp_continue
> +		    }
> +		}
> +		-re "^$gdb_prompt " {
> +		    verbose -log "XXX: Consume the final prompt"
> +		    gdb_assert { $stepi_error_count == 0 }
> +		    gdb_assert { $stepi_new_thread_count == 1 }
> +		    set seen_prompt true
> +		    if { $third_thread } {
> +			if { $non_stop } {
> +			    # In non-stop mode if we are trying to start a
> +			    # third thread (from the second thread), then the
> +			    # second thread should hit the breakpoint in clone
> +			    # before actually starting the third thread.  And
> +			    # so, at this point both thread 1, and thread 2
> +			    # should now be stopped.
> +			    if { !$thread_1_stopped || !$thread_2_stopped } {
> +				verbose -log "XXX: Continue looking for an additional stop event"
> +				exp_continue
> +			    }
> +			} else {
> +			    # All stop mode.  Something should have stoppped
> +			    # by now otherwise we shouldn't have a prompt, but
> +			    # we can't know which thread will have stopped as
> +			    # that is a race condition.
> +			    gdb_assert { $thread_1_stopped || $thread_2_stopped }
> +			}
> +		    }
> +
> +		    if {$non_stop && !$hello_first_thread} {
> +			exp_continue
> +		    }
> +
> +		}
> +		-re "^Thread 2\[^\r\n\]+ hit Breakpoint $decimal, $hex in clone\[23\]? \\(\\)\r\n" {
> +		    verbose -log "XXX: Consume thread 2 hit breakpoint"
> +		    set thread_2_stopped true
> +		    if { !$seen_prompt } {
> +			verbose -log "XXX: Continuing to look for the prompt"
> +			exp_continue
> +		    }
> +		}
> +		-re "^PC register is not available\r\n" {
> +		    # This is the error we'd see for remote targets.
> +		    verbose -log "XXX: Consume error line"
> +		    incr stepi_error_count
> +		    exp_continue
> +		}
> +		-re "^Couldn't get registers: No such process\\.\r\n" {
> +		    # This is the error we see'd for native linux
> +		    # targets.
> +		    verbose -log "XXX: Consume error line"
> +		    incr stepi_error_count
> +		    exp_continue
> +		}
> +	    }
> +
> +	    # Ensure we are back at a GDB prompt, resynchronise.
> +	    verbose -log "XXX: Have completed scanning the 'stepi' output"
> +	    gdb_test "p 1 + 2 + 3" " = 6"
> +
> +	    # Check the number of threads we have, it should be exactly two.
> +	    set thread_count 0
> +	    set bad_threads 0
> +
> +	    # Build up our expectations for what the current thread state
> +	    # should be.  Thread 1 is the easiest, this is the thread we are
> +	    # stepping, so this thread should always be stopped, and should
> +	    # always still be in clone.
> +	    set match_code {}
> +	    lappend match_code {
> +		-re "\\*?\\s+1\\s+Thread\[^\r\n\]+clone\[23\]? \\(\\)\r\n" {
> +		    incr thread_count
> +		    exp_continue
> +		}
> +	    }
> +
> +	    # What state should thread 2 be in?
> +	    if { $non_stop == "on" } {
> +		if { $third_thread } {
> +		    # With non-stop mode on, and creation of a third thread
> +		    # having been requested, we expect Thread 2 to exist, and
> +		    # be stopped at the breakpoint in clone (just before the
> +		    # third thread is actually created).
> +		    lappend match_code {
> +			-re "\\*?\\s+2\\s+Thread\[^\r\n\]+$hex in clone\[23\]? \\(\\)\r\n" {
> +			    incr thread_count
> +			    exp_continue
> +			}
> +			-re "\\*?\\s+2\\s+Thread\[^\r\n\]+\\(running\\)\r\n" {
> +			    incr thread_count
> +			    incr bad_threads
> +			    exp_continue
> +			}
> +			-re "\\*?\\s+2\\s+Thread\[^\r\n\]+\r\n" {
> +			    verbose -log "XXX: thread 2 is bad, unknown state"
> +			    incr thread_count
> +			    incr bad_threads
> +			    exp_continue
> +			}
> +		    }
> +
> +		} else {
> +		    # With non-stop mode on, and no third thread having been
> +		    # requested, then we expect Thread 2 to exist, and still
> +		    # be running.
> +		    lappend match_code {
> +			-re "\\*?\\s+2\\s+Thread\[^\r\n\]+\\(running\\)\r\n" {
> +			    incr thread_count
> +			    exp_continue
> +			}
> +			-re "\\*?\\s+2\\s+Thread\[^\r\n\]+\r\n" {
> +			    verbose -log "XXX: thread 2 is bad, unknown state"
> +			    incr thread_count
> +			    incr bad_threads
> +			    exp_continue
> +			}
> +		    }
> +		}
> +	    } else {
> +		# With non-stop mode off then we expect Thread 2 to exist, and
> +		# be stopped.  We don't have any guarantee about where the
> +		# thread will have stopped though, so we need to be vague.
> +		lappend match_code {
> +		    -re "\\*?\\s+2\\s+Thread\[^\r\n\]+\\(running\\)\r\n" {
> +			verbose -log "XXX: thread 2 is bad, unexpectedly running"
> +			incr thread_count
> +			incr bad_threads
> +			exp_continue
> +		    }
> +		    -re "\\*?\\s+2\\s+Thread\[^\r\n\]+_start\[^\r\n\]+\r\n" {
> +			# We know that the thread shouldn't be stopped
> +			# at _start, though.  This is the location of
> +			# the scratch pad on Linux at the time of
> +			# writting.
> +			verbose -log "XXX: thread 2 is bad, stuck in scratchpad"
> +			incr thread_count
> +			incr bad_threads
> +			exp_continue
> +		    }
> +		    -re "\\*?\\s+2\\s+Thread\[^\r\n\]+\r\n" {
> +			incr thread_count
> +			exp_continue
> +		    }
> +		}
> +	    }
> +
> +	    # We don't expect to ever see a thread 3.  Even when we are
> +	    # requesting that this third thread be created, thread 2, the
> +	    # thread that creates thread 3, should stop before executing the
> +	    # clone syscall.  So, if we do ever see this then something has
> +	    # gone wrong.
> +	    lappend match_code {
> +		-re "\\s+3\\s+Thread\[^\r\n\]+\r\n" {
> +		    incr thread_count
> +		    incr bad_threads
> +		    exp_continue
> +		}
> +	    }
> +
> +	    lappend match_code {
> +		-re "$gdb_prompt $" {
> +		    gdb_assert { $thread_count == 2 }
> +		    gdb_assert { $bad_threads == 0 }
> +		}
> +	    }
> +
> +	    set match_code [join $match_code]
> +	    gdb_test_multiple "info threads" "" $match_code
> +	}
> +    }
> +}
> +
> +# Run the test in all suitable configurations.
> +foreach_with_prefix third_thread { false true } {
> +    foreach_with_prefix non-stop { "on" "off" } {
> +	foreach_with_prefix displaced { "off" "on" } {
> +	    test ${non-stop} ${displaced} ${third_thread}
> +	}
> +    }
> +}


-- 
Cheers,
Guinevere Larsen
She/Her/Hers