[Bug testsuite/31632] New: [gdb/testsuite] Connecting to wrong gdbserver during parallel testing

[Bug testsuite/31632] New: [gdb/testsuite] Connecting to wrong gdbserver during parallel testing

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31632

            Bug ID: 31632
           Summary: [gdb/testsuite] Connecting to wrong gdbserver during
                    parallel testing
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: testsuite
          Assignee: unassigned at sourceware dot org
          Reporter: vries at gcc dot gnu.org
  Target Milestone: ---

The opensuse gdb package runs the testsuite using:
...
$ make -j 16 \
  check//unix/-m64 \
  check//unix/-m64/-fPIE/-pie\
  check//unix/-m32 \
  check//unix/-m32/-fPIE/-pie
...
or alternatively with -fno-PIE/-no-pie, depending on the default.

So, AFAIU, four target boards are tested in parallel, and then each target
board is also tested in parallel.

I'm seeing (with gdb 13.2 based package) these FAILs for one target board:
...
FAIL: gdb.server/connect-without-multi-process.exp: multiprocess=off: target
remote (got interactive prompt)
FAIL: gdb.server/connect-without-multi-process.exp: multiprocess=off: continue
to main
FAIL: gdb.server/connect-without-multi-process.exp: multiprocess=off: continue
until exit
...
and these for another:
...
FAIL: gdb.server/reconnect-ctrl-c.exp: second: continue for ctrl-c (the program
is no longer running)
FAIL: gdb.server/reconnect-ctrl-c.exp: second: stop with control-c
...

Curiously, when investigating the first we see:
...
(gdb) PASS: gdb.server/connect-without-multi-process.exp: multiprocess=off:
break -q main
target remote localhost:2346^M
Remote debugging using localhost:2346^M
warning: Build ID mismatch between current exec-file
/home/abuild/rpmbuild/BUILD/gdb-13.2/build-x86_64-suse-linux/gdb/testsuite.unix.-m32.-fno-PIE.-no-pie/outputs/gdb.server/connect-without-multi-process/connect-without-multi-process^M
and automatically determined exec-file
/home/abuild/rpmbuild/BUILD/gdb-13.2/build-x86_64-suse-linux/gdb/testsuite.unix.-m32/outputs/gdb.server/reconnect-ctrl-c/reconnect-ctrl-c^M
exec-file-mismatch handling is currently "ask"^M
Load new symbol table from
"/home/abuild/rpmbuild/BUILD/gdb-13.2/build-x86_64-suse-linux/gdb/testsuite.unix.-m32/outputs/gdb.server/reconnect-ctrl-c/reconnect-ctrl-c"?
(y or n) n^M
warning: loading
/home/abuild/rpmbuild/BUILD/gdb-13.2/build-x86_64-suse-linux/gdb/testsuite.unix.-m32/outputs/gdb.server/reconnect-ctrl-c/reconnect-ctrl-c
Not confirmed.^M
warning: Could not load shared library symbols for linux-gate.so.1.^M
Do you need "set solib-search-path" or "set sysroot"?^M
Reading symbols from /lib/libc.so.6...^M
(No debugging symbols found in /lib/libc.so.6)^M
Reading symbols from /lib/ld-linux.so.2...^M
(No debugging symbols found in /lib/ld-linux.so.2)^M
0xf7fc7579 in ?? ()^M
(gdb) FAIL: gdb.server/connect-without-multi-process.exp: multiprocess=off:
target remote (got interactive prompt)
...

Looking at the other test-case, the same portnum is used.

It seems we're connecting to the wrong gdbserver.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/31632] [gdb/testsuite] Connecting to wrong gdbserver during parallel testing

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31632

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
I'm not sure exactly how this failure is caused, but I think making sure
test-cases run in parallel get unique portnums is a way to fix this.

The current approach is that we have:
...
global portnum
set portnum "2345"
...
and in gdbserver_start we do:
...
        # Bump the port number to avoid conflicts with hung ports.              
        incr portnum
...
which works ok for serial testing.

There there's this bit:
...
           -re "Can't (bind address|listen on socket): Address already in
use\\.\r\n" {
                verbose -log "Port $portnum is already in use."
                if ![target_info exists gdb,socketport] {
                    # Bump the port number to avoid the conflict.               
                    wait -i $expect_out(spawn_id)
                    incr portnum
                    continue
                }
            }
...
which should avoid clashes with other uses.

We can't avoid clashes completely, given that things may happen outside of the
scope of gdb testing, but we should avoid running into clashes due to parallel
gdb testing.

So, we're gonna need a unique dir that needs to be created/emptied when making
make-target check//%, and passed down using a variable say, GDB_LOCK_DIR.

Each make invocation can then check GDB_LOCK_DIR, if it doesn't exist (because
of not using make check//), create it's own and use it, and if it does, use
that one.

[ I then wonder what happens when using check// and plain check next to each
other.  But perhaps this is already unsupported/broken today. ]

Then we can track portnum in $GDB_LOCK_DIR/portnum and use
$GDB_LOCK_DIR/portnum.lock to serialize parallel access.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/31632] [gdb/testsuite] Connecting to wrong gdbserver during parallel testing

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31632

--- Comment #2 from Tom de Vries <vries at gcc dot gnu.org> ---
https://sourceware.org/pipermail/gdb-patches/2024-April/208117.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/31632] [gdb/testsuite] Connecting to wrong gdbserver during parallel testing

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31632

--- Comment #3 from Sourceware Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Tom de Vries <vries@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=c479e964a86984cbf3278845615c04ada8adfb50

commit c479e964a86984cbf3278845615c04ada8adfb50
Author: Tom de Vries <tdevries@suse.de>
Date:   Sat May 4 10:41:09 2024 +0200

    [gdb/testsuite] Use unique portnum in parallel testing (check//% case)

    Make target check//% is the gdb variant of a similar gcc make target [1].

    When running tests using check//%:
    ...
    $ cd build/gdb
    $ make check//unix/{-fPIE/-pie,-fno-PIE/-no-pie} -j2 TESTS=gdb.server/*.exp
    ...
    we get:
    ...
    $ cat build/gdb/testsuite.unix.-fPIE.-pie/cache/portnum
    2427
    $ cat build/gdb/testsuite.unix.-fno-PIE.-no-pie/cache/portnum
    2423
    ...

    The problem is that there are two portnum files used in parallel.

    Fix this by:
    - creating a common lockdir build/gdb/testsuite.lockdir for make target
      check//%,
    - passing this down to the runtests invocations using variable
GDB_LOCK_DIR,
      and
    - using GDB_LOCK_DIR in lock_dir.

    Tested on aarch64-linux.

    Approved-By: Tom Tromey <tom@tromey.com>

    PR testsuite/31632
    Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=31632

    [1] https://gcc.gnu.org/install/test.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/31632] [gdb/testsuite] Connecting to wrong gdbserver during parallel testing

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=31632

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |15.1
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #4 from Tom de Vries <vries at gcc dot gnu.org> ---
Fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.