[Bug gdb/27907] New: Breakpoints set by label stop at the wrong line.

[Bug gdb/27907] New: Breakpoints set by label stop at the wrong line.

[skip.montanaro at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

            Bug ID: 27907
           Summary: Breakpoints set by label stop at the wrong line.
           Product: gdb
           Version: 9.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: gdb
          Assignee: unassigned at sourceware dot org
          Reporter: skip.montanaro at gmail dot com
  Target Milestone: ---

I am running on XUbuntu 20.04 with GCC and GDB at these versions:

  % gcc --version
  gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
  Copyright (C) 2019 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

  % gdb --version
  GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
  Copyright (C) 2020 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

I've been working on the Python virtual machine, developing new register-based
opcodes. Python's VM is implemented as a big switch statement with computed
gotos by default where supported by the compiler. Each opcode is implemented as
a case in that switch statement. Here's the LOAD_CONST instruction's code:

        case TARGET(LOAD_CONST): {
            PREDICTED(LOAD_CONST);
            PyObject *value = GETITEM(consts, oparg);
            Py_INCREF(value);
            PUSH(value);
            DISPATCH();
        }

The TARGET macro expands to a simple label:

#define TARGET(op) op: TARGET_##op

The DISPATCH macro is a bit more interesting:

#define DISPATCH() \
    { \
        if (trace_info.cframe.use_tracing OR_DTRACE_LINE OR_LLTRACE) { \
            goto tracing_dispatch; \
        } \
        f->f_lasti = INSTR_OFFSET(); \
        NEXTOPARG(); \
        DISPATCH_GOTO(); \
    }

and finally, the DISPATCH_GOTO macro expands to a computed goto:

#define DISPATCH_GOTO() goto *opcode_targets[opcode]

For the purposes of my work, I would find it handy to set breakpoints via
labels. That would allow me to save breakpoints and their commands to a file I
could source in different GDB sessions without having to worry about line
number changes. For instance, if I set a breakpoint at the LOAD_FAST
implementation it would look like this:

break ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST

When evaluated, I get this:

  (gdb) b ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST
  Breakpoint 1 at 0x116a36: file Python/ceval.c, line 1836.

and in fact, line 1836 is the start of the implementation of LOAD_CONST. Having
set that breakpoint, then run, I get an incorrect break, in this case at line
1833, which is in the midst of the previous instruction in the file, LOAD_FAST:

  (gdb) r
  Starting program: /home/skip/src/python/rvm/python 
  warning: the debug information found in "/lib64/ld-2.31.so" does not match
"/lib64/ld-linux-x86-64.so.2" (CRC mismatch).
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

  Breakpoint 1, 0x000055555566aa36 in _PyEval_EvalFrameDefault
(tstate=0x55555597d7e0, f=Frame 0x7ffff6c12140, for file <frozen
importlib._bootstrap>, line 23, in <module> (), throwflag=0) at
Python/ceval.c:1833
  1833              DISPATCH();
  (gdb)

I have tried everything I can think of (or that others in the Python dev
community have suggested), all to no avail. -g, -ggdb, -O0, -Og,
-fno-omit-frame-pointer. 

To reproduce (on Linux systems):

1. Clone https://github.com/python/cpython
2. Build an interpreter using this command:

    ./configure ; make clean ; make -j

   By default, Python/ceval.c will be compiled with a gcc command like this:

    gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
-Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter
-Wno-missing-field-initializers -Werror=implicit-function-declaration
-fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE
-o Python/ceval.o Python/ceval.c

  You can tweak the optimization flags by setting the OPT environment variable,
for example:

  export OPT='-ggdb -O0 -Wall'
  ./configure ; make clean ; make -j

3. Run gdb against the generated executable.
4. Set a breakpoint at the case for the LOAD_CONST instruction:

  b ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST

5. Run the executable and note how the line at which execution stops is a few
lines before the desired label:

  (gdb) b ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST
  Breakpoint 1 at 0x116a36: file Python/ceval.c, line 1836.
  (gdb) r
  Starting program: /home/skip/src/python/rvm/python 
  warning: the debug information found in "/lib64/ld-2.31.so" does not match
"/lib64/ld-linux-x86-64.so.2" (CRC mismatch).
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

  Breakpoint 1, 0x000055555566aa36 in _PyEval_EvalFrameDefault
(tstate=0x55555597d7e0, f=Frame 0x7ffff6c12140, for file <frozen
importlib._bootstrap>, line 23, in <module> (), throwflag=0) at
Python/ceval.c:1833
  1833              DISPATCH();
  (gdb) 

I don't know if this is a problem with GCC or GDB, but I lean toward it being a
GDB problem. It clearly announces the correct line number when the break
command is executed, but breaks at the wrong line.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[keiths at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

Keith Seitz <keiths at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |keiths at redhat dot com

--- Comment #1 from Keith Seitz <keiths at redhat dot com> ---
I have tried this on a variety of compilers:

1) Fedora 34 / system compiler (GCC 11.1.1)
2) RHEL8 / system compiler (GCC 8.4.1)
3) RHEL8 / GTS 9 (GCC 9.2.1)
4) RHEL8 / GTS 10 (GCC 10.2.1)

I cannot reproduce this with either gdb-10.1 or gdb-9.2 (FSF versions):

$ /path/to/gdb-9.2/gdb -q \
  -ex "b ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST" \
  -ex r -ex list python
Reading symbols from python...
Breakpoint 1 at 0x4cd783: file Python/ceval.c, line 1836.
Starting program: /home/keiths/tmp/gdb27907/cpython/python 
warning: Expected absolute pathname for libpthread in the inferior, but got
.gnu_debugdata for /lib64/libpthread.so.0.
warning: Unable to find libthread_db matching inferior's thread library, thread
debugging will not be available.

Breakpoint 1, _PyEval_EvalFrameDefault (tstate=0x7d08a0, f=Frame
0x7fffea65e140, for file <frozen importlib._bootstrap>, line 1, in <module> (),
throwflag=0) at Python/ceval.c:1838
1838                PyObject *value = GETITEM(consts, oparg);
1833                DISPATCH();
1834            }
1835    
1836            case TARGET(LOAD_CONST): {
1837                PREDICTED(LOAD_CONST);
1838                PyObject *value = GETITEM(consts, oparg);
1839                Py_INCREF(value);
1840                PUSH(value);
1841                DISPATCH();
1842            }
(gdb) 

While this might still be a GDB bug, it is difficult to know without being able
to reproduce it.

In any case, it might be worth a try to compile the master branch. We often
have patches to workaround compiler issues.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[skip.montanaro at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

--- Comment #2 from Skip Montanaro <skip.montanaro at gmail dot com> ---
Thanks for trying. What's configure/compile commands did you use?

Master branch of GDB? GCC?

I've not made changes to either GDB or GCC. I'm happy to try building GDB. I
will report back.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[keiths at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

--- Comment #3 from Keith Seitz <keiths at redhat dot com> ---
(In reply to Skip Montanaro from comment #2)
> Thanks for trying. What's configure/compile commands did you use?
> 
> Master branch of GDB? GCC?
> 

I'm talking only about building GDB. For compilers, I've just been using
my distro's versions.

For GDB, configure flags shouldn't really change much, but for the
record, I use:

$ flags="-g3 -Og" CFLAGS="$flags" CXXFLAGS="$flags" ../src/configure \
  --disable-binutils --disable-elfcpp --disable-gas --disable-gold \
  --disable-gprof --disable-ld --enable-unit-tests --with-system-readline \
  --with-separate-debug-dir=/usr/lib/debug --enable-targets=all \
  --with-lzma --prefix=/usr

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[skip.montanaro at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

--- Comment #4 from Skip Montanaro <skip.montanaro at gmail dot com> ---
I must not be looking at the right code. I cloned this repo:

git://sourceware.org/git/binutils-gdb.git

but that doesn't have a src directory. Sorry to ask such a basic question, but
what source code should I be using? I doubt I've built GDB in 20 years.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[keiths at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

--- Comment #5 from Keith Seitz <keiths at redhat dot com> ---
(In reply to Skip Montanaro from comment #4)
> I must not be looking at the right code. I cloned this repo:
> 
> git://sourceware.org/git/binutils-gdb.git
> 
> but that doesn't have a src directory. Sorry to ask such a basic question,
> but what source code should I be using? I doubt I've built GDB in 20 years.

You're good: s/src/binutils-gdb/ . I just name my repos "src" instead of the
default "binutils-gdb". So, I do:

$ git clone git://sourceware.org/git/binutils-gdb.git src
$ mkdir linux; cd linux
$ flags="-g3 -O0" CFLAGS="$flags" CXXFLAGS="$flags" ../src/configure ...
$ make all-gdb

By using the configure command I listed previously, this will by default
install into *system* directories, so don't do that. [I never do.] GDB will run
from the build directory just fine.

If you run into any issues, just let me/us know.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[skip.montanaro at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

--- Comment #6 from Skip Montanaro <skip.montanaro at gmail dot com> ---
After a bit of effort I got gdb built on the master branch. When using it to
examine the python executable, I get the same result as before:

  % ~/src/binutils-gdb/gdb/gdb ./python
  GNU gdb (GDB) 11.0.50.20210524-git
  Copyright (C) 2021 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.
  Type "show copying" and "show warranty" for details.
  This GDB was configured as "x86_64-pc-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <https://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
      <http://www.gnu.org/software/gdb/documentation/>.

  For help, type "help".
  Type "apropos word" to search for commands related to "word"...
  Reading symbols from ./python...
  (gdb) b ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST
  Breakpoint 1 at 0x116a36: file Python/ceval.c, line 1836.
  (gdb) r
  Starting program: /home/skip/src/python/rvm/python 
  warning: the debug information found in "/lib64/ld-2.31.so" does not match
"/lib64/ld-linux-x86-64.so.2" (CRC mismatch).

  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

  Breakpoint 1, 0x000055555566aa36 in _PyEval_EvalFrameDefault
(tstate=0x55555597d7e0, f=0x7ffff6c12140, throwflag=0) at Python/ceval.c:1833
  1833              DISPATCH();

More for grins than anything else, I tried setting breakpoints at the labels
for several other opcodes. In all cases the actual line number at which
execution stopped was three less than the line at which the label was actually
defined (and where GDB told me execution would stop).

Is the warning about a CRC mismatch in ld shared objects meaningful? Here's
what I see:

  % ls -l /lib64/ld-2.31.so 
  -rwxr-xr-x 1 root root 182168 Dec 16 05:04 /lib64/ld-2.31.so
  % ls -l /lib64/ld-linux-x86-64.so.2
  lrwxrwxrwx 1 root root 32 May 18 07:08 /lib64/ld-linux-x86-64.so.2 ->
/lib/x86_64-linux-gnu/ld-2.31.so
  % ls -Ll /lib64/ld-linux-x86-64.so.2
  -rwxr-xr-x 1 root root 191472 Dec 16 05:04 /lib64/ld-linux-x86-64.so.2

It's not too surprising the CRCs don't match. OTOH, neither the python nor gdb
executables are linked to /lib64/ld-2.31.so.

If the CRC diffs aren't meaningful, I'm not sure what else I can do. If there's
more I can do to dig into the reasons behind this problem, let me know.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[ssbssa at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

Hannes Domani <ssbssa at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ssbssa at sourceware dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[keiths at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

--- Comment #7 from Keith Seitz <keiths at redhat dot com> ---
(In reply to Skip Montanaro from comment #6)
I wouldn't expect those warnings to cause any problems.

> If the CRC diffs aren't meaningful, I'm not sure what else I can do. If
> there's more I can do to dig into the reasons behind this problem, let me
> know.

I'm downloading/installing Xubuntu 20.04 into a VM now. Let me see if I
can reproduce it with that environment.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[keiths at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

Keith Seitz <keiths at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|                            |2021-06-02
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1

--- Comment #8 from Keith Seitz <keiths at redhat dot com> ---
(In reply to Keith Seitz from comment #7)
> I'm downloading/installing Xubuntu 20.04 into a VM now. Let me see if I
> can reproduce it with that environment.

Yeah, that does reproduce it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/27907] Breakpoints set by label stop at the wrong line.

[skip.montanaro at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=27907

--- Comment #9 from Skip Montanaro <skip.montanaro at gmail dot com> ---
Good to know that I'm not going crazy. (Well, I still might be, but for other
reasons.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.