[Bug breakpoints/10557] New: HW watchpoints silently degrade into SW ones

[Bug breakpoints/10557] New: HW watchpoints silently degrade into SW ones

[ppluzhnikov at google dot com]

I've set some hardware watchpoints, the rerun the program, and noticed
that it didn't go anywhere, with gdb consuming all the CPU.

It appears that under some conditions HW watchpoints "degenerate" into
single-stepping. To reproduce on Linux/x86_64:


--- cut --- t.c ---
#include <stdlib.h>

#ifndef LIMIT
#define LIMIT 100000
#endif

int main()
{
    int i;
    for (i = 0; i < LIMIT; ++i)
      {
	int *ip = malloc(sizeof(int));
	*ip = i;
	if (i == LIMIT - 1)
	  abort();
      }
}
--- cut --- t.c ---

Using "GNU gdb (GDB) 6.8.50.20090821-cvs", I see this:

gcc -g t.c -DLIMIT=1000 && time gdb64-cvs -q -ex run -ex 'up 2' -ex 'print ip'
-ex 'watch *$1' -ex 'run' -ex quit ./a.out

Program received signal SIGABRT, Aborted.
0x00007ffff7ab0095 in raise () from /lib/libc.so.6
#2  0x000000000040051e in main () at t.c:15
15                abort();
$1 = (int *) 0x608cf0
Hardware watchpoint 1: *$1
Hardware watchpoint 1: *$1

Old value = <unreadable>
New value = 0
main () at t.c:13
13              *ip = i;

real    0m0.231s
user    0m0.148s
sys     0m0.024s


gcc -g t.c -DLIMIT=10000 && time gdb64-cvs -q -ex run -ex 'up 2' -ex 'print ip'
-ex 'watch *$1' -ex 'run' -ex quit ./a.out

Program received signal SIGABRT, Aborted.
0x00007ffff7ab0095 in raise () from /lib/libc.so.6
#2  0x000000000040051e in main () at t.c:15
15                abort();
$1 = (int *) 0x64f1f0
Hardware watchpoint 1: *$1
Hardware watchpoint 1: *$1

Old value = <unreadable>
New value = 0
main () at t.c:13
13              *ip = i;

real    0m1.512s
user    0m1.060s
sys     0m0.340s


gcc -g t.c -DLIMIT=100000 && time gdb64-cvs -q -ex run -ex 'up 2' -ex 'print ip'
-ex 'watch *$1' -ex 'run' -ex quit ./a.out

Program received signal SIGABRT, Aborted.
0x00007ffff7ab0095 in raise () from /lib/libc.so.6
#2  0x000000000040051e in main () at t.c:15
15                abort();
$1 = (int *) 0x90e3f0
                                       <<< 15s delay here
Hardware watchpoint 1: *$1
Hardware watchpoint 1: *$1

Old value = <unreadable>
New value = 0
main () at t.c:13
13              *ip = i;

real    0m14.542s
user    0m11.733s
sys     0m2.644s


Why does it take 15s to do 100000 iterations with a HW watchpoint?
'set debug infrun 1' shows unending stream of:

infrun: prepare_to_wait
infrun: target_wait (-1, status) =
infrun:   18710 [process 18710],
infrun:   status->kind = stopped, signal = SIGTRAP
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x400507
infrun: stopped by watchpoint
infrun: stopped data address = 0x7fffffffd9f0
infrun: no stepping, continue
infrun: resume (step=0, signal=0), trap_expected=0
infrun: prepare_to_wait
infrun: target_wait (-1, status) =
infrun:   18710 [process 18710],
infrun:   status->kind = stopped, signal = SIGTRAP
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x400507
infrun: stopped by watchpoint
infrun: stopped data address = 0x7fffffffd9f0
infrun: no stepping, continue
infrun: resume (step=0, signal=0), trap_expected=0
...

-- 
           Summary: HW watchpoints silently degrade into SW ones
           Product: gdb
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: breakpoints
        AssignedTo: unassigned at sourceware dot org
        ReportedBy: ppluzhnikov at google dot com
                CC: gdb-prs at sourceware dot org
 GCC build triplet: x86_64-unknown-linux-gnu
  GCC host triplet: x86_64-unknown-linux-gnu
GCC target triplet: x86_64-unknown-linux-gnu


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2010-08-14 02:47 -------
Does this still happen with current mainline?


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2010-08-25 20:17 -------
Appears to still be a problem with GNU gdb (GDB) 7.2.50.20100825-cvs:

Timings for 1000, 10000 and 100000 iterations:
0.05user 0.00system 0:00.06elapsed 104%CPU (0avgtext+0avgdata 0maxresident)k
0.58user 0.27system 0:00.91elapsed 93%CPU (0avgtext+0avgdata 0maxresident)k
6.38user 3.14system 0:09.69elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k

without "watch *$1", the 100000 iteration times at:
0.05user 0.02system 0:00.07elapsed 101%CPU (0avgtext+0avgdata 0maxresident)k
so it's not the loop iteration itself that is taking the time.



-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|                            |1


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2010-08-25 20:50 -------
I don't see evidence of degeneration into single-stepping:

> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0x400507
> infrun: stopped by watchpoint
> infrun: stopped data address = 0x7fffffffd9f0
> infrun: no stepping, continue
> infrun: resume (step=0, signal=0), trap_expected=0

It appears to me the target is continuously reporting that something is writing
to the address you are watching, but, if the watched expression evaluates to the
same at each of those, then gdb will silently resume the inferior again.
This is the nature of write watchpoints.  Try an access watchpoint instead (awatch).

Your testcase looks bogus to me.  The heap memory address you are watching in
the new run of the program is most likely being used for something else during
program startup?


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2010-08-25 21:33 -------
(In reply to comment #3)
> I don't see evidence of degeneration into single-stepping:

Except the time it takes for the watchpoint to fire is unreasonable.

> It appears to me the target is continuously reporting that something
> is writing to the address you are watching,

Except when the WP fires, I get this:

Old value = <unreadable>
New value = 0
main () at t.c:13
13              *ip = i;
(gdb) p ip
$2 = (int *) 0x8f7ff0

(gdb) p $1
$3 = (int *) 0x90e3f0

So the memory was inaccessible, and just became accessible, but hasn't been
written to in a loop. I believe this proves your theory incorrect.

It I set 'awatch', here is what I observe:


gcc -g t.c -DLIMIT=100000 && /usr/bin/time gdb64-cvs -q -ex run -ex 'up 2' -ex
'print ip' -ex 'awatch *$1' -ex 'run'  ./a.out
Reading symbols from /tmp/pr10557/a.out...done.

Program received signal SIGABRT, Aborted.
0x00007ffff7ab0095 in raise () from /lib/libc.so.6
#2  0x000000000040051e in main () at t.c:15
15                abort();
$1 = (int *) 0x90e3f0
Hardware access (read/write) watchpoint 1: *$1
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
0x00007ffff7a9bf3d in ?? () from /lib/libc.so.6
(gdb) c
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
0x00007ffff7de9628 in ?? () from /lib64/ld-linux-x86-64.so.2
(gdb) c
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
0x00007ffff7de9628 in ?? () from /lib64/ld-linux-x86-64.so.2
(gdb) c
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
0x00007ffff7ab3325 in __cxa_atexit () from /lib/libc.so.6
(gdb) c
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
0x00007ffff7ab3371 in __cxa_atexit () from /lib/libc.so.6
(gdb) c
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
0x0000000000400562 in __libc_csu_init ()


Looks like *something* is single-stepping :-(


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2010-08-25 22:37 -------
On Wednesday 25 August 2010 22:33:22, ppluzhnikov at google dot com wrote:
> So the memory was inaccessible, and just became accessible, but hasn't been
> written to in a loop. I believe this proves your theory incorrect.

It actually does not.  :-)  See below.

> It I set 'awatch', here is what I observe:

Thanks.

> gcc -g t.c -DLIMIT=100000 && /usr/bin/time gdb64-cvs -q -ex run -ex 'up 2' -ex
> 'print ip' -ex 'awatch *$1' -ex 'run'  ./a.out
> Reading symbols from /tmp/pr10557/a.out...done.
> 
> Program received signal SIGABRT, Aborted.
> 0x00007ffff7ab0095 in raise () from /lib/libc.so.6
> #2  0x000000000040051e in main () at t.c:15
> 15                abort();
> $1 = (int *) 0x90e3f0
> Hardware access (read/write) watchpoint 1: *$1
> Hardware access (read/write) watchpoint 1: *$1

(...)

> Value = <unreadable>
> Hardware access (read/write) watchpoint 1: *$1
> 
> Value = <unreadable>
> 0x0000000000400562 in __libc_csu_init ()
> 
> 
> Looks like something is single-stepping :-(

I don't think it is.  Go back to using "watch", reenable
"set debug infrun 1", and confirm whether see a stream of
things like these:

> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0x400507
> infrun: stopped by watchpoint
> infrun: stopped data address = 0x7fffffffd9f0
> infrun: no stepping, continue
> infrun: resume (step=0, signal=0), trap_expected=0

If you don't see "resume (step=1,...", then there's no single-stepping.
"stopped by watchpoint" means that the target thinks the thread hit
a watchpoint (because the debug registers claim so).
But I now notice something: "0x7fffffffd9f0" is a suspicious
address, and it does not look like 'i'.  Is that a stack address?
That might explain why does the access watchpoint cause a stop,
only to then not be able to evaluate the expression (<unreadable>).
I sounds like gdb thought it needed to set a watchpoint on &i to be
able to watch your expression?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2010-08-25 23:02 -------
(In reply to comment #5)

> > Looks like something is single-stepping :-(
> 
> I don't think it is.  Go back to using "watch", reenable
> "set debug infrun 1", and confirm whether see a stream of
> things like these:
> 
> > infrun: TARGET_WAITKIND_STOPPED
> > infrun: stop_pc = 0x400507
> > infrun: stopped by watchpoint
> > infrun: stopped data address = 0x7fffffffd9f0
> > infrun: no stepping, continue
> > infrun: resume (step=0, signal=0), trap_expected=0
> 
> If you don't see "resume (step=1,...", then there's no single-stepping.

Confirmed: I don't.

> "stopped by watchpoint" means that the target thinks the thread hit
> a watchpoint (because the debug registers claim so).
> But I now notice something: "0x7fffffffd9f0" is a suspicious
> address, and it does not look like 'i'.  Is that a stack address?

Yes, it's &ip:

infrun: clear_proceed_status_thread (process 29122)
infrun: proceed (addr=0xffffffffffffffff, signal=144, step=0)
infrun: resume (step=0, signal=0), trap_expected=0
infrun: wait_for_inferior (treat_exec_as_sigtrap=0)
infrun: target_wait (-1, status) =
infrun:   29122 [process 29122],
infrun:   status->kind = stopped, signal = SIGTRAP
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x400507
infrun: stopped by watchpoint
infrun: stopped data address = 0x7fffffffd8d0
infrun: BPSTAT_WHAT_STOP_NOISY
infrun: stop_stepping
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
Hardware access (read/write) watchpoint 1: *$1

Value = <unreadable>
main () at t.c:13
13              *ip = i;
(gdb) p &i
$2 = (int *) 0x7fffffffd8dc
(gdb) p &ip
$3 = (int **) 0x7fffffffd8d0

> That might explain why does the access watchpoint cause a stop,
> only to then not be able to evaluate the expression (<unreadable>).
> I sounds like gdb thought it needed to set a watchpoint on &i to be
> able to watch your expression?

Good guess.

But if I use 'watch *(int*)0x601010' instead of 'watch *$1',
things get *much* worse, and GDB does start stepping:

gcc -g t.c -DLIMIT=1 && /usr/bin/time gdb64-cvs -q -ex run -ex 'up 2' -ex 'print
ip' -ex 'watch *(int*)0x90e3f0' -ex run -ex q  ./a.out
Reading symbols from /tmp/pr10557/a.out...done.

Program received signal SIGABRT, Aborted.
0x00007ffff7ab0095 in raise () from /lib/libc.so.6
#2  0x000000000040051b in main () at t.c:15
15                abort();
$1 = (int *) 0x601010
Watchpoint 1: *(int*)0x601010

Program received signal SIGABRT, Aborted.
0x00007ffff7ab0095 in raise () from /lib/libc.so.6
5.83user 4.71system 0:10.68elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k


Things to note above: 'Watchpoint 1' instead of 'Hardware watchpoint 1',
and much slower execution even with just one iteration of the loop.

Setting 'debug infrun' confirms that GDB is single-stepping in that case:

infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x7ffff7df4b73
infrun: no stepping, continue
infrun: resume (step=1, signal=0), trap_expected=0
infrun: prepare_to_wait
infrun: target_wait (-1, status) =
infrun:   6262 [process 6262],
infrun:   status->kind = stopped, signal = SIGTRAP
infrun: infwait_normal_state
...


When I do the watch manually, I can see the HW degrade into W (which is
why I filed this bug in the first place):

gdb64-cvs -q ./a.out
Reading symbols from /tmp/pr10557/a.out...done.
(gdb) r

Program received signal SIGABRT, Aborted.
0x00007ffff7ab0095 in raise () from /lib/libc.so.6
(gdb) up 2
#2  0x000000000040051b in main () at t.c:15
15                abort();
(gdb) p ip
$1 = (int *) 0x601010
(gdb) watch *(int*)0x601010
Hardware watchpoint 1: *(int*)0x601010
(gdb) r

   ... long delay here; note soft watchpoint below ...

Watchpoint 1: *(int*)0x601010

Old value = <unreadable>
New value = 0
0x00007ffff7b4e360 in brk () from /lib/libc.so.6
(gdb) q


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[pedro at codesourcery dot com]

------- Additional Comments From pedro at codesourcery dot com  2010-08-26 09:40 -------
This one is known.  It is PR10645.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[ppluzhnikov at google dot com]

------- Additional Comments From ppluzhnikov at google dot com  2010-08-26 14:11 -------
(In reply to comment #7)
> This one is known.  It is PR10645.

Indeed.

So I actually stumbled on PR10645, but while working on a demo case transformed
the problem into something different (GDB watching &ip when all I care about is
*$1), and didn't notice :-(

I think that given '$1 = (int *) 0x64f1f0', it is at least unexpected that
'watch *$1' has anything to with the original 'ip', so there may still be a
separate bug here.

Feel free to close as dup of PR10645 if you wish -- that's the one that really
kills me. 



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10557

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[sergiodj at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=10557

Sergio Durigan Junior <sergiodj at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sergiodj at redhat dot com

--- Comment #9 from Sergio Durigan Junior <sergiodj at redhat dot com> ---
So, what is the status of this bug?  Can we close it as dup of PR10645?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug breakpoints/10557] HW watchpoints silently degrade into SW ones

[ssbssa at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=10557

Hannes Domani <ssbssa at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |DUPLICATE
                 CC|                            |ssbssa at sourceware dot org
             Status|WAITING                     |RESOLVED

--- Comment #10 from Hannes Domani <ssbssa at sourceware dot org> ---
(In reply to Paul Pluzhnikov from comment #8)
> Feel free to close as dup of PR10645 if you wish -- that's the one that
> really kills me.

Done.

*** This bug has been marked as a duplicate of bug 10645 ***

-- 
You are receiving this mail because:
You are on the CC list for the bug.