Dynamic watchpoints in dynamic memory

Dynamic watchpoints in dynamic memory

[Giles Atkinson]

Greetings,

I have a suspicion that these may be somewhat stupid questions, but I feel I have done enough searching of documentation, FAQs etc.,
without result, to be ready to ask.  I think this could be a FAQ-level query, but found no reference to the topic.

I have a C program that crashes on a modify access via a dynamically-allocated structure member with an illegal address, not NULL.
The core file, circumstances, and code review suggest that an unusual event has previously corrupted a pointer
in the structure, which the failing instruction dereferences.

My approach was to set a breakpoint on structure initialisation, with a command list to set  a watchpoint on the 
pointer member.  The watchpoint command list is backtrace and continue, output to file.  By setting the breakpoint on the
right instruction, the breakpoint command list can pick the structure address from a register, adding an offset.

That works: so far, so good.

The target structures are created and destroyed fairly frequently, responding to user input.
To avoid noise in the output, and limit the number of watchpoints, there is a breakpoint before deallocation
that attempts to remove the watchpoint on the structure, again using a register value.

Questions:
Does this make sense, or have I missed a better way?
How to remove the watchpoint? I can not identify the right syntax.

The watchpoint is set like this: watch -location *(void **)($esi + $offset)
I have tried to remove it with 'clear', but that seems to match watchpoints
using the 'watch' expression, and what I have is a different expression that should yield the same address.
The register is different on deallocation and the original command is not unique.

More puzzlement: my printf commands in (nested) command lists do nothing.

Environment: Ubuntu  X86 32-bit Ubuntu15.10, system supplied gdb 7.10. Program compiled on vanilla Debian 5.

Any advice will be appreciated,

Thanks,

Giles

Re: Dynamic watchpoints in dynamic memory

[Simon Marchi]

Hi Giles,

On 2016-02-05 18:21, Giles Atkinson wrote:
> Greetings,
> 
> I have a suspicion that these may be somewhat stupid questions, but I
> feel I have done enough searching of documentation, FAQs etc.,
> without result, to be ready to ask.  I think this could be a FAQ-level
> query, but found no reference to the topic.
> 
> I have a C program that crashes on a modify access via a
> dynamically-allocated structure member with an illegal address, not
> NULL.
> The core file, circumstances, and code review suggest that an unusual
> event has previously corrupted a pointer
> in the structure, which the failing instruction dereferences.

You might have done that already, but my first thought would be to try 
Valgrind.  It might not help here, since the rogue write spills into 
allocated code, but we never know.  That program does some amazing 
things.

Just to be sure, is it the pointer to the structure that becomes 
corrupted, or a pointer inside the structure, that points to something 
else?  Some times, it can appear to the the later, while it's the former 
(and you were just "lucky" that the first dereference did not produce a 
segfault, because it happened to be in mapped memory).

> My approach was to set a breakpoint on structure initialisation, with
> a command list to set  a watchpoint on the
> pointer member.  The watchpoint command list is backtrace and
> continue, output to file.  By setting the breakpoint on the
> right instruction, the breakpoint command list can pick the structure
> address from a register, adding an offset.

Silly question: since you are debugging C, don't you have access to a 
variable that holds the address of the structure?  It would be simpler 
to use that than to figure out which register to use.

> That works: so far, so good.
> 
> The target structures are created and destroyed fairly frequently,
> responding to user input.
> To avoid noise in the output, and limit the number of watchpoints,
> there is a breakpoint before deallocation
> that attempts to remove the watchpoint on the structure, again using a
> register value.
> 
> Questions:
> Does this make sense, or have I missed a better way?

I have done that in the past (using Python though), I think it makes 
sense.  It's good to keep in mind that if you track more than a few 
bytes, GDB will have to resort to use software watchpoints, which will 
slow down execution of the program by a few orders of magnitude.  It can 
be useful nonetheless.

> How to remove the watchpoint? I can not identify the right syntax.
> 
> The watchpoint is set like this: watch -location *(void **)($esi + 
> $offset)

Again, using C (if possible) would be much easier.  Something like

   (gdb) watch -location structure->field

might do the trick (untested, may require some more & or *).

> I have tried to remove it with 'clear', but that seems to match 
> watchpoints
> using the 'watch' expression, and what I have is a different
> expression that should yield the same address.
> The register is different on deallocation and the original command is
> not unique.

There might be a super clever way to do this using only base gdb 
commands, but for these things I go straight to Python:

https://sourceware.org/gdb/onlinedocs/gdb/Breakpoints-In-Python.html

Create three breakpoint classes: two for the allocation and 
de-allocation functions and one of type watchpoint, that will watch the 
actual structures.  In the allocation breakpoint's stop handler, you:

   1. get the address of the structure (e.g.: gdb.parse_and_eval('ptr'), 
where ptr is a pointer to the structure)
   2. create an instance of the watchpoint class based on the structure 
address
   3. put that in a global dict, that maps structure addresses to 
watchpoint object instances

In the de-allocation's stop handler, you:

   1. get the address of the structure
   2. find and remove the watchpoint instance from the global dict
   3. call .delete() on the watchpoint.

This way, you should have watchpoints that follow the lifetime of your 
objects (which is what I understood you wanted).  I hope it makes sense 
and is actually helpful.

> More puzzlement: my printf commands in (nested) command lists do 
> nothing.

Hmm I am not sure I understand, could you give a reproducible example?

Simon

Re: Dynamic watchpoints in dynamic memory

[Giles Atkinson]

Simon,

Thank you for your comments.  Some related comments/answers below:

> ... but my first thought would be to try Valgrind

Mine too, but that idea had already been discarded before the problem came to me.

> Just to be sure, is it the pointer to the structure that becomes corrupted, or a pointer inside the structure, that points to something else? 
Some times, it can appear to the the later, while it's the former (and you were just "lucky" that the first dereference did not produce a segfault,
because it happened to be in mapped memory).

It could be that.  It is actually a structure with a pointer to another of the same type and then a pointer to a different type.  
I parameterised the commands by structure offset to check both possibilities in separate test runs, and it could also look for
corruption of the linked list of the primary structures.

> Silly question: since you are debugging C, don't you have access to a variable that holds the address of the structure? 

Not silly at all.  This is commercial software, with a stripped binary.  The idea is to create a script that can be sent to a customer, as the bug
is not reproduced elsewhere.  So although it is C, I think it makes sense to work at assembler level.

> ... GDB will have to resort to use software watchpoints, which will slow down execution of the program by a few orders of magnitude.  It can be useful nonetheless.

I hope that will be OK.  It is very interactive, so with luck the user can reproduce it slowly.

> There might be a super clever way to do this using only base gdb commands, but for these things I go straight to Python:

So would I, if I knew the language, and thought we should persuade a customer to install it.
I think a variant of your idea can work with base gdb, although it is not clever.
There are a few bytes in the primary structure where I think I can hide the related watchpoint number.
Very ugly, but worth trying.

Many thanks,

Giles

On 6 Feb 2016, at 03:43, Simon Marchi wrote:

> Hi Giles,
> 
> On 2016-02-05 18:21, Giles Atkinson wrote:
>> Greetings,
>> I have a suspicion that these may be somewhat stupid questions, but I
>> feel I have done enough searching of documentation, FAQs etc.,
>> without result, to be ready to ask.  I think this could be a FAQ-level
>> query, but found no reference to the topic.
>> I have a C program that crashes on a modify access via a
>> dynamically-allocated structure member with an illegal address, not
>> NULL.
>> The core file, circumstances, and code review suggest that an unusual
>> event has previously corrupted a pointer
>> in the structure, which the failing instruction dereferences.
> 
> You might have done that already, but my first thought would be to try Valgrind.  It might not help here, since the rogue write spills into allocated code, but we never know.  That program does some amazing things.
> 
> Just to be sure, is it the pointer to the structure that becomes corrupted, or a pointer inside the structure, that points to something else?  Some times, it can appear to the the later, while it's the former (and you were just "lucky" that the first dereference did not produce a segfault, because it happened to be in mapped memory).
> 
>> My approach was to set a breakpoint on structure initialisation, with
>> a command list to set  a watchpoint on the
>> pointer member.  The watchpoint command list is backtrace and
>> continue, output to file.  By setting the breakpoint on the
>> right instruction, the breakpoint command list can pick the structure
>> address from a register, adding an offset.
> 
> Silly question: since you are debugging C, don't you have access to a variable that holds the address of the structure?  It would be simpler to use that than to figure out which register to use.
> 
>> That works: so far, so good.
>> The target structures are created and destroyed fairly frequently,
>> responding to user input.
>> To avoid noise in the output, and limit the number of watchpoints,
>> there is a breakpoint before deallocation
>> that attempts to remove the watchpoint on the structure, again using a
>> register value.
>> Questions:
>> Does this make sense, or have I missed a better way?
> 
> I have done that in the past (using Python though), I think it makes sense.  It's good to keep in mind that if you track more than a few bytes, GDB will have to resort to use software watchpoints, which will slow down execution of the program by a few orders of magnitude.  It can be useful nonetheless.
> 
>> How to remove the watchpoint? I can not identify the right syntax.
>> The watchpoint is set like this: watch -location *(void **)($esi + $offset)
> 
> Again, using C (if possible) would be much easier.  Something like
> 
>  (gdb) watch -location structure->field
> 
> might do the trick (untested, may require some more & or *).
> 
>> I have tried to remove it with 'clear', but that seems to match watchpoints
>> using the 'watch' expression, and what I have is a different
>> expression that should yield the same address.
>> The register is different on deallocation and the original command is
>> not unique.
> 
> There might be a super clever way to do this using only base gdb commands, but for these things I go straight to Python:
> 
> https://sourceware.org/gdb/onlinedocs/gdb/Breakpoints-In-Python.html
> 
> Create three breakpoint classes: two for the allocation and de-allocation functions and one of type watchpoint, that will watch the actual structures.  In the allocation breakpoint's stop handler, you:
> 
>  1. get the address of the structure (e.g.: gdb.parse_and_eval('ptr'), where ptr is a pointer to the structure)
>  2. create an instance of the watchpoint class based on the structure address
>  3. put that in a global dict, that maps structure addresses to watchpoint object instances
> 
> In the de-allocation's stop handler, you:
> 
>  1. get the address of the structure
>  2. find and remove the watchpoint instance from the global dict
>  3. call .delete() on the watchpoint.
> 
> This way, you should have watchpoints that follow the lifetime of your objects (which is what I understood you wanted).  I hope it makes sense and is actually helpful.
> 
>> More puzzlement: my printf commands in (nested) command lists do nothing.
> 
> Hmm I am not sure I understand, could you give a reproducible example?
> 
> Simon