[RFC] [PATCH v3] [PR python/29603] Disable out-of-scope watchpoints

[Johnson Sun <j3.soon777@gmail.com>]

Request for comments: 
<https://sourceware.org/pipermail/gdb-patches/2023-January/196068.html>.

Johnson

On 2/19/2023 12:26 AM, JohnsonSun wrote:
> Ping for: 
> <https://sourceware.org/pipermail/gdb-patches/2023-January/196068.html>.
>
> I would like to ask for some feedback before submitting the v4 patch.
>
> Johnson
>
> On 1/23/2023 6:15 PM, Johnson Sun wrote:
>> Hi,
>>
>> Thanks for reviewing this!
>>
>> > the supplied test does not pass for me
>>
>> The current test doesn't seem to produce consistent results across 
>> different
>> machines, I'll try to fix it in the next version.
>>
>> > Is there a reason to do it here in particular, and not, let's say as
>> > soon as we change the disposition to disp_del_at_next_stop?
>>
>> I have implemented this in the v1 patch, I called 
>> `disable_breakpoint' as
>> soon as we change the disposition to `disp_del_at_next_stop'
>> (in `watchpoint_del_at_next_stop'). However,
>> LancelotSIX <lsix@lancelotsix.com> mentioned that the fix is in a
>> non-extension-related code path, and suggested disabling it in
>> `bpstat_check_breakpoint_conditions' instead (the v3 patch).
>> See: 
>> https://sourceware.org/pipermail/gdb-patches/2022-September/192120.html
>>
>> Both the v1 and v3 patch fixes the issue. I personally prefer the v1 
>> patch.
>> (i.e., modifying `watchpoint_del_at_next_stop')
>>
>> > shouldn't marking the watchpoint as
>> > disp_del_at_next_stop make it so it won't be inserted next time we
>> > resume?  After all should_be_inserted returns false for breakpoint
>> > locations that are disp_del_at_next_stop.  Perhaps it's because 
>> since we
>> > don't do a "normal" stop, breakpoint locations stay inserted, and
>> > there's nothing that pulls this location out of the target? Therefore,
>> > maybe a solution, to keep things coherent, would be to pull the
>> > breakpoint locations out when marking the breakpoint as
>> > disp_del_at_next_stop?  This way, we would respect the invariant 
>> that a
>> > breakpoint disp_del_at_next_stop can't be inserted (I don't know if 
>> this
>> > is really what is happening though, it's just speculation).
>>
>> For software watchpoints, they cannot be pulled out of the target, since
>> they may not be inserted into the target in the first place:
>>
>>      /* Locations of type bp_loc_other and
>>     bp_loc_software_watchpoint are only maintained at GDB side,
>>     so there is no need to remove them.  */
>>
>>     -- gdb/breakpoint.c:3840
>>
>> Their expressions are re-evaluated by single-stepping through the 
>> program:
>>
>>     Software watchpoints are very slow, since GDB needs to single-step
>>     the program being debugged and test the value of the watched
>>     expression(s) after each instruction.
>>
>>     -- https://sourceware.org/gdb/wiki/Internals%20Watchpoints
>>
>> Therefore, we must either disable or delete the software watchpoint.
>> We cannot pull it out of the target since it's only maintained on the
>> GDB side.
>>
>> > And, in a broader scope, why wait until the next normal stop to delete
>> > the watchpoint?  A "normal" stop being a stop that we present to the
>> > user (the normal_stop function).  We currently call
>> > breakpoint_auto_delete in normal_stop, which is why we don't reach it
>> > when your Breakpoint.stop method returns False.  At the end of, let's
>> > say, handle_signal_stop, could we go over the bpstat chain and delete
>> > any breakpoint we've just hit that is marked for deletion?
>>
>> I believe this choice is due to performance issues.
>>
>> In an early attempt, I tried to call `breakpoint_auto_delete' on all 
>> kinds
>> of stops. However, this implementation is very slow when we have a large
>> number of breakpoints, since we need to go over the entire bpstat 
>> chain on
>> all stops. I recall that this implementation fails on certain 
>> testcases with
>> a large number of breakpoints with many breakpoint stops.
>>
>> Furthermore, the reason for using the `disp_del_at_next_stop' state 
>> remains
>> unclear, as mentioned in the comments:
>>
>>     /* NOTE drow/2003-09-08: This state only exists for removing
>>        watchpoints.  It's not clear that it's necessary...  */
>>
>>     -- gdb/breakpoint.c:2914
>>
>> By tracing the git history, the `disp_del_at_next_stop' state is 
>> introduced
>> in commit c906108c21474dfb4ed285bcc0ac6fe02cd400cc, which doesn't 
>> provide
>> any proper explanation of this state.
>>
>> I think the best way to deal with this is to avoid going over the entire
>> bpstat chain when deleting breakpoints. Potentially by keeping track of
>> a chain of breakpoints that should be deleted, and changing the 
>> bpstat chain
>> to a doubly linked list for the ease of deletion. Such changes deserve a
>> dedicated patch, though.
>>
>> To sum up, I prefer modifying `watchpoint_del_at_next_stop' (i.e., 
>> the v1 patch).
>> If you also think it's appropriate, I'll fix the failing test and 
>> prepare the
>> v4 patch accordingly.
>>
>> Johnson
>>
>> On 1/13/2023 11:40 PM, SimonMarchi wrote:
>>>
>>> On 10/20/22 14:11, Johnson Sun via Gdb-patches wrote:
>>>> Currently, when a local software watchpoint goes out of scope, GDB 
>>>> sets the
>>>> watchpoint's disposition to `delete at next stop' and then normal 
>>>> stops
>>>> (i.e., stop and wait for the next GDB command). When GDB normal 
>>>> stops, it
>>>> automatically deletes the breakpoints with their disposition set to
>>>> `delete at next stop'.
>>>>
>>>> Suppose a Python script decides not to normal stop when a local 
>>>> software
>>>> watchpoint goes out of scope, the watchpoint will not be automatically
>>>> deleted even when its disposition is set to `delete at next stop'.
>>>>
>>>> Since GDB single-steps the program and tests the watched expression 
>>>> after each
>>>> instruction, not deleting the watchpoint causes the watchpoint to 
>>>> be hit many
>>>> more times than it should, as reported in PR python/29603.
>>>>
>>>> This was happening because the watchpoint is not deleted or 
>>>> disabled when
>>>> going out of scope.
>>>>
>>>> This commit fixes this issue by disabling the watchpoint when going 
>>>> out of
>>>> scope. It also adds a test to ensure this feature isn't regressed 
>>>> in the
>>>> future.
>>>>
>>>> Two other solutions seem to solve this issue, but are in fact 
>>>> inappropriate:
>>>> 1. Automatically delete breakpoints on all kinds of stops
>>>>     (in `fetch_inferior_event'). This solution is very slow since 
>>>> the deletion
>>>>     requires O(N) time for N breakpoints.
>>>> 2. Disable the watchpoint after the watchpoint's disposition is set to
>>>>     `delete at next stop' (in `watchpoint_del_at_next_stop'). This 
>>>> solution
>>>>     modifies a non-extension-related code path, and isn't preferred 
>>>> since this
>>>>     issue cannot occur without extension languages. (gdb scripts 
>>>> always normal
>>>>     stop before continuing execution)
>>> I have a bit of trouble reviewing this, because I'm not too familiar
>>> with the lifecycle of watchpoints (I know the principle, but not the
>>> specifically where things happen in GDB).  So it's difficult to tell
>>> whether this is right or if there's a better way to handle it.
>>>
>>> However, the supplied test does not pass for me:
>>>
>>>      source 
>>> /home/simark/build/binutils-gdb/gdb/testsuite/outputs/gdb.python/py-watchpoint/py-watchpoint.py
>>>      Watchpoint 2: i
>>>      Python script imported
>>>      (gdb) PASS: gdb.python/py-watchpoint.exp: import python scripts
>>>      python print(len(gdb.breakpoints()))
>>>      2
>>>      (gdb) PASS: gdb.python/py-watchpoint.exp: check modified BP count
>>>      gdb_expect_list pattern: /Watchpoint Hit: ./
>>>      continue
>>>      Continuing.
>>>      Watchpoint Hit: 1
>>>      gdb_expect_list pattern: /[
>>>      ]+Watchpoint . deleted because the program has left the block in/
>>>      FAIL: gdb.python/py-watchpoint.exp: run until program stops 
>>> (pattern 2) (timeout)
>>>      gdb_expect_list pattern: /[
>>>      ]+which its expression is valid./
>>>      gdb_expect_list pattern: /Watchpoint Hit: ./
>>>      gdb_expect_list pattern: /[
>>>      ]+012\[Inferior 1 \(process .*\) exited normally\]/
>>>      gdb_expect_list pattern: //
>>>      python print(len(gdb.breakpoints()))
>>>      Watchpoint Hit: 2
>>>      Watchpoint Hit: 3
>>>      Watchpoint Hit: 4
>>>
>>>      Watchpoint 2 deleted because the program has left the block in
>>>      which its expression is valid.
>>>      Watchpoint Hit: 5
>>>      012[Inferior 1 (process 2381681) exited normally]
>>>      (gdb) FAIL: gdb.python/py-watchpoint.exp: check BP count (the 
>>> program exited)
>>>
>>>> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=29603
>>>> ---
>>>>   gdb/breakpoint.c                           |  2 +
>>>>   gdb/testsuite/gdb.python/py-watchpoint.c   | 27 ++++++++++++
>>>>   gdb/testsuite/gdb.python/py-watchpoint.exp | 48 
>>>> ++++++++++++++++++++++
>>>>   gdb/testsuite/gdb.python/py-watchpoint.py  | 30 ++++++++++++++
>>>>   4 files changed, 107 insertions(+)
>>>>   create mode 100644 gdb/testsuite/gdb.python/py-watchpoint.c
>>>>   create mode 100644 gdb/testsuite/gdb.python/py-watchpoint.exp
>>>>   create mode 100644 gdb/testsuite/gdb.python/py-watchpoint.py
>>>>
>>>> diff --git a/gdb/breakpoint.c b/gdb/breakpoint.c
>>>> index bff3bac7d1a..15f4ae2131c 100644
>>>> --- a/gdb/breakpoint.c
>>>> +++ b/gdb/breakpoint.c
>>>> @@ -5340,6 +5340,8 @@ bpstat_check_breakpoint_conditions (bpstat 
>>>> *bs, thread_info *thread)
>>>>     /* Evaluate extension language breakpoints that have a "stop" 
>>>> method
>>>>        implemented.  */
>>>>     bs->stop = breakpoint_ext_lang_cond_says_stop (b);
>>>> +  if (b->disposition == disp_del_at_next_stop)
>>>> +    disable_breakpoint(b);
>>> Is there a reason to do it here in particular, and not, let's say as
>>> soon as we change the disposition to disp_del_at_next_stop?
>>>
>>> Other question: shouldn't marking the watchpoint as
>>> disp_del_at_next_stop make it so it won't be inserted next time we
>>> resume?  After all should_be_inserted returns false for breakpoint
>>> locations that are disp_del_at_next_stop.  Perhaps it's because 
>>> since we
>>> don't do a "normal" stop, breakpoint locations stay inserted, and
>>> there's nothing that pulls this location out of the target? Therefore,
>>> maybe a solution, to keep things coherent, would be to pull the
>>> breakpoint locations out when marking the breakpoint as
>>> disp_del_at_next_stop?  This way, we would respect the invariant that a
>>> breakpoint disp_del_at_next_stop can't be inserted (I don't know if 
>>> this
>>> is really what is happening though, it's just speculation).
>>>
>>> And, in a broader scope, why wait until the next normal stop to delete
>>> the watchpoint?  A "normal" stop being a stop that we present to the
>>> user (the normal_stop function).  We currently call
>>> breakpoint_auto_delete in normal_stop, which is why we don't reach it
>>> when your Breakpoint.stop method returns False.  At the end of, let's
>>> say, handle_signal_stop, could we go over the bpstat chain and delete
>>> any breakpoint we've just hit that is marked for deletion?
>>>
>>> Simon
>>>
>