internal-error: insert_step_resume_breakpoint_at_sal

internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

Debugging emacs in CVS with gdb in CVS, I often get an error if I try to step
through the program after setting a breakpoint:

To reproduce:

gdb emacs
GNU gdb 6.3.50_2004-11-24-cvs
...
(gdb) b Fsplit_window    (for example)
(gdb) run

`C-x 2' in Emacs

(gdb) n
infrun.c:2763: internal-error: insert_step_resume_breakpoint_at_sal: Assertion `step_resume_breakpoint == NULL' failed.
A problem internal to GDB has been detected,

GDB then offers to create a core. As few people on this list have CVS Emacs,
I attach it below. There appears to be no problem with older versions of
GDB e.g 5.2.1.


   Nick



#0  0x400babf1 in kill () from /lib/libc.so.6
#1  0x400baa05 in raise () from /lib/libc.so.6
#2  0x400bc01b in abort () from /lib/libc.so.6
#3  0x08085630 in internal_vproblem (problem=0x8270950, 
    file=0x8220d17 "infrun.c", line=2763, 
    fmt=0x81eed01 "%s: Assertion `%s' failed.", ap=0xbfffed5c " ü!\b")
    at utils.c:853
#4  0x08085650 in internal_verror (file=0x8220d17 "infrun.c", line=2763, 
    fmt=0x81eed01 "%s: Assertion `%s' failed.", ap=0xbfffed5c " ü!\b")
    at utils.c:867
#5  0x0808567a in internal_error (file=0x8220d17 "infrun.c", line=2763, 
    string=0x81eed01 "%s: Assertion `%s' failed.") at utils.c:876
#6  0x080fc68a in insert_step_resume_breakpoint_at_sal (sr_sal=
      {symtab = 0x0, section = 0x0, line = 0, pc = 134815836, end = 0}, 
    sr_id=
      {stack_addr = 3221220768, code_addr = 134815824, special_addr = 0, stack_ad
dr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2767
#7  0x080fc6ff in insert_step_resume_breakpoint_at_frame (return_frame=0x0)
    at infrun.c:2794
#8  0x080faeac in handle_inferior_event (ecs=0xbfffeec0) at infrun.c:2044
#9  0x080fa43b in wait_for_inferior () at infrun.c:986
#10 0x080fa1f7 in proceed (addr=1, siggnal=TARGET_SIGNAL_HUP, step=1)
    at infrun.c:806
#11 0x080f79a9 in step_1 (skip_subroutines=1, single_inst=0, 
    count_string=0x0) at infcmd.c:688
#12 0x080f7833 in next_command (count_string=0x0, from_tty=1)
    at infcmd.c:585
#13 0x080afa54 in do_cfunc (c=0x0, args=0x0, from_tty=1)
    at cli/cli-decode.c:57
#14 0x080b16f4 in cmd_func (cmd=0x82aef20, args=0x0, from_tty=1)
    at cli/cli-decode.c:1627
#15 0x08083632 in execute_command (p=0x8296c51 "", from_tty=1) at top.c:733
#16 0x08106f42 in command_handler (command=0x8296c50 "n") at event-top.c:500
#17 0x08107378 in command_line_handler (rl=0x1 <Address 0x1 out of bounds>)
    at event-top.c:793
#18 0x081d6419 in rl_callback_read_char () at callback.c:123
#19 0x081068fa in rl_callback_read_char_wrapper (client_data=0x0)
    at event-top.c:166
#20 0x08106e3f in stdin_event_handler (error=0, client_data=0x0)
    at event-top.c:416
#21 0x081061dd in handle_file_event (event_file_desc=1075480440)
    at event-loop.c:721
#22 0x08105ce4 in process_event () at event-loop.c:334
#23 0x08105d46 in gdb_do_one_event (data=0x0) at event-loop.c:371
#24 0x0808328f in do_catch_errors (uiout=0x82c3f18, data=0x0) at top.c:524
#25 0x08083165 in catcher (func=0x8083280 <do_catch_errors>, 
    func_uiout=0x82c3f18, func_args=0xbffff2a0, func_val=0xbffff298, 
    func_caught=0xbffff29c, errstring=0x0, gdberrmsg=0x0, mask=6)
    at top.c:431
#26 0x080832d8 in catch_errors (func=0, func_args=0x0, 
    errstring=0x81ed1dc "", mask=6) at top.c:536
#27 0x080bd243 in tui_command_loop (data=0x0) at tui/tui-interp.c:150
#28 0x08103e19 in current_interp_command_loop () at interps.c:277
#29 0x0807b02a in captured_command_loop (data=0x0) at main.c:91
#30 0x0808328f in do_catch_errors (uiout=0x82c3f18, data=0x0) at top.c:524
#31 0x08083165 in catcher (func=0x8083280 <do_catch_errors>, 
    func_uiout=0x82c3f18, func_args=0xbffff450, func_val=0xbffff448, 
    func_caught=0xbffff44c, errstring=0x0, gdberrmsg=0x0, mask=6)
    at top.c:431
#32 0x080832d8 in catch_errors (func=0, func_args=0x0, 
    errstring=0x81ed1dc "", mask=6) at top.c:536
#33 0x0807bb56 in captured_main (data=0x8296970) at main.c:801
#34 0x0808328f in do_catch_errors (uiout=0x8276400, data=0x0) at top.c:524
#35 0x08083165 in catcher (func=0x8083280 <do_catch_errors>, 
    func_uiout=0x8276400, func_args=0xbffff6f0, func_val=0xbffff6e8, 
    func_caught=0xbffff6ec, errstring=0x0, gdberrmsg=0x0, mask=6)
    at top.c:431
#36 0x080832d8 in catch_errors (func=0, func_args=0x0, 
    errstring=0x81ed1dc "", mask=6) at top.c:536
#37 0x0807bc72 in gdb_main (args=0x401a8778) at main.c:810
#38 0x0807b00f in main (argc=0, argv=0x0) at gdb.c:35

internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

Following my earlier e-mail (Wed, 24 Nov 2004 16:48:06 +1300):

 > Debugging emacs in CVS with gdb in CVS, I often get an error if I try to step
 > through the program after setting a breakpoint:
 > 
 > To reproduce:
 > 
 > gdb emacs
 > GNU gdb 6.3.50_2004-11-24-cvs
 > ...
 > (gdb) b Fsplit_window    (for example)
 > (gdb) run
 > 
 > `C-x 2' in Emacs
 > 
 > (gdb) n
 > infrun.c:2763: internal-error: insert_step_resume_breakpoint_at_sal: Assertion `step_resume_breakpoint == NULL' failed.
 > A problem internal to GDB has been detected,

infrun.c (in insert_step_resume_breakpoint_at_sal) says:

  /* There should never be more than one step-resume breakpoint per
     thread, so we should never be setting a new
     step_resume_breakpoint when one is already active.  */

However, in this case (and presumably others too) there is more than one
step-resume breakpoint. insert_step_resume_breakpoint_at_sal is called
at infrun.c:1931 and then infrun.c:1949 through handle_inferior_event:

First time:

#0  insert_step_resume_breakpoint_at_sal (sr_sal=
      {symtab = 0x0, section = 0x0, line = 0, pc = 134872212, end = 0}, sr_id=
      {stack_addr = 3221220224, code_addr = 134872206, special_addr = 0, stack_addr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2671
During symbol reading, incomplete CFI data; unspecified registers (e.g., eax) at#1  0x080fbe4f in insert_step_resume_breakpoint_at_frame (return_frame=0x0)
    at infrun.c:2699
#2  0x080fa6c3 in handle_inferior_event (ecs=0xbffff180) at infrun.c:1931
#3  0x080f9cb8 in wait_for_inferior () at infrun.c:974

Second time:

#0  internal_error (file=0x8221657 "infrun.c", line=2668, 
    string=0x81ef7a1 "%s: Assertion `%s' failed.") at utils.c:789
#1  0x080fbdda in insert_step_resume_breakpoint_at_sal (sr_sal=
      {symtab = 0x0, section = 0x0, line = 0, pc = 134872212, end = 0}, sr_id=
      {stack_addr = 3221220224, code_addr = 134872206, special_addr = 0, stack_addr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2672
#2  0x080fbe4f in insert_step_resume_breakpoint_at_frame (return_frame=0x0)
    at infrun.c:2699
#3  0x080fa677 in handle_inferior_event (ecs=0xbffff180) at infrun.c:1949
#4  0x080f9cb8 in wait_for_inferior () at infrun.c:974


Previously GDB just issued a warning, but there never seemed to be any
harm in dropping the old step_resume_breakpoint. How about replacing:

gdb_assert (step_resume_breakpoint == NULL);

in insert_step_resume_breakpoint_at_sal with:

warning
 ("GDB bug: infrun.c (wait_for_inferior): dropping old step_resume breakpoint");

which used to be in check_for_old_step_resume_breakpoint?


Nick

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

Nick Roberts wrote:
> Following my earlier e-mail (Wed, 24 Nov 2004 16:48:06 +1300):
> 
>  > Debugging emacs in CVS with gdb in CVS, I often get an error if I try to step
>  > through the program after setting a breakpoint:
>  > 
>  > To reproduce:
>  > 
>  > gdb emacs
>  > GNU gdb 6.3.50_2004-11-24-cvs
>  > ...
>  > (gdb) b Fsplit_window    (for example)
>  > (gdb) run
>  > 
>  > `C-x 2' in Emacs
>  > 
>  > (gdb) n
>  > infrun.c:2763: internal-error: insert_step_resume_breakpoint_at_sal: Assertion `step_resume_breakpoint == NULL' failed.
>  > A problem internal to GDB has been detected,
> 
> infrun.c (in insert_step_resume_breakpoint_at_sal) says:
> 
>   /* There should never be more than one step-resume breakpoint per
>      thread, so we should never be setting a new
>      step_resume_breakpoint when one is already active.  */
> 
> However, in this case (and presumably others too) there is more than one
> step-resume breakpoint. insert_step_resume_breakpoint_at_sal is called
> at infrun.c:1931 and then infrun.c:1949 through handle_inferior_event:
> 
> First time:
> 
> #0  insert_step_resume_breakpoint_at_sal (sr_sal=
>       {symtab = 0x0, section = 0x0, line = 0, pc = 134872212, end = 0}, sr_id=
>       {stack_addr = 3221220224, code_addr = 134872206, special_addr = 0, stack_addr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2671
> During symbol reading, incomplete CFI data; unspecified registers (e.g., eax) at#1  0x080fbe4f in insert_step_resume_breakpoint_at_frame (return_frame=0x0)
>     at infrun.c:2699

This sal/id ...

> #2  0x080fa6c3 in handle_inferior_event (ecs=0xbffff180) at infrun.c:1931
> #3  0x080f9cb8 in wait_for_inferior () at infrun.c:974
> 
> Second time:
> 
> #0  internal_error (file=0x8221657 "infrun.c", line=2668, 
>     string=0x81ef7a1 "%s: Assertion `%s' failed.") at utils.c:789
> #1  0x080fbdda in insert_step_resume_breakpoint_at_sal (sr_sal=
>       {symtab = 0x0, section = 0x0, line = 0, pc = 134872212, end = 0}, sr_id=
>       {stack_addr = 3221220224, code_addr = 134872206, special_addr = 0, stack_addr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2672

... and this sal/id look identical (true?).

This suggests that rather than inserting two different step-resume 
breakpoints it's inserting the same one twice (for possibly different 
reasons).

Is it possible to determine exactly why the step-resume breakpoint is 
being inserted for each of these cases?  If we know that the testsuite 
becomes possible, and with a testsuite a fix.

Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

 > > Second time:
 > > 
 > > #0  internal_error (file=0x8221657 "infrun.c", line=2668, 
 > >     string=0x81ef7a1 "%s: Assertion `%s' failed.") at utils.c:789
 > > #1  0x080fbdda in insert_step_resume_breakpoint_at_sal (sr_sal=
 > >       {symtab = 0x0, section = 0x0, line = 0, pc = 134872212, end = 0}, sr_id=
 > >       {stack_addr = 3221220224, code_addr = 134872206, special_addr = 0, stack_addr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2672
 > 
 > ... and this sal/id look identical (true?).

Well symtab = 0x0 looks unassigned to me.

 > This suggests that rather than inserting two different step-resume 
 > breakpoints it's inserting the same one twice (for possibly different 
 > reasons).
 > 
 > Is it possible to determine exactly why the step-resume breakpoint is 
 > being inserted for each of these cases?  If we know that the testsuite 
 > becomes possible, and with a testsuite a fix.

You'll probably need to give me a few clues. First time round,
stop_signal = TARGET_SIGNAL_0,
 and second time,
stop_signal = TARGET_SIGNAL_IO.

Does that tell you anything?


Nick

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

Nick Roberts wrote:
>  > > Second time:
>  > > 
>  > > #0  internal_error (file=0x8221657 "infrun.c", line=2668, 
>  > >     string=0x81ef7a1 "%s: Assertion `%s' failed.") at utils.c:789
>  > > #1  0x080fbdda in insert_step_resume_breakpoint_at_sal (sr_sal=
>  > >       {symtab = 0x0, section = 0x0, line = 0, pc = 134872212, end = 0}, sr_id=
>  > >       {stack_addr = 3221220224, code_addr = 134872206, special_addr = 0, stack_addr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2672
>  > 
>  > ... and this sal/id look identical (true?).
> 
> Well symtab = 0x0 looks unassigned to me.

More likely not-found, what's at that address?

>  > This suggests that rather than inserting two different step-resume 
>  > breakpoints it's inserting the same one twice (for possibly different 
>  > reasons).
>  > 
>  > Is it possible to determine exactly why the step-resume breakpoint is 
>  > being inserted for each of these cases?  If we know that the testsuite 
>  > becomes possible, and with a testsuite a fix.
> 
> You'll probably need to give me a few clues. First time round,
> stop_signal = TARGET_SIGNAL_0,
>  and second time,
> stop_signal = TARGET_SIGNAL_IO.
> 
> Does that tell you anything?

It makes me wonder if the inferior got an I/O signal and GDB to trying 
to skip that simultaneous to something else.  That, however, is idle 
speculation on my part.  It might be able to test this by modifying 
gdb.base/sigstep.c to schedule two signals instead of one.

Can you capture the output from "set debug infrun 1"?  It should help.

Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

 > > Well symtab = 0x0 looks unassigned to me.
 > 
 > More likely not-found, what's at that address?

Whats at 0x0?! Are you winding me up, or is it some kind of offset?

 > Can you capture the output from "set debug infrun 1"?  It should help.

See below. I can send the full transcript if necessary.

Nick

...
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x4012e7e9
infrun: random signal 20
infrun: resume (step=0, signal=20)
infrun: prepare_to_wait
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x4012e7e9
infrun: random signal 20
infrun: resume (step=0, signal=20)

infrun: prepare_to_wait
Breakpoint 3, Fsplit_window (window=137726961, size=137726961, 
    horflag=137726961) at window.c:3688
3688      if (NILP (window))
(gdb) n
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x4012e7e9
infrun: random signal 20
infrun: resume (step=0, signal=20)
infrun: prepare_to_wait
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x4012e7e9
infrun: random signal 20
infrun: resume (step=0, signal=20)
infrun: prepare_to_wait
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x80850d5
infrun: BPSTATE_WHAT_STOP_NOISY
infrun: stop_stepping

Breakpoint 1, internal_error (file=0x82214d7 "infrun.c", line=2671, 
    string=0x81ef681 "%s: Assertion `%s' failed.") at utils.c:789
789       va_start (ap, string);

RE: internal-error: insert_step_resume_breakpoint_at_sal

[Dave Korn]

> -----Original Message-----
> From: gdb-owner On Behalf Of Nick Roberts
> Sent: 20 January 2005 00:53

>  > > Well symtab = 0x0 looks unassigned to me.
>  > 
>  > More likely not-found, what's at that address?
> 
> Whats at 0x0?! Are you winding me up, or is it some kind of offset?

  :) I suspect he means "at the address given for pc= in the same function
call".

>  > > #1  0x080fbdda in insert_step_resume_breakpoint_at_sal (sr_sal=
>  > >       {symtab = 0x0, section = 0x0, line = 0, pc = 134872212, end = 0},
sr_id=
>  > >       {stack_addr = 3221220224, code_addr = 134872206, special_addr = 0,
stack_addr_p = 1, code_addr_p = 1, special_addr_p = 0}) at infrun.c:2672
>  > 

>  > Well symtab = 0x0 looks unassigned to me.

> More likely not-found, what's at that address?

  i.e. the symtab value is 0 because the lookup didn't find any entry
corresponding to the pc= address; what symbol table entry _ought_ to have been
found but wasn't?

    cheers, 
      DaveK
-- 
Can't think of a witty .sigline today....

RE: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

 > > -----Original Message-----
 > > From: gdb-owner On Behalf Of Nick Roberts
 > > Sent: 20 January 2005 00:53
 > 
 > >  > > Well symtab = 0x0 looks unassigned to me.
 > >  > 
 > >  > More likely not-found, what's at that address?
 > > 
 > > Whats at 0x0?! Are you winding me up, or is it some kind of offset?
 > 
 >   :) I suspect he means "at the address given for pc= in the same function
 > call".

"inf symbol 134872704" gives:

get_offsets + 416 in section .text

Nick

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

Nick Roberts wrote:
>  > > Well symtab = 0x0 looks unassigned to me.
>  > 
>  > More likely not-found, what's at that address?
> 
> Whats at 0x0?! Are you winding me up, or is it some kind of offset?

What's at $pc (where PC is 0x4012e7e9 or 0x80850d5), symtab==0 indicates 
a symtab lookup failure.

>  > Can you capture the output from "set debug infrun 1"?  It should help.
> 
> See below. I can send the full transcript if necessary.

How much more?

> Nick
> 
> ...
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0x4012e7e9
> infrun: random signal 20
> infrun: resume (step=0, signal=20)
> infrun: prepare_to_wait
> infrun: infwait_normal_state
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0x4012e7e9
> infrun: random signal 20
> infrun: resume (step=0, signal=20)
> 
> infrun: prepare_to_wait
> Breakpoint 3, Fsplit_window (window=137726961, size=137726961, 
>     horflag=137726961) at window.c:3688
> 3688      if (NILP (window))
> (gdb) n

There appears to be stuff missing here, the output should contain 
something like:

	resume(step=1, signal=0)
	resume(step=0, signal=0)

as GDB single-steps the thread off breakpoint 3.  The PC should be near 
or at 0x80850d5 found in the below.

> infrun: infwait_normal_state
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0x4012e7e9
> infrun: random signal 20
> infrun: resume (step=0, signal=20)
> infrun: prepare_to_wait
> infrun: infwait_normal_state
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0x4012e7e9
> infrun: random signal 20
> infrun: resume (step=0, signal=20)
> infrun: prepare_to_wait
> infrun: infwait_normal_state
> infrun: TARGET_WAITKIND_STOPPED
> infrun: stop_pc = 0x80850d5
> infrun: BPSTATE_WHAT_STOP_NOISY
> infrun: stop_stepping
> 
> Breakpoint 1, internal_error (file=0x82214d7 "infrun.c", line=2671, 
>     string=0x81ef681 "%s: Assertion `%s' failed.") at utils.c:789
> 789       va_start (ap, string);
> 

Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 1314 bytes --]

 > What's at $pc (where PC is 0x4012e7e9 or 0x80850d5), symtab==0 indicates 
 > a symtab lookup failure.

See my earlier message to Dave Korn.

 > >  > Can you capture the output from "set debug infrun 1"?  It should help.
 > > 
 > > See below. I can send the full transcript if necessary.
 > 
 > How much more?

Well its only the stuff before I type "next" i.e. Emacs starting up and
lots of:

 ...
 infrun: TARGET_WAITKIND_STOPPED
 infrun: stop_pc = 0x4012e7e9
 infrun: random signal 20
 infrun: resume (step=0, signal=20)
 infrun: prepare_to_wait
 infrun: infwait_normal_state
 infrun: TARGET_WAITKIND_STOPPED
 infrun: stop_pc = 0x4012e7e9
 infrun: random signal 20
 ...

 > There appears to be stuff missing here, the output should contain 
 > something like:
 > 
 > 	resume(step=1, signal=0)
 > 	resume(step=0, signal=0)
 > 
 > as GDB single-steps the thread off breakpoint 3.  The PC should be near 
 > or at 0x80850d5 found in the below.

Thats much earlier. I attach the full transcript below.

Maybe its too hard to debug over the Internet as I don't understand the
internals well enough. It doesn't seem to be causing a problem for anybody
else and I can get round it in the way I suggested in my first message.

I'll raise the matter again when I understand it better.

Thanks for looking at it,

Nick


[-- Attachment #2: gdb log --]
[-- Type: application/octet-stream, Size: 963 bytes --]

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

Nick Roberts wrote:
>  > What's at $pc (where PC is 0x4012e7e9 or 0x80850d5), symtab==0 indicates 
>  > a symtab lookup failure.
> 
> See my earlier message to Dave Korn.
> 
>  > >  > Can you capture the output from "set debug infrun 1"?  It should help.
>  > > 
>  > > See below. I can send the full transcript if necessary.
>  > 
>  > How much more?
> 
> Well its only the stuff before I type "next" i.e. Emacs starting up and
> lots of:
> 
>  ...
>  infrun: TARGET_WAITKIND_STOPPED
>  infrun: stop_pc = 0x4012e7e9
>  infrun: random signal 20
>  infrun: resume (step=0, signal=20)
>  infrun: prepare_to_wait
>  infrun: infwait_normal_state
>  infrun: TARGET_WAITKIND_STOPPED
>  infrun: stop_pc = 0x4012e7e9
>  infrun: random signal 20
>  ...
> 
>  > There appears to be stuff missing here, the output should contain 
>  > something like:
>  > 
>  > 	resume(step=1, signal=0)
>  > 	resume(step=0, signal=0)
>  > 
>  > as GDB single-steps the thread off breakpoint 3.  The PC should be near 
>  > or at 0x80850d5 found in the below.
> 
> Thats much earlier. I attach the full transcript below.
> 
> Maybe its too hard to debug over the Internet as I don't understand the
> internals well enough. It doesn't seem to be causing a problem for anybody
> else and I can get round it in the way I suggested in my first message.

Tracking down these sorts of bugs is really important - it makes the 
difference between a toy and a real debugger, and I think your efforts 
have paid off :-)

With the log you provided I've been able to identify one bug - 
back-to-back signals (where the inferior was receiving the next signal 
just as the previous handler returned) would lead to a panic.

I'm about to commit a testcase and fix.

thanks,
Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

 > Tracking down these sorts of bugs is really important - it makes the 
 > difference between a toy and a real debugger, and I think your efforts 
 > have paid off :-)
 > 
 > With the log you provided I've been able to identify one bug - 
 > back-to-back signals (where the inferior was receiving the next signal 
 > just as the previous handler returned) would lead to a panic.
 > 
 > I'm about to commit a testcase and fix.

With todays cvs (GNU gdb 6.3.50.20050126-cvs) I no longer get an internal
error but everything hangs instead. I presume that it is GDB hanging and
not Emacs, as Emacs does not hang at this place when its not running under
debug (I'm only splitting a window).

However, I don't know how to get a backtrace as sending SIGINT (actually
SIGSTOP with Emacs) interrupts the bottom level process (Emacs in this
case). I can then interrupt the GDB being debugged, but presumably by then
things have changed.

Nick

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

Nick Roberts wrote:
>  > Tracking down these sorts of bugs is really important - it makes the 
>  > difference between a toy and a real debugger, and I think your efforts 
>  > have paid off :-)
>  > 
>  > With the log you provided I've been able to identify one bug - 
>  > back-to-back signals (where the inferior was receiving the next signal 
>  > just as the previous handler returned) would lead to a panic.
>  > 
>  > I'm about to commit a testcase and fix.
> 
> With todays cvs (GNU gdb 6.3.50.20050126-cvs) I no longer get an internal
> error but everything hangs instead. I presume that it is GDB hanging and
> not Emacs, as Emacs does not hang at this place when its not running under
> debug (I'm only splitting a window).

"sick".  Is the testcase passing?

> However, I don't know how to get a backtrace as sending SIGINT (actually
> SIGSTOP with Emacs) interrupts the bottom level process (Emacs in this
> case). I can then interrupt the GDB being debugged, but presumably by then
> things have changed.

You can attach to gdb - $ gdb gdb <pid> - and then get a backtrace that way.

Anyway, same as before, lets look at a transcript (see the script 
command) with "set debug infrun 1" (and perhaphs also "set debug target 
1" - that one is really verbose).

Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 1229 bytes --]

 > > However, I don't know how to get a backtrace as sending SIGINT (actually
 > > SIGSTOP with Emacs) interrupts the bottom level process (Emacs in this
 > > case). I can then interrupt the GDB being debugged, but presumably by then
 > > things have changed.
 > 
 > You can attach to gdb - $ gdb gdb <pid> - and then get a backtrace that way.

Ah, yes! It just tells me GDB is waiting. Perhaps that was obvious but I had
hoped it would be looping as that would have been a problem that I could have
debugged.

#0  0x4012e7e9 in wait4 () from /lib/libc.so.6
#1  0x4012e787 in waitpid () from /lib/libc.so.6
#2  0x0809b2e2 in child_wait (ptid={pid = -1, lwp = 0, tid = 0}, 
    ourstatus=0xbffff1d0) at linux-nat.c:1689
During symbol reading, incomplete CFI data; unspecified registers (e.g., eax) at 0x809b2fd.
#3  0x080f98ea in wait_for_inferior () at infrun.c:973


 > Anyway, same as before, lets look at a transcript (see the script 
 > command) with "set debug infrun 1" (and perhaphs also "set debug target 
 > 1" - that one is really verbose).

Its attached below. GDB isn't hanging as it spews out more output from infrun
if I move the mouse around in Emacs. Its just not letting Emacs reach the next
statement.


Nick



[-- Attachment #2: gdb infrun log --]
[-- Type: application/octet-stream, Size: 881 bytes --]

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

Nick Roberts wrote:
>  > > However, I don't know how to get a backtrace as sending SIGINT (actually
>  > > SIGSTOP with Emacs) interrupts the bottom level process (Emacs in this
>  > > case). I can then interrupt the GDB being debugged, but presumably by then
>  > > things have changed.
>  > 
>  > You can attach to gdb - $ gdb gdb <pid> - and then get a backtrace that way.
> 
> Ah, yes! It just tells me GDB is waiting. Perhaps that was obvious but I had
> hoped it would be looping as that would have been a problem that I could have
> debugged.
> 
> #0  0x4012e7e9 in wait4 () from /lib/libc.so.6
> #1  0x4012e787 in waitpid () from /lib/libc.so.6
> #2  0x0809b2e2 in child_wait (ptid={pid = -1, lwp = 0, tid = 0}, 
>     ourstatus=0xbffff1d0) at linux-nat.c:1689
> During symbol reading, incomplete CFI data; unspecified registers (e.g., eax) at 0x809b2fd.
> #3  0x080f98ea in wait_for_inferior () at infrun.c:973
> 
> 
>  > Anyway, same as before, lets look at a transcript (see the script 
>  > command) with "set debug infrun 1" (and perhaphs also "set debug target 
>  > 1" - that one is really verbose).
> 
> Its attached below. GDB isn't hanging as it spews out more output from infrun
> if I move the mouse around in Emacs. Its just not letting Emacs reach the next
> statement.

This sounds like something was dropped on the floor :-(

Looking at the logs, it appears that GDB is continuously receiving and 
then delivering signals (resume step=0, signal=20) and never getting an 
oportunity to do a step :-(

We're going to need more info, try

$ script -c gdb
(gdb) file ...
(gdb) set debug target 1
(gdb) set debug infrun 1
(gdb) run
..
.
(gdb) step

..
(gdb) quit

it will create a transcript file that interleaves the log output and the 
commands you've typed (and it will be big!).

Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 1001 bytes --]

 > We're going to need more info, try
 > 
 > $ script -c gdb

$ script -c gdb
script: invalid option -- c
usage: script [-a] [-f] [-q] [-t] [file]

I don't know why script needs to be used but this is roughly what I've done:

$ gdb gdb
(top-gdb) set logging on
(top-gdb) cd ~/emacs/src            (to get in the right directory for Emacs)
(top-gdb) set debug target 1
(top-gdb) set debug infrun 1
(top-gdb) run emacs
...
.
(gdb) b Fsplit_window
(gdb) run
Starting program: /home/nick/emacs/src/emacs -geometry 80x40+0+0
...
.

        C-x 2 in Emacs

...
.
Breakpoint 3, Fsplit_window (window=137731057, size=137731057, 
    horflag=137731057) at window.c:3697
3697      if (NILP (window))
(gdb) n
...
.
^C           (kills Emacs)
^C           (kills GDB)
Program received signal SIGINT, Interrupt.
0x401563d4 in poll () from /lib/libc.so.6
(top-gdb) ^D
The program is running.  Exit anyway? (y or n) y

(I'm looking at gdb.base/sigrepeat.exp now.)

Nick


[-- Attachment #2: gdb infrun target log --]
[-- Type: application/octet-stream, Size: 3006 bytes --]

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

[chug chug chug]

This is what I've noticed ...

(gdb) n
infrun: proceed (addr=0xffffffff, signal=144, step=1)
GDB tries to single step off the breakpoint @0x80a05f6, (when doing this 
all breakpoints are removed).  GDB is expecting to get a SIGTRAP back ...

infrun: resume (step=1, signal=0)
target_terminal_inferior ()
child:target_xfer_partial (2, (null), 0xbffff18a,  0x0,  0x80a05f6, 2) = 
2, bytes = 8b 45
target_resume (2896, step, 0)
infrun: wait_for_inferior

... but instead the inferior executes _no_ instructions and instead 
returns a [pending] SIGIO (still @0x80a05f6) ...

target_wait (-1, status) = 2896,   status->kind = stopped, signal = SIGIO
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
target_fetch_registers (eip) = f6050a08 0x80a05f6 134874614
infrun: stop_pc = 0x80a05f6
infrun: random signal 23
... so GDB again tries to step the inferior, this time passing in the 
SIGNAL, and expecting back a SIGTRAP when the inferior reaches the 
signal handlers first instruction ...

[wonder if, for this case GDB can insert all breakpoints and do a continue]

infrun: resume (step=1, signal=23)
target_terminal_inferior ()
child:target_xfer_partial (2, (null), 0xbfffeffa,  0x0,  0x80a05f6, 2) = 
2, bytes = 8b 45
target_resume (2896, step, SIGIO)
infrun: prepare_to_wait

... but instead GDB gets back another pending SIGIO with the PC still 
back at the original BP instruction!

=> that's a known kernel bug.  Instead of single-stepping into the 
signal handler, the kernel runs the signal handler to completion.  Fixed 
in very recent 2.6 kernels.

... so gdb tries again to step into the signal handler again (remember, 
GDB's expecting the inferior to return a sigtrap from the process 
reaching the signal handler's first instruction) ...

target_wait (-1, status) = 2896,   status->kind = stopped, signal = SIGIO
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
target_fetch_registers (eip) = f6050a08 0x80a05f6 134874614
infrun: stop_pc = 0x80a05f6
infrun: random signal 23
infrun: resume (step=1, signal=23)
target_terminal_inferior ()
child:target_xfer_partial (2, (null), 0xbfffeffa,  0x0,  0x80a05f6, 2) = 
2, bytes = 8b 45
target_resume (2896, step, SIGIO)
infrun: prepare_to_wait

... but instead the signal handler runs, returns, and then executes 
_one_ instruction before trapping!

=> that's part of the same kernel bug

target_wait (-1, status) = 2896,   status->kind = stopped, signal = SIGTRAP
target_fetch_registers (eip) = f9050a08 0x80a05f9 134874617
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x80a05f9
infrun: trap expected
infrun: step-resume breakpoint
... so I think we've now got gdb thinking that the inferior got a 
sigtrap  [tick] indicating that the process reached the handler [cross] 
and hence the the inferior should continue until the signal handler 
returns ...

child:target_xfer_partial (2, (null), 0x84ff020,  0x0,  0x810bdc0, 1) = 
1, bytes =
 83
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x810bdc0, 1) = 
1, bytes =
 cc
target_insert_breakpoint (0x810bdc0, xxx) = 0
child:target_xfer_partial (2, (null), 0x85004c8,  0x0,  0x80e74c1, 1) = 
1, bytes = 68
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x80e74c1, 1) = 
1, bytes =
 cc
target_insert_breakpoint (0x80e74c1, xxx) = 0
child:target_xfer_partial (2, (null), 0x874d058,  0x0,  0x80a05f6, 1) = 
1, bytes = 8b
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x80a05f6, 1) = 
1, bytes =
 cc
target_insert_breakpoint (0x80a05f6, xxx) = 0
child:target_xfer_partial (2, (null), 0x874d348,  0x0,  0x40009bd0, 1) = 
1, bytes = c3
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x40009bd0, 1) = 
1, bytes =
 cc
target_insert_breakpoint (0x40009bd0, xxx) = 0
child:target_xfer_partial (2, (null), 0x8aacb20,  0x0,  0x4000c2e0, 1) = 
1, bytes =
 55
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x4000c2e0, 1) = 
1, bytes =
 cc
target_insert_breakpoint (0x4000c2e0, xxx) = 0
child:target_xfer_partial (2, (null), 0x87f2118,  0x0,  0x4032e860, 1) = 
1, bytes = 55
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x4032e860, 1) = 
1, bytes =
 cc
target_insert_breakpoint (0x4032e860, xxx) = 0
infrun: resume (step=0, signal=0)

... consequently GDB inserts breakpoints, including the step-resume 
breakpoint at expected signal return address, and continues the target. 
 The inferior is ment to next run the handler, return, and hit the 
step-resume breakpoint.  Funny enough this doesn't work (it's well past 
that step resume breakpoint).

First thing I'd do is run the sig*.exp tests to see which are failing - 
if the kernel's working correctly they all pass.

The next thing we can consider is, for the case of GDB wanting to "next" 
over a signal optimizing things so that instead it does a 
"continue/signal".  That might help a little, but then it might not.

Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

[chug chug chug]

This is what I've noticed ...

(gdb) n
infrun: proceed (addr=0xffffffff, signal=144, step=1)

GDB tries to single step off the breakpoint @0x80a05f6, (when doing this 
all breakpoints are removed).  GDB is expecting to get a SIGTRAP back ...

infrun: resume (step=1, signal=0)
target_terminal_inferior ()
child:target_xfer_partial (2, (null), 0xbffff18a,  0x0,  0x80a05f6, 2) = 
2, bytes = 8b 45
target_resume (2896, step, 0)
infrun: wait_for_inferior

... but instead the inferior executes _no_ instructions and instead 
returns a [pending] SIGIO (still @0x80a05f6) ...

target_wait (-1, status) = 2896,   status->kind = stopped, signal = SIGIO
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
target_fetch_registers (eip) = f6050a08 0x80a05f6 134874614
infrun: stop_pc = 0x80a05f6
infrun: random signal 23

... so GDB again tries to step the inferior, this time passing in the 
SIGNAL, and expecting back a SIGTRAP when the inferior reaches the 
signal handlers first instruction ...

[wonder if, for this case GDB can insert all breakpoints and do a continue]

infrun: resume (step=1, signal=23)
target_terminal_inferior ()
child:target_xfer_partial (2, (null), 0xbfffeffa,  0x0,  0x80a05f6, 2) = 
2, bytes = 8b 45
target_resume (2896, step, SIGIO)
infrun: prepare_to_wait

... but instead GDB gets back another pending SIGIO with the PC still 
back at the original BP instruction!

=> that's a known kernel bug.  Instead of single-stepping into the 
signal handler, the kernel runs the signal handler to completion.  Fixed 
in very recent 2.6 kernels.

... so gdb tries again to step into the signal handler again (remember, 
GDB's expecting the inferior to return a sigtrap from the process 
reaching the signal handler's first instruction) ...

target_wait (-1, status) = 2896,   status->kind = stopped, signal = SIGIO
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
target_fetch_registers (eip) = f6050a08 0x80a05f6 134874614
infrun: stop_pc = 0x80a05f6
infrun: random signal 23
infrun: resume (step=1, signal=23)
target_terminal_inferior ()
child:target_xfer_partial (2, (null), 0xbfffeffa,  0x0,  0x80a05f6, 2) = 
2, bytes = 8b 45
target_resume (2896, step, SIGIO)
infrun: prepare_to_wait

... but instead the signal handler runs, returns, and then executes 
_one_ instruction before trapping!

=> that's part of the same kernel bug

target_wait (-1, status) = 2896,   status->kind = stopped, signal = SIGTRAP
target_fetch_registers (eip) = f9050a08 0x80a05f9 134874617
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x80a05f9
infrun: trap expected
infrun: step-resume breakpoint

... so I think we've now got gdb thinking that the inferior got a 
sigtrap  [tick] indicating that the process reached the handler [cross] 
and hence the the inferior should continue until the signal handler 
returns ...

child:target_xfer_partial (2, (null), 0x84ff020,  0x0,  0x810bdc0, 1) = 
1, bytes =
  83
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x810bdc0, 1) = 
1, bytes =
  cc
target_insert_breakpoint (0x810bdc0, xxx) = 0
child:target_xfer_partial (2, (null), 0x85004c8,  0x0,  0x80e74c1, 1) = 
1, bytes = 68
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x80e74c1, 1) = 
1, bytes =
  cc
target_insert_breakpoint (0x80e74c1, xxx) = 0
child:target_xfer_partial (2, (null), 0x874d058,  0x0,  0x80a05f6, 1) = 
1, bytes = 8b
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x80a05f6, 1) = 
1, bytes =
  cc
target_insert_breakpoint (0x80a05f6, xxx) = 0
child:target_xfer_partial (2, (null), 0x874d348,  0x0,  0x40009bd0, 1) = 
1, bytes = c3
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x40009bd0, 1) = 
1, bytes =
  cc
target_insert_breakpoint (0x40009bd0, xxx) = 0
child:target_xfer_partial (2, (null), 0x8aacb20,  0x0,  0x4000c2e0, 1) = 
1, bytes =
  55
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x4000c2e0, 1) = 
1, bytes =
  cc
target_insert_breakpoint (0x4000c2e0, xxx) = 0
child:target_xfer_partial (2, (null), 0x87f2118,  0x0,  0x4032e860, 1) = 
1, bytes = 55
child:target_xfer_partial (2, (null), 0x0,  0x8271b10,  0x4032e860, 1) = 
1, bytes =
  cc
target_insert_breakpoint (0x4032e860, xxx) = 0
infrun: resume (step=0, signal=0)

... consequently GDB inserts breakpoints, including the step-resume 
breakpoint at expected signal return address, and continues the target. 
  The inferior is ment to next run the handler, return, and hit the 
step-resume breakpoint.  Funny enough this doesn't work (it's well past 
that step resume breakpoint).

First thing I'd do is run the sig*.exp tests to see which are failing - 
if the kernel's working correctly they all pass.

The next thing we can consider is, for the case of GDB wanting to "next" 
over a signal optimizing things so that instead it does a 
"continue/signal".  That might help a little, but then it might not.

Andrew

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Nick Roberts]

 > First thing I'd do is run the sig*.exp tests to see which are failing - 
 > if the kernel's working correctly they all pass.

Running ./gdb.base/sigall.exp ...
Running ./gdb.base/sigaltstack.exp ...
Running ./gdb.base/sigbpt.exp ...
KFAIL: gdb.base/sigbpt.exp: stepi; stepi out of handler (executed fault insn) (PRMS: gdb/1702)
KFAIL: gdb.base/sigbpt.exp: stepi bp before segv; stepi out of handler (executed fault insn) (PRMS: gdb/1702)
KFAIL: gdb.base/sigbpt.exp: stepi bp at segv; stepi out of handler (corrupt pc) (PRMS: gdb/1702)
KFAIL: gdb.base/sigbpt.exp: stepi bp before and at segv; stepi out of handler (corrupt pc) (PRMS: gdb/1702)
Running ./gdb.base/siginfo.exp ...
Running ./gdb.base/signals.exp ...
KFAIL: gdb.base/signals.exp: override SIGINT (PRMS: gdb/1707)
Running ./gdb.base/signull.exp ...
Running ./gdb.base/sigrepeat.exp ...
Running ./gdb.base/sigstep.exp ...
KFAIL: gdb.base/sigstep.exp: step on breakpoint, to handler; performing step (PRMS: gdb/1738)
KFAIL: gdb.base/sigstep.exp: next on breakpoint, to handler; performing next (PRMS: gdb/1738)
KFAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler; performing continue (PRMS: gdb/1738)
KFAIL: gdb.base/sigstep.exp: step on breakpoint, to handler entry; performing step (PRMS: gdb/1738)
KFAIL: gdb.base/sigstep.exp: next on breakpoint, to handler entry; performing next (PRMS: gdb/1738)
KFAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler entry; performing continue (PRMS: gdb/1738)


Note though, that many other tests fail too. What's the simplest way to run an
individual test? I used:

/home/nick/src/gdb/../dejagnu/runtest -tool gdb GDB=../gdb ./gdb.base/sigbpt.exp

but that can't be right.

Nick

Re: internal-error: insert_step_resume_breakpoint_at_sal

[Andrew Cagney]

Nick Roberts wrote:
>  > First thing I'd do is run the sig*.exp tests to see which are failing - 
>  > if the kernel's working correctly they all pass.

Definitly kernel bugs.  See http://sources.redhat.com/gdb/bugs/1702 for 
instance.

> Running ./gdb.base/sigall.exp ...
> Running ./gdb.base/sigaltstack.exp ...
> Running ./gdb.base/sigbpt.exp ...
> KFAIL: gdb.base/sigbpt.exp: stepi; stepi out of handler (executed fault insn) (PRMS: gdb/1702)
> KFAIL: gdb.base/sigbpt.exp: stepi bp before segv; stepi out of handler (executed fault insn) (PRMS: gdb/1702)
> KFAIL: gdb.base/sigbpt.exp: stepi bp at segv; stepi out of handler (corrupt pc) (PRMS: gdb/1702)
> KFAIL: gdb.base/sigbpt.exp: stepi bp before and at segv; stepi out of handler (corrupt pc) (PRMS: gdb/1702)
> Running ./gdb.base/siginfo.exp ...
> Running ./gdb.base/signals.exp ...
> KFAIL: gdb.base/signals.exp: override SIGINT (PRMS: gdb/1707)
> Running ./gdb.base/signull.exp ...
> Running ./gdb.base/sigrepeat.exp ...
> Running ./gdb.base/sigstep.exp ...
> KFAIL: gdb.base/sigstep.exp: step on breakpoint, to handler; performing step (PRMS: gdb/1738)
> KFAIL: gdb.base/sigstep.exp: next on breakpoint, to handler; performing next (PRMS: gdb/1738)
> KFAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler; performing continue (PRMS: gdb/1738)
> KFAIL: gdb.base/sigstep.exp: step on breakpoint, to handler entry; performing step (PRMS: gdb/1738)
> KFAIL: gdb.base/sigstep.exp: next on breakpoint, to handler entry; performing next (PRMS: gdb/1738)
> KFAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler entry; performing continue (PRMS: gdb/1738)
> 
> 
> Note though, that many other tests fail too. What's the simplest way to run an
> individual test? I used:
> 
> /home/nick/src/gdb/../dejagnu/runtest -tool gdb GDB=../gdb ./gdb.base/sigbpt.exp
> 
> but that can't be right.

It works right :-)

The other way is:
	make check RUNTESTFLAGS=sigbpt.exp
(RUNTESTFLAGS gets passed to dejagnu)

Andrew