[ECOS] gdb 'next' problem with i386 HAL

[ECOS] gdb 'next' problem with i386 HAL

[John Gumb]

Hi,

I've managed to get the i386 HAL up and running without too much bother - the only oddity I have is with GDB. The 'step' command works fine but the 'next' command simply results in the program continuing without breaking at the next line.

I have seen some posts related to this in that the gdb front end translates 'next' into a setting of a temporary breakpoint and then issuing a 'continue' so I could imagine I might see this sort of behaviour.

After poking around a bit, I can't see anything obvious.

Any suggestions as to what's up would be appreciated.

Cheers,

John

Re: [ECOS] gdb 'next' problem with i386 HAL

[Jonathan Larmour]

John Gumb wrote:
> 
> Hi,
> 
> I've managed to get the i386 HAL up and running without too much bother - the only oddity I have is with GDB. The 'step' command works fine but the 'next' command simply results in the program continuing without breaking at the next line.
> 
> I have seen some posts related to this in that the gdb front end translates 'next' into a setting of a temporary breakpoint and then issuing a 'continue' so I could imagine I might see this sort of behaviour.
> 
> After poking around a bit, I can't see anything obvious.

This works for me. What version of eCos/RedBoot? What version of GDB?

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] gdb 'next' problem with i386 HAL

[Boris V. Guzhov]

Hi,
I also have such behaviour of the debugger.
My host Linux Redhat 6.2.
gdb version: i386-elf-gdb --verison returns GDB 5.0.
eCos version (with Redboot)  from anon CVS  (Juli 25)
--
Boris Guzhov,
St.Petersburg, Russia

> John Gumb wrote:
> >
> > Hi,
> >
> > I've managed to get the i386 HAL up and running without too much
bother - the only oddity I have is with GDB. The 'step' command works fine
but the 'next' command simply results in the program continuing without
breaking at the next line.
> >
> > I have seen some posts related to this in that the gdb front end
translates 'next' into a setting of a temporary breakpoint and then issuing
a 'continue' so I could imagine I might see this sort of behaviour.
> >
> > After poking around a bit, I can't see anything obvious.
>
> This works for me. What version of eCos/RedBoot? What version of GDB?
>
> Jifl
> --
> Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
> Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine
>

Re: [ECOS] gdb 'next' problem with i386 HAL

[Jonathan Larmour]

"Boris V. Guzhov" wrote:
> 
> Hi,
> I also have such behaviour of the debugger.
> My host Linux Redhat 6.2.
> gdb version: i386-elf-gdb --verison returns GDB 5.0.
> eCos version (with Redboot)  from anon CVS  (Juli 25)

Okay, I don't know why that shouldn't work - certainly I was debugging on
the PC target from the anon cvs code just last week.

Perhaps it might behave better if the debugged app was built with no
optimization. Other than that, you'll need to investigate further (e.g. by
using "set debug remote 1" and seeing where GDB sets its "next" breakpoint
and see if that's valid).

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] gdb 'next' problem with i386 HAL

[Jonathan Larmour]

Mark Salter wrote:
> 
> Ok. And all I was saying is that I have a potential fix which doesn't just
> add 16 to the SP passed to GDB, but rather fixes the SP passed to GDB as
> part of a larger fix.

So it does change the SP - I thoguht you said it didn't :-). I guess we
might have to talk elsewhere about what the larger fix is for, because just
adding interrupt stack switching for exceptions wouldn't be that big I'm
sure (since it's already there for interrupts) so I presume there's
something else, or some complication.

> You said you might try to fix the SP and I wanted
> to let you know that it may be wasted effort. That's all.

Are you fixing it in hal_get_gdb_registers()? If so, I don't think that's
the right place because kernel exceptions should also get a
HAL_SavedRegisters with an accurate SP - it's not just the stub that's
affected.

As it happens, I already checked in a fix for this one issue (internally)
though. I also added the correction for interrupts, but it's probably not
needed actually.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] gdb 'next' problem with i386 HAL

[Mark Salter]

>>>>> Jonathan Larmour writes:

> Mark Salter wrote:
>> 
>> >>>>> Jonathan Larmour writes:
>> 
>> > So you've presumably added something to switch to the interrupt stack. Fair
>> > enough, but isn't that a separate issue? The SP value stored in the
>> > HAL_SavedRegisters is still off by 16 no matter what stack you're running
>> > on. The SP value to report to GDB must still be the value at the time of
>> > the exception, not the value just after it, no matter what.
>> 
>> I pop those values off the application stack before copying them to the
>> stub stack frame. The correct SP just comes out of that. I'm not sure
>> that simply adjusting the SP is sufficient. It may be for 'next' but
>> inferior function calls may corrupt the area under the SP passed to
>> GDB.

> I'm not saying it's _sufficient_ for inferior function calls to work. I'm
> saying it is necessary in all situations i.e. even ignoring inferior
> function calls.

Ok. And all I was saying is that I have a potential fix which doesn't just
add 16 to the SP passed to GDB, but rather fixes the SP passed to GDB as
part of a larger fix. You said you might try to fix the SP and I wanted 
to let you know that it may be wasted effort. That's all.

--Mark

Re: [ECOS] gdb 'next' problem with i386 HAL

[Jonathan Larmour]

Mark Salter wrote:
> 
> >>>>> Jonathan Larmour writes:
> 
> > So you've presumably added something to switch to the interrupt stack. Fair
> > enough, but isn't that a separate issue? The SP value stored in the
> > HAL_SavedRegisters is still off by 16 no matter what stack you're running
> > on. The SP value to report to GDB must still be the value at the time of
> > the exception, not the value just after it, no matter what.
> 
> I pop those values off the application stack before copying them to the
> stub stack frame. The correct SP just comes out of that. I'm not sure
> that simply adjusting the SP is sufficient. It may be for 'next' but
> inferior function calls may corrupt the area under the SP passed to
> GDB.

I'm not saying it's _sufficient_ for inferior function calls to work. I'm
saying it is necessary in all situations i.e. even ignoring inferior
function calls.

The correct SP simply cannot come out of any value on the stack. If you are
getting the correct SP, then I don't know how. When __default_exception_vsr
is called, SP will already be 16 bytes further down before the VSR handler
has had a chance to do the pusha - because the CPU has saved stuff on the
stack and the trampoline has saved the vector number, adding up to 16
bytes. You have to re-adjust it manually within the Hal_SavedRegisters
structure. I would imagine it's only after that point that (subject to
HAL_USE_INTERRUPT_STACK configury) you'll be able to switch stacks and do
anything to pop values off the application stack.

NB if you're playing in this area, be aware of any ramifications of
hal_fpu_push_int_annex when doing lazy FPU switching. You may not be using
that code in the configuration you are using.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] gdb 'next' problem with i386 HAL

[Mark Salter]

>>>>> Jonathan Larmour writes:

> So you've presumably added something to switch to the interrupt stack. Fair
> enough, but isn't that a separate issue? The SP value stored in the
> HAL_SavedRegisters is still off by 16 no matter what stack you're running
> on. The SP value to report to GDB must still be the value at the time of
> the exception, not the value just after it, no matter what.

I pop those values off the application stack before copying them to the
stub stack frame. The correct SP just comes out of that. I'm not sure
that simply adjusting the SP is sufficient. It may be for 'next' but
inferior function calls may corrupt the area under the SP passed to
GDB.

--Mark

 

Re: [ECOS] gdb 'next' problem with i386 HAL

[Jonathan Larmour]

Mark Salter wrote:
> 
> >>>>> Jonathan Larmour writes:
> 
> > Mark Salter wrote:
> >>
> >> >>>>> John Gumb writes:
> >>
> >> > The problem only occurs when 'nexting' over a function call.
> > [snip]
> >> > The trouble is, the return address isn't there. I had a poke around and it actually is 16 bytes further down the stack. [snip]
> >>
> >> I think this is a problem in the HAL code. The HAL is passing
> >> the wrong SP value to GDB. The problem is that the HAL stub
> >> uses the same stack as the app being debugged. The HAL should
> >> be switching to a dedicated GDB stub stack.
> 
> > Actually the problem is that on the x86 16 bytes automatically get saved on
> > the stack by the CPU before we have a chance to do anything about it. The
> > solution is to adjust the SP stored by the __default_exception_vsr into the
> > HAL_Saved_Registers struct by 16.
> 
> That still doesn't fix the underlying problem. The stub has to operate on a separate
> stack in order for inferior function calls to work.

Not a problem I was intending to solve here :-).

> GDB will make use of the area
> under the stack to build a call frame. If the HAL stub is using that stack, then
> bad things happen.

So you've presumably added something to switch to the interrupt stack. Fair
enough, but isn't that a separate issue? The SP value stored in the
HAL_SavedRegisters is still off by 16 no matter what stack you're running
on. The SP value to report to GDB must still be the value at the time of
the exception, not the value just after it, no matter what.
 
Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] gdb 'next' problem with i386 HAL

[Mark Salter]

>>>>> Jonathan Larmour writes:

> Mark Salter wrote:
>> 
>> >>>>> John Gumb writes:
>> 
>> > The problem only occurs when 'nexting' over a function call. 
> [snip] 
>> > The trouble is, the return address isn't there. I had a poke around and it actually is 16 bytes further down the stack. [snip]
>> 
>> I think this is a problem in the HAL code. The HAL is passing
>> the wrong SP value to GDB. The problem is that the HAL stub
>> uses the same stack as the app being debugged. The HAL should
>> be switching to a dedicated GDB stub stack.

> Actually the problem is that on the x86 16 bytes automatically get saved on
> the stack by the CPU before we have a chance to do anything about it. The
> solution is to adjust the SP stored by the __default_exception_vsr into the
> HAL_Saved_Registers struct by 16.

That still doesn't fix the underlying problem. The stub has to operate on a separate
stack in order for inferior function calls to work. GDB will make use of the area
under the stack to build a call frame. If the HAL stub is using that stack, then
bad things happen.

I've got a set of half-baked patches to fix GDB testsuite failures on i386. I'm
hoping to get them cleaned up and checked in this week.

--Mark

Re: [ECOS] gdb 'next' problem with i386 HAL

[Jonathan Larmour]

Mark Salter wrote:
> 
> >>>>> John Gumb writes:
> 
> > The problem only occurs when 'nexting' over a function call. 
[snip] 
> > The trouble is, the return address isn't there. I had a poke around and it actually is 16 bytes further down the stack. [snip]
> 
> I think this is a problem in the HAL code. The HAL is passing
> the wrong SP value to GDB. The problem is that the HAL stub
> uses the same stack as the app being debugged. The HAL should
> be switching to a dedicated GDB stub stack.

Actually the problem is that on the x86 16 bytes automatically get saved on
the stack by the CPU before we have a chance to do anything about it. The
solution is to adjust the SP stored by the __default_exception_vsr into the
HAL_Saved_Registers struct by 16.

I'll check something in to hopefully fix it that should be in anon cvs some
time soon.

Jifl
-- 
Red Hat, Rustat House, Clifton Road, Cambridge, UK. Tel: +44 (1223) 271062
Maybe this world is another planet's Hell -Aldous Huxley || Opinions==mine

Re: [ECOS] gdb 'next' problem with i386 HAL

[Mark Salter]

>>>>> John Gumb writes:

> Hi,
> I seem to have tracked down the cause of the problem and have a (nasty) workaround but I'm not sure why this is happening in the first place.

> The problem only occurs when 'nexting' over a function call. Immediately on entry to the called function, gdb attempts to work out the return PC address. It does this using the SAVED_PC_AFTER_CALL macro in /src/gdb/insight-5.0/gdb/config/i386/tm-i386.h invoked from step_over_function in /src/gdb/insight-5.0/gdb/infrun.c  What this does is to look on stack for the return address in order that it can set a breakpoint there. SAVED_PC_AFTER_CALL expands to

> read_memory_integer ( read_register (SP_REGNUM), 4) )

> The trouble is, the return address isn't there. I had a poke around and it actually is 16 bytes further down the stack. So I modified the above code to

> read_memory_integer ( read_register (SP_REGNUM)+0x10, 4) )

I think this is a problem in the HAL code. The HAL is passing
the wrong SP value to GDB. The problem is that the HAL stub
uses the same stack as the app being debugged. The HAL should
be switching to a dedicated GDB stub stack.

--Mark

[ECOS] gdb 'next' problem with i386 HAL

[John Gumb]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1894 bytes --]

Hi,

I seem to have tracked down the cause of the problem and have a (nasty) workaround but I'm not sure why this is happening in the first place.

The problem only occurs when 'nexting' over a function call. Immediately on entry to the called function, gdb attempts to work out the return PC address. It does this using the SAVED_PC_AFTER_CALL macro in /src/gdb/insight-5.0/gdb/config/i386/tm-i386.h invoked from step_over_function in /src/gdb/insight-5.0/gdb/infrun.c  What this does is to look on stack for the return address in order that it can set a breakpoint there. SAVED_PC_AFTER_CALL expands to

read_memory_integer ( read_register (SP_REGNUM), 4) )

The trouble is, the return address isn't there. I had a poke around and it actually is 16 bytes further down the stack. So I modified the above code to

read_memory_integer ( read_register (SP_REGNUM)+0x10, 4) )

and I appear to have correct 'next' behaviour. I'd need to do a *lot* more testing to be sure this is working properly though.

I'd be most grateful if anyone could shed light on what's actually going on. I'm guessing that another stack frame has been pushed on somehow.

FYI we can actually see this going wrong in the trace I posted:

Sending packet: $m11575c,4#33...Ack
Packet received: 01000000
Sending packet: $g#67...Ack
Packet received:
2083110004000000e0821100040000005c57110088571100207f01006caa110
0a4dd100007000000080000000000000000000000
Sending packet: $m1,1#fb...Ack
Packet received: e7
Sending packet: $X1,1:Ì#ec...Ack

The $m11575c,4 is attempting to get the return PC. The target responds with 0x00000001. After getting the registers GDB then attempts to set a break at address1 (Sending packet: $X1,1:Ì#ec...Ack). This will not be of much use.

You can imagine it was a blast debugging this :-)

Cheers,

John Gumb
Ridgeway Systems and Software
john@gumb.org (home)
jgumb@ridgeway-sys.com (work)

RE: [ECOS] gdb 'next' problem with i386 HAL

[Patrick O'Grady]

I may have some clues about gdb's next problem--as I recall, GDB does a
'next' by 1) poking a breakpoint instruction after the current statement,
and 2) sending a continue instruction to the stub.  I would guess that step
(1) isn't supported in the current i386 stub: it's likely that the earlier
GDB computed the length of the current instruction, and used a memory write
command to put the breakpoint in there, and that the current version asks
the target to do that work.  HTH...

-patrick

RE: [ECOS] gdb 'next' problem with i386 HAL

[John Gumb]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 6275 bytes --]

Folks,

>For what it's worth I get proper 'gdb next' behavior with the same
>gcc/gdb/binutil tool chain with an ancient, pre-redboot, 1.3.1 version of
>eCos.
many thanks - this is useful.

I originally saw this under RedHat 7.1 and a recent (around 10 Aug) cvs
tree.

To rule out Linux (and get a feel for how well the tools run under
NT/cygwin), 
I've managed to build the example images under NT using this morning's
ecos-latest.tar.gz snapshot from ftp://ftp.skynet.ie/cvs/ and I see exactly 
the same behaviour.

I built the i386-elf-xxx tools from source under NT successfully
(gcc-2.95.2, insight-5.0, binutils-2.10.1).

To see if I stuffed up building redboot, I've just downloaded the pre-built
redboot binary and I see exactly the same behaviour using that.

> Perhaps it might behave better if the debugged app was built with no
> optimization. Other than that, you'll need to investigate further
> (e.g. by using "set debug remote 1" and seeing where GDB sets its 
> "next" breakpoint and see if that's valid).

I tried 'set debug remote 1' but not being intimately familiar with the
innards of 
remote gdb the output didn't mean a lot. Here's a log of what I see when
'nexting'
at the initial printf in 'twothreads'. You can see I get bored and hit
CTRL-C and
we drop nicely back into gdb.

bash-2.05$ i386-elf-gdb -nw --command=~/.gdbinit
GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "--host=i686-pc-cygwin --target=i386-elf".
0x97ac in ?? ()
(gdb) load twothreads.exe
Loading section .text, size 0xc1df lma 0x108000
Loading section .rodata, size 0x2ed lma 0x1141e0
Loading section .data, size 0x230 lma 0x1144d0
Start address 0x108000 , load size 50940
Transfer rate: 67920 bits/sec, 74 bytes/write.
(gdb) symbol twothreads.exe
Reading symbols from twothreads.exe...done.
(gdb) b cyg_user_start
Breakpoint 1 at 0x10860a: file twothreads.c, line 25.
(gdb) cont
Continuing.
[New thread 1]
[Switching to thread 1]

Breakpoint 1, cyg_user_start () at twothreads.c:25
25        printf("Entering twothreads' cyg_user_start() function\n");
Current language:  auto; currently c
(gdb) info registers
eax            0x118320 1147680
ecx            0x4      4
edx            0x1182e0 1147616
ebx            0x4      4
esp            0x115770 0x115770
ebp            0x115788 0x115788
esi            0x17f20  98080
edi            0x11aa6c 1157740
eip            0x10860a 0x10860a
eflags         0x2      2
cs             0x8      8
ss             0x0      0
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
(gdb) set debug remote 1
(gdb) next
Sending packet: $s#73...Ack
Packet received: T05thread:00000001;08:0d861000;04:64571100;
Sending packet: $m10860a,1#2a...Ack
Packet received: 83
Sending packet: $X10860a,1:Ì#1b...Ack
Packet received: OK
Sending packet: $s#73...Ack
Packet received: T05thread:00000001;08:12861000;04:60571100;
Sending packet: $s#73...Ack
Packet received: T05thread:00000001;08:a4dd1000;04:5c571100;
Sending packet: $m11575c,4#33...Ack
Packet received: 01000000
Sending packet: $g#67...Ack
Packet received:
2083110004000000e0821100040000005c57110088571100207f01006caa110
0a4dd100007000000080000000000000000000000
Sending packet: $m1,1#fb...Ack
Packet received: e7
Sending packet: $X1,1:Ì#ec...Ack
Packet received: OK
Sending packet: $c#63...Ack
Packet received:
O456E746572696E672074776F7468726561647327206379675F757365725F73
7461727428292066756E6374696F6E0A
Entering twothreads' cyg_user_start() function
Packet received:
O426567696E6E696E6720657865637574696F6E3B2074687265616420646174
6120697320300A
Beginning execution; thread data is 0
Packet received:
O426567696E6E696E6720657865637574696F6E3B2074687265616420646174
6120697320310A
Beginning execution; thread data is 1
Packet received:
O54687265616420303A20616E64206E6F7720612064656C6179206F66203233
3920636C6F636B207469636B730A
Thread 0: and now a delay of 239 clock ticks
Packet received:
O54687265616420313A20616E64206E6F7720612064656C6179206F66203233
3020636C6F636B207469636B730A
Thread 1: and now a delay of 230 clock ticks
remote_interrupt called
remote_stop called
Packet received: T02thread:00000001;08:15821000;04:d05e1100;

Program received signal SIGINT, Interrupt.
Sending packet: $X1,1:ç#07...Ack
Packet received: OK
Sending packet: $g#67...Ack
Packet received:
96010000000000006058110014b91000d05e1100fc5e1100000011110000111
11582100007020000080000000000000000000000
Sending packet: $X10860a,1:\203#d2...Ack
Packet received: OK
0x108215 in hal_idle_thread_action ()
    at
/d/home/jag/ecos/packages/language/c/libc/startup/current/src/atexit.cxx:
123
123     } // atexit()
Current language:  auto; currently c++
(gdb)

-----Original Message-----
From: Allan Young [ mailto:AYoung@YottaYotta.com ]
Sent: 22 August 2001 20:20
To: ecos-discuss@sourceware.cygnus.com
Subject: RE: [ECOS] gdb 'next' problem with i386 HAL


I've been playing around with a recent (via anonymous cvs, grabbed around
August 10th) version of eCos and I've also observed the i386 gdb "next that
continues" problem too.  This is repeatable with the redboot floppy I've
built from sources as well as one created with the pre-built redboot binary
available at the redhat/ecos site.

Like Boris I'm also using gdb version 5.0 but I'm running on RedHat 7.1.
Also, the problem is repeatable when gdb is connected by serial port or
ethernet.

For what it's worth I get proper 'gdb next' behavior with the same
gcc/gdb/binutil tool chain with an ancient, pre-redboot, 1.3.1 version of
eCos.

> Perhaps it might behave better if the debugged app was built with no
> optimization. Other than that, you'll need to investigate further
> (e.g. by using "set debug remote 1" and seeing where GDB sets its 
> "next" breakpoint and see if that's valid).

It will take me a while to decode the gdb packet information from my 'gdb
next' test with debug remote set to 1.  Unfortunately, I won't have time to
look at this for at least a few days...

RE: [ECOS] gdb 'next' problem with i386 HAL

[Allan Young]

I've been playing around with a recent (via anonymous cvs, grabbed around
August 10th) version of eCos and I've also observed the i386 gdb "next that
continues" problem too.  This is repeatable with the redboot floppy I've
built from sources as well as one created with the pre-built redboot binary
available at the redhat/ecos site.

Like Boris I'm also using gdb version 5.0 but I'm running on RedHat 7.1.
Also, the problem is repeatable when gdb is connected by serial port or
ethernet.

For what it's worth I get proper 'gdb next' behavior with the same
gcc/gdb/binutil tool chain with an ancient, pre-redboot, 1.3.1 version of
eCos.

> Perhaps it might behave better if the debugged app was built with no
> optimization. Other than that, you'll need to investigate further
> (e.g. by using "set debug remote 1" and seeing where GDB sets its 
> "next" breakpoint and see if that's valid).

It will take me a while to decode the gdb packet information from my 'gdb
next' test with debug remote set to 1.  Unfortunately, I won't have time to
look at this for at least a few days...