RedBoot built with gcc 3.4.4

RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

Hi all

I've built redboot for an XSCALE PXA270 based board from eCos CVS snapshot 
around December 2006. What built with gcc 3.3.2 binutils 2.14.90 redboot 
works fine. When built with gcc 3.4.4 binutils 2.15.96 it comes to

Reset caused by: hardware reset pin
Core voltage will be set to 1.150 Volts.
Clock speed will be changed to 208 MHz
clock speed set to 208 MHz ...
+

and then comes a string

$T0a0f:58ab00a0;0d:34d500a0;#0b

and it keeps repeating on every ENTER. I was told that this string comes 
from some debugging mode, which is invoked due to some unrecognised 
exception. Here are the used flags

-O2 -ffunction-sections -fdata-sections -fno-rtti -fno-builtin \
-fno-exceptions -fvtable-gc -finit-priority -mapcs-frame

Originally there was also a "-g" there, which I now removed, and haven't 
yet checked if the problem is still there, but I think it is not the 
reason.

Is the problem known to anybody / how do we solve it?

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Andrew Lunn]

On Wed, Feb 07, 2007 at 12:52:43PM +0100, Guennadi Liakhovetski wrote:
> Hi all
> 
> I've built redboot for an XSCALE PXA270 based board from eCos CVS snapshot 
> around December 2006. What built with gcc 3.3.2 binutils 2.14.90 redboot 
> works fine. When built with gcc 3.4.4 binutils 2.15.96 it comes to
> 
> Reset caused by: hardware reset pin
> Core voltage will be set to 1.150 Volts.
> Clock speed will be changed to 208 MHz
> clock speed set to 208 MHz ...
> +
> 
> and then comes a string
> 
> $T0a0f:58ab00a0;0d:34d500a0;#0b
> 
> and it keeps repeating on every ENTER. I was told that this string comes 
> from some debugging mode, which is invoked due to some unrecognised 
> exception.

Correct. Use gdb and find out where it crashed. It can decode the
string to give you an address.

You might also want to try the old gcc with the new binutils, and the
new gcc with the old binutils etc to find out which is the problem.

    Andrew

Re: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

On Wed, 7 Feb 2007, Andrew Lunn wrote:

>> $T0a0f:58ab00a0;0d:34d500a0;#0b
>>
>> and it keeps repeating on every ENTER. I was told that this string comes
>> from some debugging mode, which is invoked due to some unrecognised
>> exception.
>
> Correct. Use gdb and find out where it crashed. It can decode the
> string to give you an address.

Ok, a short look through eCos / RedBoot docs didn't reveal anything. Could 
you, please, give some short instructions? I used gdb also remotely, but 
with the bdi2000 stub... And never heard of such cryptic lines. How do I 
debug it? I'll try to look further.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Andrew Lunn]

On Wed, Feb 07, 2007 at 01:43:35PM +0100, Guennadi Liakhovetski wrote:
> On Wed, 7 Feb 2007, Andrew Lunn wrote:
> 
> >>$T0a0f:58ab00a0;0d:34d500a0;#0b
> >>
> >>and it keeps repeating on every ENTER. I was told that this string comes
> >>from some debugging mode, which is invoked due to some unrecognised
> >>exception.
> >
> >Correct. Use gdb and find out where it crashed. It can decode the
> >string to give you an address.
> 
> Ok, a short look through eCos / RedBoot docs didn't reveal anything. Could 
> you, please, give some short instructions? I used gdb also remotely, but 
> with the bdi2000 stub... And never heard of such cryptic lines. How do I 
> debug it? I'll try to look further.

http://ecos.sourceware.org/docs-latest/user-guide/user-guide-installation-target.html

Connect the serial port of your PC to the serial port of your device.
start gdb with the elf or redboot.

gdb redboot.elf
target remote /dev/ttyS0
bt

        Andrew

Re: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

On Wed, 7 Feb 2007, Andrew Lunn wrote:

> gdb redboot.elf
> target remote /dev/ttyS0
> bt

Ok, thanks. Here:

(gdb) bt
#0  0xa0011bb4 in workspace_start ()
#1  0x18e59ff0 in ?? ()
#2  0xa000d79c in __startup_stack_base ()
Cannot access memory at address 0xe3e85dd9

Will look at that workspace_start, if I can identify anything...

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Gary Thomas]

Guennadi Liakhovetski wrote:
> On Wed, 7 Feb 2007, Andrew Lunn wrote:
> 
>> gdb redboot.elf
>> target remote /dev/ttyS0
>> bt
> 
> Ok, thanks. Here:
> 
> (gdb) bt
> #0  0xa0011bb4 in workspace_start ()
> #1  0x18e59ff0 in ?? ()
> #2  0xa000d79c in __startup_stack_base ()
> Cannot access memory at address 0xe3e85dd9
> 
> Will look at that workspace_start, if I can identify anything...

Those are all data addresses (that RedBoot uses), so it
looks like it has jumped into space...

You'll be better off restarting it using GDB and step
through to find out where it goes wrong.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

On Wed, 7 Feb 2007, Gary Thomas wrote:

> You'll be better off restarting it using GDB and step
> through to find out where it goes wrong.

hm, that requires gdb support from the redboot, which I am not sure if I 
have. It crashes somewhere inside bootp. "+" is where it starts sending 
bootp requests. So, I could set a breakpoint on __bootp_find_local_ip (is 
this the right function for it?), but how do I reset the board from under 
gdb? and will it directly contact the gdb stub on the board and set the 
breakpoint?

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Gary Thomas]

Guennadi Liakhovetski wrote:
> On Wed, 7 Feb 2007, Gary Thomas wrote:
> 
>> You'll be better off restarting it using GDB and step
>> through to find out where it goes wrong.
> 
> hm, that requires gdb support from the redboot, which I am not sure if I 
> have. It crashes somewhere inside bootp. "+" is where it starts sending 
> bootp requests. So, I could set a breakpoint on __bootp_find_local_ip 
> (is this the right function for it?), but how do I reset the board from 
> under gdb? and will it directly contact the gdb stub on the board and 
> set the breakpoint?

To debug RedBoot, you'll need to use your BDI.

Simply connect GDB to the BDI instead of the serial port.
Something like this:
   (gdb) tar rem bdi:2001

where 'bdi' is the DNS name or IP of your unit.  It has commands
for resetting the device, setting hardware breakpoints, etc.  Read
the BDI manual for the details.

Then you can step through the RedBoot code to find out where it's
going bad.


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

On Wed, 7 Feb 2007, Gary Thomas wrote:

> To debug RedBoot, you'll need to use your BDI.
>
> Simply connect GDB to the BDI instead of the serial port.
> Something like this:
>  (gdb) tar rem bdi:2001

Aha, ok, this makes sense, but, unfortunately, bdi doesn't work with this 
board. It worked somehow with pxa255 (not very reliably) and I haven't 
been able to use it with pxa270. Yes, I do have the necessary register 
description etc. files, just it wouldn't work. So, I guess, it's not an 
option for me. I could disable bootp, but the point was not to somehow get 
it to build and run, but to find the reason of the mis-compilation and 
(preferrably) fix redboot. Besides, every non-booting redboot means about 
20 minutes jtag flashing...

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Gary Thomas]

Guennadi Liakhovetski wrote:
> On Wed, 7 Feb 2007, Gary Thomas wrote:
> 
>> To debug RedBoot, you'll need to use your BDI.
>>
>> Simply connect GDB to the BDI instead of the serial port.
>> Something like this:
>>  (gdb) tar rem bdi:2001
> 
> Aha, ok, this makes sense, but, unfortunately, bdi doesn't work with 
> this board. It worked somehow with pxa255 (not very reliably) and I 
> haven't been able to use it with pxa270. Yes, I do have the necessary 
> register description etc. files, just it wouldn't work. So, I guess, 
> it's not an option for me. I could disable bootp, but the point was not 
> to somehow get it to build and run, but to find the reason of the 
> mis-compilation and (preferrably) fix redboot. Besides, every 
> non-booting redboot means about 20 minutes jtag flashing...

In that case, what I would do is install a working RedBoot
(built with 3.3.2) and then experiment with a RAM version
built with the new tools.  Put in some diag_printf() to
trace the code.  You might even be able to debug this
using the serial connection and GDB (since that does
seem to be alive in your ROM/ROMRAM version)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

On Wed, 7 Feb 2007, Gary Thomas wrote:

> In that case, what I would do is install a working RedBoot
> (built with 3.3.2) and then experiment with a RAM version
> built with the new tools.  Put in some diag_printf() to
> trace the code.  You might even be able to debug this
> using the serial connection and GDB (since that does
> seem to be alive in your ROM/ROMRAM version)

Well, yeah, looked shortly at this, but RAM build is not supported on this 
plaform and a quick try wasn't successful.

OTOH, I realised, that I, probably, was wrong about where it crashes. The 
last messages that I see are from CPU clock switching code. I looked in 
those disassembled functions - they are VERY different. And I noticed that 
RedBoot / eCos doesn't specify any ARM version specific options. This 
toolchain builds kernels just fine, but the kernel does provide a few 
CPU-specific options. It also compiles user-space applications fine 
without any options - but that is user-space. So, there might be a problem 
with some supervisor mode insn. I tried adding "-mapcs-frame 
-march=armv5te -mtune=xscale -Wa,-mcpu=xscale" to c-flags, it didn't help.

Looking further

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

On Fri, 9 Feb 2007, Guennadi Liakhovetski wrote:

> Looking further

Auch... I came as far as to the loop

/*------------------------------------------------------------------------*/
/* C++ support - run initial constructors                                 */

#ifdef CYGSEM_HAL_STOP_CONSTRUCTORS_ON_FLAG
cyg_bool cyg_hal_stop_constructors;
#endif

typedef void (*pfunc) (void);
extern pfunc __CTOR_LIST__[];
extern pfunc __CTOR_END__[];

void
cyg_hal_invoke_constructors (void)
{
#ifdef CYGSEM_HAL_STOP_CONSTRUCTORS_ON_FLAG
     static pfunc *p = &__CTOR_END__[-1];

     cyg_hal_stop_constructors = 0;
     for (; p >= __CTOR_LIST__; p--) {
 	    diag_printf("Invoking constructor @ 0x%08x\n", *p);
         (*p) ();
         if (cyg_hal_stop_constructors) {
             p--;
             break;
         }
     }
#else
     pfunc *p;

     for (p = &__CTOR_END__[-1]; p >= __CTOR_LIST__; p--) {
 	    diag_printf("Invoking constructor @ 0x%08x\n", *p);
         (*p) ();
     }
#endif

And put the diag_printf() there... And it prints 1 address and that's 
it... So, do I understand it right that here it is supposed to call C++ 
constructors?... Oh, no... and already in the first one it crashes... So, 
some C++ internal calling conventions have changed. I am pretty helpless 
in what concerns C++ ABI... Anyone?

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

Hm, can it be "-finit-priority". It errors in 3.4.4, so, I took it out, 
which works fine with other "recent" compilers. And here:
http://sources.redhat.com/ml/ecos-devel/2003-10/msg00061.html

<quote>
     Gary> That's really too bad - we can't take it out globally
     Gary> because that could possibly break things for folks using
     Gary> older compilers and we may have troubles using newer ones..
 				^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</quote>

It has been verified that it is not needed on even 2.95.x and taken away 
from in-tree platforms. Can it be that we DO have a problem now?

Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

RE: RedBoot built with gcc 3.4.4

[Kurt Siedenburg]

I'd be interested in the answer to this question also - although our
situation is different:
- We have seen the same crash you observed.
However
- It happens *very* rarely.
  Over a period of 12 months I've seen it 3 times.
  Thus in >99.9(9)% of bootups it works fine.
  However - once it gets into this state - chances are it can be
repro'd.
- We have no idea how to reproduce the problem.
- We use a pretty old version of ecos (1.3 based).
- We use gcc 2.95.3
- Our CPU is Xscale

Looking at the differences (eg. reproducibility) I think that the root
cause of our problems is likely to be different.  Nevertheless I'd be
interested in the solution to your problem.  Maybe it applies to us
also.
Just looking at our symptoms I'd think there's some uninitialized
variable somewhere in early ecos startup.  And most of the times it
contains a value not causing trouble.  Only occasionally ... .


Regards,
  Kurt Siedenburg


-----Original Message-----
From: ecos-devel-owner@ecos.sourceware.org
[mailto:ecos-devel-owner@ecos.sourceware.org] On Behalf Of Guennadi
Liakhovetski
Sent: Friday, February 09, 2007 1:53 AM
To: Gary Thomas
Cc: ecos-devel@ecos.sourceware.org
Subject: Re: RedBoot built with gcc 3.4.4

On Fri, 9 Feb 2007, Guennadi Liakhovetski wrote:

> Looking further

Auch... I came as far as to the loop

/*----------------------------------------------------------------------
--*/
/* C++ support - run initial constructors
*/

#ifdef CYGSEM_HAL_STOP_CONSTRUCTORS_ON_FLAG
cyg_bool cyg_hal_stop_constructors;
#endif

typedef void (*pfunc) (void);
extern pfunc __CTOR_LIST__[];
extern pfunc __CTOR_END__[];

void
cyg_hal_invoke_constructors (void)
{
#ifdef CYGSEM_HAL_STOP_CONSTRUCTORS_ON_FLAG
     static pfunc *p = &__CTOR_END__[-1];

     cyg_hal_stop_constructors = 0;
     for (; p >= __CTOR_LIST__; p--) {
 	    diag_printf("Invoking constructor @ 0x%08x\n", *p);
         (*p) ();
         if (cyg_hal_stop_constructors) {
             p--;
             break;
         }
     }
#else
     pfunc *p;

     for (p = &__CTOR_END__[-1]; p >= __CTOR_LIST__; p--) {
 	    diag_printf("Invoking constructor @ 0x%08x\n", *p);
         (*p) ();
     }
#endif

And put the diag_printf() there... And it prints 1 address and that's
it... So, do I understand it right that here it is supposed to call C++
constructors?... Oh, no... and already in the first one it crashes...
So, some C++ internal calling conventions have changed. I am pretty
helpless in what concerns C++ ABI... Anyone?

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

RE: RedBoot built with gcc 3.4.4

[Guennadi Liakhovetski]

On Mon, 12 Feb 2007, Kurt Siedenburg wrote:

> I'd be interested in the answer to this question also - although our
> situation is different:

Yes, looks like your situation is very different. You have a runtime 
problem, whereas I have a compile / link-time one.

...and I am afraid I am not much nearer to the solution now than before. I 
thought I identified the exact crash location with the help of debugging 
printf's, but it looks like the crash / run behaviour differs also with 
the amount and kind of debugging.

Now what I currently cannot understand is how one is supposed to handle 
the .fixed_vectors section (ARM). This section contains some objects, 
which are allocated in packages/hal/arm/arch/current/src/vectors.S 
(although, I don't see any need for assembly, couldn't one just use an 
appropriate __attribute__ and define those objects in a C file?) and then 
should be described in the system linker script using the macro:

#define SECTION_fixed_vectors(_region_, _vma_, _lma_) \
     .fixed_vectors _vma_ : _lma_ \
     { FORCE_OUTPUT; KEEP (*(.fixed_vectors)) } \
     > _region_

in packages/hal/arm/arch/current/src/arm.ld. Now, those objects will be 
written at run time, so, the section has to be in RAM. However, I do not 
understand how this is achieved? As you dont have any loader, you have to 
initialise it explicitely like the .bss or the .data, which I can 
perfectly recognize in the above 2 files - as _start and _end addresses 
are defined, and then the area between them is either cleared (for .bss) 
or copied from the original image location (.data). But I don't understand 
how this is supposed to work with .fixed_vectors.

Please, explain.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: RedBoot built with gcc 3.4.4

[Edgar Grimberg]

> 
> Ok, a short look through eCos / RedBoot docs didn't reveal anything. 
> Could you, please, give some short instructions? I used gdb also 
> remotely, but with the bdi2000 stub... And never heard of such cryptic 
> lines. How do I debug it? I'll try to look further.
> 

Replace the

target remote <IP>:2001

with

target remote <serial_channel>

Using gdb:

(gdb) help target remote
Use a remote computer via a serial line, using a gdb-specific protocol.
Specify the serial device it is connected to
(e.g. /dev/ttyS0, /dev/ttya, COM1, etc.).

Edgar

PS: if you have the BDI2000 connected to the target, you better use that 
instead of serial debugging.