[ECOS] Erroneous Network Traffic Affects Ethernet Debugging on PowerPC

[ECOS] Erroneous Network Traffic Affects Ethernet Debugging on PowerPC

[Mike Arthur]

We have an in house HAL for a PowerPC target that can be fatally
affected by erroneous network traffic while debugging over the
ethernet.

If there is network traffic going to the RedBoot IP (i.e. ping
flooding the RedBoot IP) or there are network broadcasts during tests
such as heaptest, the target will become unresponsive to the point
that GDB will lose its connection to the target.  We are not sure what
is causing GDB to timeout from the target.

It appears the target gets stuck in the RedBoot ethernet code
processing packets; we have determined this by the use of a JTAG.
This of course does not happen if the host and target are on an
isolated network without any other traffic besides GDB packets going
to RedBoot.  Debugging works great over ethernet on an isolated
network.

Does anybody have any ideas of where the bug could reside?  Could the
bug be in the ethernet drivers, in the stand alone stack, somewhere in
the variant or platform HALs?

Has anyone ever ran into problems with network traffic during network
debug sessions?  What caused the problem?

Any tips on how to debug problems with ethernet debugging?

The target becomes unresponsive at different points depending on the template.


Here is how far the target gets with the 'default' template running heaptest:
---------------------------------------------------------------------------------------------------------
(gdb) load
Loading section .text, size 0x15950 lma 0x100000
Loading section .rodata, size 0x103c lma 0x115950
Loading section .eh_frame, size 0x84 lma 0x116990
Loading section .data, size 0x3a40 lma 0x116a18
Start address 0x100000, load size 107600
Transfer rate: 286933 bits/sec, 505 bytes/write.
(gdb) c
Continuing.
INFO: <code from 0x00100000 -> 0x00115950, CRC a75f>
INFO:<Starting heaptest - testing all memory usable as heap>
INFO:<Any failures reported may indicate failing RAM hardware,>
INFO:<or an invalid memory map>
INFO:<Testing memory at 0x00125c18 of size 1072538588 for 10 iterations>
PASS:<all zeroes>



Here is how far the target gets with the 'net' template running heaptest:
----------------------------------------------------------------------------------------------------
(gdb) load
Loading section .text, size 0x63958 lma 0x100000
Loading section .rodata, size 0x310c lma 0x163958
Loading section .eh_frame, size 0x84 lma 0x166a68
Loading section .data, size 0x4988 lma 0x166af0
Start address 0x100000, load size 439408
Transfer rate: 292938 bits/sec, 510 bytes/write.
(gdb) c
Continuing.
[cyg_net_init] Init: mbinit(0x00000000)
Remote communication error: No route to host.


Thanks,
Mike

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Erroneous Network Traffic Affects Ethernet Debugging on PowerPC

[Gary Thomas]

Mike Arthur wrote:
> We have an in house HAL for a PowerPC target that can be fatally
> affected by erroneous network traffic while debugging over the
> ethernet.
> 
> If there is network traffic going to the RedBoot IP (i.e. ping
> flooding the RedBoot IP) or there are network broadcasts during tests
> such as heaptest, the target will become unresponsive to the point
> that GDB will lose its connection to the target.  We are not sure what
> is causing GDB to timeout from the target.
> 
> It appears the target gets stuck in the RedBoot ethernet code
> processing packets; we have determined this by the use of a JTAG.
> This of course does not happen if the host and target are on an
> isolated network without any other traffic besides GDB packets going
> to RedBoot.  Debugging works great over ethernet on an isolated
> network.
> 
> Does anybody have any ideas of where the bug could reside?  Could the
> bug be in the ethernet drivers, in the stand alone stack, somewhere in
> the variant or platform HALs?
> 
> Has anyone ever ran into problems with network traffic during network
> debug sessions?  What caused the problem?
> 
> Any tips on how to debug problems with ethernet debugging?
> 
> The target becomes unresponsive at different points depending on the template.
> 
> 
> Here is how far the target gets with the 'default' template running heaptest:
> ---------------------------------------------------------------------------------------------------------
> (gdb) load
> Loading section .text, size 0x15950 lma 0x100000
> Loading section .rodata, size 0x103c lma 0x115950
> Loading section .eh_frame, size 0x84 lma 0x116990
> Loading section .data, size 0x3a40 lma 0x116a18
> Start address 0x100000, load size 107600
> Transfer rate: 286933 bits/sec, 505 bytes/write.
> (gdb) c
> Continuing.
> INFO: <code from 0x00100000 -> 0x00115950, CRC a75f>
> INFO:<Starting heaptest - testing all memory usable as heap>
> INFO:<Any failures reported may indicate failing RAM hardware,>
> INFO:<or an invalid memory map>
> INFO:<Testing memory at 0x00125c18 of size 1072538588 for 10 iterations>
> PASS:<all zeroes>
> 
> 
> 
> Here is how far the target gets with the 'net' template running heaptest:
> ----------------------------------------------------------------------------------------------------
> (gdb) load
> Loading section .text, size 0x63958 lma 0x100000
> Loading section .rodata, size 0x310c lma 0x163958
> Loading section .eh_frame, size 0x84 lma 0x166a68
> Loading section .data, size 0x4988 lma 0x166af0
> Start address 0x100000, load size 439408
> Transfer rate: 292938 bits/sec, 510 bytes/write.
> (gdb) c
> Continuing.
> [cyg_net_init] Init: mbinit(0x00000000)
> Remote communication error: No route to host.

It seems to me that you're describing two different problems.

The first - debugging over the network fails when the network
is saturated.

Remember that debugging via the network is a cooperating effort.
RedBoot and your eCos application share the same hardware and
if something gets unhappy in this process, then the whole thing
will fail.  Given the hardware and it's limitations (e.g. you
don't have a separate ethernet controller and stack, etc. for
debugging), then there is no other way for it to work.  One
would have to diagnose the problem, probably using an out of
band tool like JTAG.

The second problem you describe is an old one.  Again, since
the ethernet hardware is being shared, things can get confused,
especially during system startup.  What's happening is that the
eCos application is reinitializing the network hardware, but at
the same time, trying to print messages using that same hardware.
This simply can't work.  I put a change into the stack (years ago)
that side-steps this by forcing those initialization messages
to go to the raw serial console, rather than via the network.

A couple of questions:
 * What's your target?  CPU?  There are many PowerPC choices and
   most have their own drivers, etc.  It would be nice to know
   which one you are having trouble with.
 * What's the vintage of your eCos source tree.  As mentioned,
   some of these things have been worked on and may be fixed, just
   not in the sources you are using.


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Erroneous Network Traffic Affects Ethernet Debugging on PowerPC

[Mike Arthur]

>
> It seems to me that you're describing two different problems.
>
> The first - debugging over the network fails when the network
> is saturated.
>
> Remember that debugging via the network is a cooperating effort.
> RedBoot and your eCos application share the same hardware and
> if something gets unhappy in this process, then the whole thing
> will fail.  Given the hardware and it's limitations (e.g. you
> don't have a separate ethernet controller and stack, etc. for
> debugging), then there is no other way for it to work.  One
> would have to diagnose the problem, probably using an out of
> band tool like JTAG.
>
> The second problem you describe is an old one.  Again, since
> the ethernet hardware is being shared, things can get confused,
> especially during system startup.  What's happening is that the
> eCos application is reinitializing the network hardware, but at
> the same time, trying to print messages using that same hardware.
> This simply can't work.  I put a change into the stack (years ago)
> that side-steps this by forcing those initialization messages
> to go to the raw serial console, rather than via the network.
>
> A couple of questions:
>  * What's your target?
Motorola MVME6100

>   * CPU?
MPC7457 1.3GHz

> There are many PowerPC choices and
>    most have their own drivers, etc.  It would be nice to know
>    which one you are having trouble with.
We wrote the HAL and the ethernet driver.


>  * What's the vintage of your eCos source tree.  As mentioned,
>    some of these things have been worked on and may be fixed, just
>    not in the sources you are using.
>
We are currently using a release from eCosCentric v2.0.51.  We were
able to compile a GRUB RedBoot i386 target w/i82559 driver.  The i386
target ran heaptest (net template), just a little slower than normal.
We did not test the "default" target.


Thanks for your help,
Mike

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Erroneous Network Traffic Affects Ethernet Debugging on PowerPC

[Mike Arthur]

> The second problem you describe is an old one.  Again, since
> the ethernet hardware is being shared, things can get confused,
> especially during system startup.  What's happening is that the
> eCos application is reinitializing the network hardware, but at
> the same time, trying to print messages using that same hardware.
> This simply can't work.  I put a change into the stack (years ago)
> that side-steps this by forcing those initialization messages
> to go to the raw serial console, rather than via the network.
>

I think I found the config option that you speak of:
CYGPKG_NET_FORCE_SERIAL_CONSOLE.  Once I set this option, the
'default' and the 'net' templates behaved the same.  Thank you!

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss