[ECOS] High priority thread versus network

[ECOS] High priority thread versus network

[Wayne Visser]

Hello all,

We're having a problem with an eCos app that has a relatively 
long-running, high priority thread (runs at priority 2 every 10 ms and 
takes about 4ms to complete).  Under high network loads, the app will 
crash with no asserts or panics.  If the high priority thread is 
disabled, the app will run fine for days without problem under high net 
loads.  Conversely, without any networking activity, the app runs fine 
for days.

We've stripped this down to a simple test app with two parts (a) the 
high priority thread basically does nothing but consume CPU time:

static void
high_thread( cyg_addrword_t arg )
{
   int i, j;
   cyg_uint16 buf[1024];

   while (1)
   {
     for ( j=0; j<90; ++j )
     {
       for ( i=0; i<1024; ++i)
         buf[i] = rand();
     }

     cyg_thread_delay(1);
   }
}

and (b) several identical networking threads that do nothing but accept 
client connections and echo data sent to them.

If several clients connect to the eCos app, a crash will occur in as 
little as a few minutes (but sometimes hours).

So my question is this: Are there any known issues in running a high 
priority thread with a relatively long running time?  i.e. this thread 
is effectively blocking the network threads from running for up to 4ms. 
  Will that create any known problems?

Thanks for any feedback.

   -- Wayne


ps: Out target is i386 and the problem is evident with both 8139 and 
82559 ethernet drivers.  Curiously, the problem does NOT appear with the 
83816 ethernet driver.


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] High priority thread versus network

[Robin Randhawa]

Hi.

Just some shots in the dark :

1. Have you enabled all assertions ? Checks for stack manipulation ?

2. Are you using a separate interrupt stack ?

3. Does changing the default stack size of the Network Thread make a
difference in either the exhibited phenomena and/or the time before the
system hangs up ?

The problem you face seems to be a stack overflow but there really isn't
sufficient data to state that as a fact.

I would try the above just to reduce some of the possibilities.

Cheers,
Robin

On Mon, 2007-05-07 at 08:53 -0400, Wayne Visser wrote:
> Hello all,
> 
> We're having a problem with an eCos app that has a relatively 
> long-running, high priority thread (runs at priority 2 every 10 ms and 
> takes about 4ms to complete).  Under high network loads, the app will 
> crash with no asserts or panics.  If the high priority thread is 
> disabled, the app will run fine for days without problem under high net 
> loads.  Conversely, without any networking activity, the app runs fine 
> for days.
> 
> We've stripped this down to a simple test app with two parts (a) the 
> high priority thread basically does nothing but consume CPU time:
> 
> static void
> high_thread( cyg_addrword_t arg )
> {
>    int i, j;
>    cyg_uint16 buf[1024];
> 
>    while (1)
>    {
>      for ( j=0; j<90; ++j )
>      {
>        for ( i=0; i<1024; ++i)
>          buf[i] = rand();
>      }
> 
>      cyg_thread_delay(1);
>    }
> }
> 
> and (b) several identical networking threads that do nothing but accept 
> client connections and echo data sent to them.
> 
> If several clients connect to the eCos app, a crash will occur in as 
> little as a few minutes (but sometimes hours).
> 
> So my question is this: Are there any known issues in running a high 
> priority thread with a relatively long running time?  i.e. this thread 
> is effectively blocking the network threads from running for up to 4ms. 
>   Will that create any known problems?
> 
> Thanks for any feedback.
> 
>    -- Wayne
> 
> 
> ps: Out target is i386 and the problem is evident with both 8139 and 
> 82559 ethernet drivers.  Curiously, the problem does NOT appear with the 
> 83816 ethernet driver.
> 
> 


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] High priority thread versus network

[Wayne Visser]

Thanks for the tips, Robin.  You may be shooting in the dark, but I *am* 
in the dark! ;-)

You're right, it does look like a stack overflow, but I have assertions 
enabled (CYGDBG_USE_ASSERTS) and CYGFUN_KERNEL_THREADS_STACK_CHECKING is 
also enabled.  I've seen no asserts or panics being raised to date.  Are 
there other assertions I've missed?

I'm using a separate interrupt stack (size = 4096).

I've gone ahead and increased the size of the NET_THREAD and 
NET_FAST_THREAD and am currently re-running the tests.

   -- Wayne

Robin Randhawa wrote:
> Hi.
> 
> Just some shots in the dark :
> 
> 1. Have you enabled all assertions ? Checks for stack manipulation ?
> 
> 2. Are you using a separate interrupt stack ?
> 
> 3. Does changing the default stack size of the Network Thread make a
> difference in either the exhibited phenomena and/or the time before the
> system hangs up ?
> 
> The problem you face seems to be a stack overflow but there really isn't
> sufficient data to state that as a fact.
> 
> I would try the above just to reduce some of the possibilities.
> 
> Cheers,
> Robin
> 
> On Mon, 2007-05-07 at 08:53 -0400, Wayne Visser wrote:
>> Hello all,
>>
>> We're having a problem with an eCos app that has a relatively 
>> long-running, high priority thread (runs at priority 2 every 10 ms and 
>> takes about 4ms to complete).  Under high network loads, the app will 
>> crash with no asserts or panics.  If the high priority thread is 
>> disabled, the app will run fine for days without problem under high net 
>> loads.  Conversely, without any networking activity, the app runs fine 
>> for days.
>>
>> We've stripped this down to a simple test app with two parts (a) the 
>> high priority thread basically does nothing but consume CPU time:
>>
>> static void
>> high_thread( cyg_addrword_t arg )
>> {
>>    int i, j;
>>    cyg_uint16 buf[1024];
>>
>>    while (1)
>>    {
>>      for ( j=0; j<90; ++j )
>>      {
>>        for ( i=0; i<1024; ++i)
>>          buf[i] = rand();
>>      }
>>
>>      cyg_thread_delay(1);
>>    }
>> }
>>
>> and (b) several identical networking threads that do nothing but accept 
>> client connections and echo data sent to them.
>>
>> If several clients connect to the eCos app, a crash will occur in as 
>> little as a few minutes (but sometimes hours).
>>
>> So my question is this: Are there any known issues in running a high 
>> priority thread with a relatively long running time?  i.e. this thread 
>> is effectively blocking the network threads from running for up to 4ms. 
>>   Will that create any known problems?
>>
>> Thanks for any feedback.
>>
>>    -- Wayne
>>
>>
>> ps: Out target is i386 and the problem is evident with both 8139 and 
>> 82559 ethernet drivers.  Curiously, the problem does NOT appear with the 
>> 83816 ethernet driver.
>>
>>
> 
> .
> 

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] High priority thread versus network - FOLLOW-UP

[Wayne Visser]

Wayne Visser wrote:
> Hello all,
> 
> We're having a problem with an eCos app that has a relatively 
> long-running, high priority thread (runs at priority 2 every 10 ms and 
> takes about 4ms to complete).  Under high network loads, the app will 
> crash with no asserts or panics.  If the high priority thread is 
> disabled, the app will run fine for days without problem under high net 
> loads.  Conversely, without any networking activity, the app runs fine 
> for days.
> <snip>


Hello all,

This is a follow-up to some mysterious crashes we were seeing related to 
network activity.  Related posts are here:

https://bugzilla.ecoscentric.com/show_bug.cgi?id=1000403
http://sourceware.org/ml/ecos-discuss/2007-03/msg00024.html
http://sourceware.org/ml/ecos-discuss/2007-05/msg00046.html

This was seen on an i386 platform (Advantech PCM3370) and we noticed 
that when the AGP aperture was reduced the observed problem apparently 
'disappeared' leaving me to think we had some type of memory conflict 
between the aperture and the ethernet card.

Whether or not a high-priority thread was running and stealing time away 
from the networking threads turned out not be causal - network activity 
on its own was enough to cause crashes.  No asserts or panics were 
raised at a crash (apart from me perhaps :-0

It's not entirely clear why a change (specifically a reduction) in AGP 
aperture eliminates the assumed memory conflict.  We also observed that 
crashing was most frequent when the aperture was set to 1/2 of the main 
memory size.

The board uses a Via VT8606 Northbridge (ProSavage PN133T) and on our 
board's BIOS, it's possible to reduce but not completely disable the 
aperture so we did some research and have a method to disable it 
programmatically, which is probably better anyway.

Since this Northbridge is fairly common in PC104 boards, maybe someone 
else is seeing crashes in a similar way so here's how we ended up 
disabling the aperture.  NOTE:  This is not a totally satisfying fix, 
since we don't completely understand the problem, but in the 2 months 
that we've started doing this we have not recorded a single crash on our 
test boards.


// *******************************************************************
#include <cyg/io/pci.h>

// <snip>
// ...
// ...
// ...

// *******************************************************************
#define DEBUG_VT8606_SETUP      0

// <snip>
// ...
// ...
// ...


// *******************************************************************
// device/vendor id matching function for VT8606 Northbridge
//
static cyg_bool
pci_find_match_func(cyg_uint16 v, cyg_uint16 d, cyg_uint32 c, void *p)
{
   // vendor ID for Via Technologies = 0x1106
   // device ID for VT8606 = 0x0605
   return ((v == 0x1106) && (d == 0x0605));
}


// *******************************************************************
// Disable graphics aperture feature in Via VT8606 Northbridge.  Call
// this function as soon after startup as possible (i.e. before setting
// up PCI devices).  This function is benign if no VT8606 exists.
//
static void
chipset_init(void)
{
   cyg_pci_device_id pci_device_id;
   cyg_pci_device    pci_device_info;
#if DEBUG_VT8606_SETUP > 0
   cyg_uint8         b;
#endif
   cyg_uint32        dw;

   pci_device_id = CYG_PCI_NULL_DEVID;
   cyg_pci_init();
   if ( cyg_pci_find_matching(&pci_find_match_func, NULL,
                                &pci_device_id))
   {
     cyg_pci_get_device_info(pci_device_id, &pci_device_info);

     if (cyg_pci_configure_device(&pci_device_info))
     {

       // read GA base
#if DEBUG_VT8606_SETUP > 0
       cyg_pci_read_config_uint32(pci_device_info.devid, 0x10, &dw);
       diag_printf(" GA BASE (0x88):    0x%08x\n", dw);
#endif

       // read TLB and disable aperture
       cyg_pci_read_config_uint32(pci_device_info.devid, 0x88, &dw);
#if DEBUG_VT8606_SETUP > 0
       diag_printf(" GA TLB (0x88):    0x%08x\n", dw);
#endif
       dw &= ~2;
       cyg_pci_write_config_uint32(pci_device_info.devid, 0x88, dw);
#if DEBUG_VT8606_SETUP > 0
       cyg_pci_read_config_uint32(pci_device_info.devid, 0x88, &dw);
       diag_printf(" GA TLB (after disabling aperture) (0x88): 
0x%08x\n", dw);
#endif

       // read aperture size and set to 0
#if DEBUG_VT8606_SETUP > 0
       cyg_pci_read_config_uint8(pci_device_info.devid, 0x84, &b);
       diag_printf(" Aperture Size (0x84):    0x%02x\n", b);
#endif
       cyg_pci_write_config_uint8(pci_device_info.devid, 0x84, 0);
#if DEBUG_VT8606_SETUP > 0
       cyg_pci_read_config_uint8(pci_device_info.devid, 0x84, &b);
       diag_printf(" Aperture Size (after setting to 0) (0x84): 
0x%02x\n", b);
#endif
     }
   }
}

// *******************************************************************

--
Wayne Visser
LSZ PaperTech Inc.




-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss