[ECOS] Can excessive/intensive serial flow cause stack overflow?

[ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

Hi all,

Working on a new eCos application & I stumble upon a disturbing problem.

I'm working with eCos CVS on an LPC2106 platform with 2 UARTs.

I have two threads running at the same time:
- First one monitors serial requests coming to UART0,
- Second one supervise the operation of the first thread to prevent it
from blocking the system too much.

When a particular request occurs on UART0, the first thread writes
information on UART1. As information can flow really fast & UART0 line
(115200 bauds) is faster than UART1 (4800 bauds). I implemented a way
to tell the terminal making request to my UART0 to calm things a bit
if the UART1 buffer is full.

My problem is: this never happens, when the UART1 buffer is full, the
processor goes into prefetch abort and the watchdog eventually resets
everything.

The code to the first thread comes down to that (whiled):

***********************************************************************
len = rqst->byte_count;
// wake up the supervisor
cyg_cond_signal(&(hModbus->super.wake_up));
// write data to UART1
wErr = cyg_io_write(hModbus->hComT, rqst->data, &len);
// tell the supervisor we returned from write operation
cyg_cond_signal(&(hModbus->super.abort));
if(wErr != ENOERR) {
    // write operation not fulfilled
    // some action
}

// some other action
***********************************************************************

The supervisor (second thread):

***********************************************************************
while(1) {
        cyg_mutex_lock(&(hModbus->super.wake_up_mutex));
        // wait for wake up signal from serial monitor
        cyg_cond_wait(&(hModbus->super.wake_up));
            // start serial timed wait
            cyg_mutex_lock(&(hModbus->super.abort_mutex));
            state = cyg_cond_timed_wait(&(hModbus->super.abort), 1);
                // time-out
                if(state == false) {
                    len = 1;
                    cyg_io_get_config(hModbus->hCom,
CYG_IO_GET_CONFIG_SERIAL_ABORT, 0, &len);
                    len = 1;
                    cyg_io_get_config(hModbus->hComT,
CYG_IO_GET_CONFIG_SERIAL_ABORT, 0, &len);
                }
            cyg_mutex_unlock(&(hModbus->super.abort_mutex));
        cyg_mutex_unlock(&(hModbus->super.wake_up_mutex));
    }
***********************************************************************

I don't use any dynamic allocation in neither threads (never in this
program anyway) and can't see what can cause stack corruption here.
The 115200 UART0 line is stressed up to its maximum capacity and
causes those two threads to run at full speed.

What I'd like to know is if there would be any way for either of the
cyg_ functions I call to cause my problem:
- cyg_io_write(...)
- cyg_cond_signal(...)
- cyg_mutex_lock(...)
- cyg_cond_wait(...)
- cyg_cond_timed_wait(...)
- cyg_get_config(...)
- cyg_mutex_unlock(...)

I've read though the different documentations available & can't see a
way for these functions to cause stack overflow if called a lot.

Thank you very much for your time,

Alex

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Andrew Lunn]

On Thu, Jul 03, 2008 at 11:57:50AM +0200, Alexandre wrote:
> Hi all,
> 
> Working on a new eCos application & I stumble upon a disturbing problem.
> 
> I'm working with eCos CVS on an LPC2106 platform with 2 UARTs.

LPC2106 is an ARM right?

First off, i would enable asserts in the INFRA package. It adds some
CPU overheads, but that might actually help in your situation....

Next put a breakpoint in the abort handler. Once it fires, look at the
stack pointers, program counter etc for the supervisor mode and
interrupt mode registers. That should point you to what is causing the
abort. Also lr - 4 is the address which caused the prefetch abort. r14
is the lr.

       Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

On Thu, Jul 3, 2008 at 12:10 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> LPC2106 is an ARM right?

It is indeed.

> First off, i would enable asserts in the INFRA package. It adds some
> CPU overheads, but that might actually help in your situation....

My software does not start anymore when I do that. Do I have to add
something ? I read on the documentation that the execution of code can
be halted by asserts if a condition is not met. Is it the problem ?

Thanks,

Alex

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Andrew Lunn]

On Thu, Jul 03, 2008 at 03:47:25PM +0200, Alexandre wrote:
> On Thu, Jul 3, 2008 at 12:10 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> > LPC2106 is an ARM right?
> 
> It is indeed.
> 
> > First off, i would enable asserts in the INFRA package. It adds some
> > CPU overheads, but that might actually help in your situation....
> 
> My software does not start anymore when I do that. Do I have to add
> something ? I read on the documentation that the execution of code can
> be halted by asserts if a condition is not met. Is it the problem ?

Where is debug output going? Given your serial port setup, would you
see it? If you have a jtag debugger, set a breakpoint in
cyg_assert_fail and see if it hits.

      Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

> Where is debug output going? Given your serial port setup, would you
> see it? If you have a jtag debugger, set a breakpoint in
> cyg_assert_fail and see if it hits.

I have set both diagnostic & debug output to UART1.
I don't have JTAG debugging setup on this platform.
Is there a way I can check things without using JTAG from there ?

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

On Thu, Jul 3, 2008 at 4:52 PM, Alexandre <thekyz@gmail.com> wrote:
>> Where is debug output going? Given your serial port setup, would you
>> see it? If you have a jtag debugger, set a breakpoint in
>> cyg_assert_fail and see if it hits.
>
> I have set both diagnostic & debug output to UART1.
> I don't have JTAG debugging setup on this platform.
> Is there a way I can check things without using JTAG from there ?
>

Okay forget that.
It stops there:

ASSERT FAIL: (1) lpc2xxx_misc.c[472]hal_interrupt_set_level() Priority
already used by another vector

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

Sorry for the spam ...
This can be useful :

***********************************************************************************
TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
                                                       {{enter

TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
                                                         ((vector=7,
priority=4, data=40000860, isr=000049ec, dsr=00004a3c))

TRACE: intr.cxx            [ 109] Cyg_Interrupt::Cyg_Interrupt()
                                                       }}return void

TRACE: intr.cxx            [ 452] void Cyg_Interrupt::attach()
                                                       {{enter

ASSERT FAIL: <1>lpc2xxx_misc.c[472]hal_interrupt_set_level() Priority
already used by another vector

ASSERT FAIL: lpc2xxx_misc.c      [ 472] hal_interrupt_set_level()
                                                             Priority
already used by another vector

***********************************************************************************

Alex

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

On Thu, Jul 3, 2008 at 4:58 PM, Alexandre <thekyz@gmail.com> wrote:
> TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
>                                                         ((vector=7,
> priority=4, data=40000860, isr=000049ec, dsr=00004a3c))

Let's try to advance by myself :)

CYGNUM_HAL_INTERRUPT_UART1 is interrupt 7 so there must be an error around this.
Can't both UARTs use the same priority level & if not, how can I change that ?

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Andrew Lunn]

On Thu, Jul 03, 2008 at 05:06:48PM +0200, Alexandre wrote:
> On Thu, Jul 3, 2008 at 4:58 PM, Alexandre <thekyz@gmail.com> wrote:
> > TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
> >                                                         ((vector=7,
> > priority=4, data=40000860, isr=000049ec, dsr=00004a3c))
> 
> Let's try to advance by myself :)
> 
> CYGNUM_HAL_INTERRUPT_UART1 is interrupt 7 so there must be an error around this.
> Can't both UARTs use the same priority level & if not, how can I change that ?

Nope. Look at the code where the asserts are. Interrupts < 16 need to be unique. 

However, i don't quite get this:

lunn@londo:~/eCos/anoncvs-clean/packages/hal/arm/lpc2xxx$ grep -r CYGNUM_HAL_INTERRUPT_UART *
var/current/src/hal_diag.c:      CYGNUM_HAL_INTERRUPT_UART0, 
var/current/src/hal_diag.c:      CYGNUM_HAL_INTERRUPT_UART1, 
var/current/include/hal_var_ints.h:#define CYGNUM_HAL_INTERRUPT_UART0   6
var/current/include/hal_var_ints.h:#define CYGNUM_HAL_INTERRUPT_UART1   7

So the default setup is they have different interrupt values.

Does your target override the values from the variant package?

Maybe there is something more subtle going on here.

      Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

On Thu, Jul 3, 2008 at 5:24 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> Nope. Look at the code where the asserts are. Interrupts < 16 need to be unique.
>
> However, i don't quite get this:
>
> lunn@londo:~/eCos/anoncvs-clean/packages/hal/arm/lpc2xxx$ grep -r CYGNUM_HAL_INTERRUPT_UART *
> var/current/src/hal_diag.c:      CYGNUM_HAL_INTERRUPT_UART0,
> var/current/src/hal_diag.c:      CYGNUM_HAL_INTERRUPT_UART1,
> var/current/include/hal_var_ints.h:#define CYGNUM_HAL_INTERRUPT_UART0   6
> var/current/include/hal_var_ints.h:#define CYGNUM_HAL_INTERRUPT_UART1   7
>
> So the default setup is they have different interrupt values.
>
> Does your target override the values from the variant package?
>
> Maybe there is something more subtle going on here.

Sorry I should have added a bit more.
As you can see here:

**********************************************************************************************
TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
                                                         ((vector=6,
priority=4, data=4000077c, isr=000049ec, dsr=00004a3c))

// lots of stuff.....

TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
                                                         ((vector=7,
priority=4, data=40000860, isr=000049ec, dsr=00004a3c))

TRACE: intr.cxx            [ 452] void Cyg_Interrupt::attach()
                                                       {{enter

ASSERT FAIL: <1>lpc2xxx_misc.c[472]hal_interrupt_set_level() Priority
already used by another vector

ASSERT FAIL: lpc2xxx_misc.c      [ 472] hal_interrupt_set_level()
                                                             Priority
already used by another vector
**********************************************************************************************

They do have different vectors, what troubles me is that they have the
same priority, & I can't find any place where I could change that.

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Andrew Lunn]

On Thu, Jul 03, 2008 at 04:58:52PM +0200, Alexandre wrote:
> Sorry for the spam ...
> This can be useful :
> 
> ***********************************************************************************
> TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
>                                                        {{enter
> 
> TRACE: intr.cxx            [  86] Cyg_Interrupt::Cyg_Interrupt()
>                                                          ((vector=7,
> priority=4, data=40000860, isr=000049ec, dsr=00004a3c))
> 
> TRACE: intr.cxx            [ 109] Cyg_Interrupt::Cyg_Interrupt()
>                                                        }}return void
> 
> TRACE: intr.cxx            [ 452] void Cyg_Interrupt::attach()
>                                                        {{enter
> 
> ASSERT FAIL: <1>lpc2xxx_misc.c[472]hal_interrupt_set_level() Priority
> already used by another vector
> 
> ASSERT FAIL: lpc2xxx_misc.c      [ 472] hal_interrupt_set_level()
>                                                              Priority
> already used by another vector

Duh... Didn't read the message correctly, so missed the obvious problem....

I thought vector, then the problem is priority.....

This is easy to explain. The actual serial driver is in
/packages/devs/serial/generic/16x5x/current/src.

It registers the interrupt handler with:

        cyg_drv_interrupt_create(ser_chan->int_num,
                                 CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY,
                                 (cyg_addrword_t)chan,
                                 pc_serial_ISR,
                                 pc_serial_DSR,
                                 &ser_chan->serial_interrupt_handle,
                                 &ser_chan->serial_interrupt);

The priority is hard coded for all serial devices as 4. 

You can sort of override this. The same file says:

#ifndef CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY
# define CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY 4
#endif

but you don't appear to be able to override it per device, just for
all devices. So i think you are not going to be able to benefit from
vectored interrupts and will have to use a priority of 16. Or you need
to have the serial driver to allow you to specify a priority level per
device.

Humm, actually, you could do something a little horrible like:

#define  CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY \
         (ser_chan->int_num == CYGNUM_HAL_INTERRUPT_UART0 ? 4 : 5)

in arm_lpc2xxx_ser.inl 

         Andrew


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Alexandre]

On Thu, Jul 3, 2008 at 5:39 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> Duh... Didn't read the message correctly, so missed the obvious problem....
>
> I thought vector, then the problem is priority.....
>
> This is easy to explain. The actual serial driver is in
> /packages/devs/serial/generic/16x5x/current/src.
>
> It registers the interrupt handler with:
>
>        cyg_drv_interrupt_create(ser_chan->int_num,
>                                 CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY,
>                                 (cyg_addrword_t)chan,
>                                 pc_serial_ISR,
>                                 pc_serial_DSR,
>                                 &ser_chan->serial_interrupt_handle,
>                                 &ser_chan->serial_interrupt);
>
> The priority is hard coded for all serial devices as 4.
>
> You can sort of override this. The same file says:
>
> #ifndef CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY
> # define CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY 4
> #endif
>
> but you don't appear to be able to override it per device, just for
> all devices. So i think you are not going to be able to benefit from
> vectored interrupts and will have to use a priority of 16. Or you need
> to have the serial driver to allow you to specify a priority level per
> device.
>
> Humm, actually, you could do something a little horrible like:
>
> #define  CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY \
>         (ser_chan->int_num == CYGNUM_HAL_INTERRUPT_UART0 ? 4 : 5)
>
> in arm_lpc2xxx_ser.inl
>
>         Andrew

That does the trick.
The program stops there (after cyg_user_start(), some scheduler, clock
and thread work) :

************************************************************************************************
TRACE: intr.cxx            [ 619] static void Cyg_Interrupt::disable_interrupts(
)                                                        {{enter

TRACE: intr.cxx            [ 638] static void Cyg_Interrupt::disable_interrupts(
)                                                        }}return void

TRACE: intr.cxx            [ 648] static void Cyg_Interrupt::enable_interrupts()
                                                         {{enter

TRACE: intr.cxx            [ 664] static void Cyg_Interrupt::enable_interrupts()
                                                         }}return void

ASSERT FAIL: <1>hal_misc.c[155]exception_handler() Exception!!!

ASSERT FAIL: hal_misc.c          [ 155] exception_handler()
                                                             Exception!!!
************************************************************************************************

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Can excessive/intensive serial flow cause stack overflow?

[Sergei Gavrikov]

[-- Attachment #1: Type: text/plain, Size: 2104 bytes --]

Alexandre wrote:
> On Thu, Jul 3, 2008 at 5:39 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> Duh... Didn't read the message correctly, so missed the obvious problem....
>>
>> I thought vector, then the problem is priority.....
>>
>> This is easy to explain. The actual serial driver is in
>> /packages/devs/serial/generic/16x5x/current/src.
>>
>> It registers the interrupt handler with:
>>
>>        cyg_drv_interrupt_create(ser_chan->int_num,
>>                                 CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY,
>>                                 (cyg_addrword_t)chan,
>>                                 pc_serial_ISR,
>>                                 pc_serial_DSR,
>>                                 &ser_chan->serial_interrupt_handle,
>>                                 &ser_chan->serial_interrupt);
>>
>> The priority is hard coded for all serial devices as 4.
>>
>> You can sort of override this. The same file says:
>>
>> #ifndef CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY
>> # define CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY 4
>> #endif
>>
>> but you don't appear to be able to override it per device, just for
>> all devices. So i think you are not going to be able to benefit from
>> vectored interrupts and will have to use a priority of 16. Or you need
>> to have the serial driver to allow you to specify a priority level per
>> device.
>>
>> Humm, actually, you could do something a little horrible like:
>>
>> #define  CYG_IO_SERIAL_GENERIC_16X5X_INT_PRIORITY \
>>         (ser_chan->int_num == CYGNUM_HAL_INTERRUPT_UART0 ? 4 : 5)
>>
>> in arm_lpc2xxx_ser.inl
>>
>>         Andrew
> 
> That does the trick.
> The program stops there (after cyg_user_start(), some scheduler, clock
> and thread work) :

If I remember, in a past I did catch same asserts on LPC2XXX UARTS

ASSERT FAIL: (1) lpc2xxx_misc.c[472]hal_interrupt_set_level() Priority
already used by another vector

So, I just dirty tweaked the lpc2xxx_misc.c to shift priority to next
free slot. I found that my patch. Try to apply the patch and rebuild
eCos build tree. Today, I have no LPC2XXX hardware to test it.

Sergei

[-- Attachment #2: lpc2xxx_misc.c.patch --]
[-- Type: text/x-diff, Size: 1097 bytes --]

diff -ur A/hal/arm/lpc2xxx/var/current/src/lpc2xxx_misc.c B/hal/arm/lpc2xxx/var/current/src/lpc2xxx_misc.c
--- A/hal/arm/lpc2xxx/var/current/src/lpc2xxx_misc.c
+++ B/hal/arm/lpc2xxx/var/current/src/lpc2xxx_misc.c
@@ -462,12 +462,18 @@
     //
     if (level < 16)
     {
+        int i;
         cyg_uint32 addr_offset =  level << 2;
         cyg_uint32 reg_val;
         
-        HAL_READ_UINT32(CYGARC_HAL_LPC2XXX_REG_VIC_BASE + 
-                        CYGARC_HAL_LPC2XXX_REG_VICVECTCNTL0 + 
-                        addr_offset, reg_val);
+        for (i = level; i < 16; i++) {
+            addr_offset = i << 2;
+            HAL_READ_UINT32(CYGARC_HAL_LPC2XXX_REG_VIC_BASE +
+                            CYGARC_HAL_LPC2XXX_REG_VICVECTCNTL0 +
+                            addr_offset, reg_val);
+            if ((reg_val == 0) || (reg_val == (vector | 0x20)))
+                break;
+        }
         CYG_ASSERT((reg_val == 0) || (reg_val == (vector | 0x20)), 
                    "Priority already used by another vector");
         HAL_WRITE_UINT32(CYGARC_HAL_LPC2XXX_REG_VIC_BASE + 


[-- Attachment #3: Type: text/plain, Size: 148 bytes --]

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss