Re: [ECOS] pbuf_alloc failures with LwIP

[Elad Yosef <elad.yosef@gmail.com>]

Hi,
I'm using multiple threads in my application.
The SYS_ARCH_PROTECT is not enabled at all.
How do I  enable it?
Is it enabled in the current CVS version?
Does the current LwIP in CVS is compatible with ecos-3.0? In yes I
would consider upgrading LwIP package only.

Thanks
Elad


On Thu, Jun 14, 2012 at 5:38 PM, Michael O'Dowd
<michael.odowd@kuantic.com> wrote:
> Hi Elad,
>
> Hmm, I've had a quick look at the pbuf management in eCos 3.0. It's quite
> different from the CVS version, so I'm not that familiar with it.
>
> Nonetheless, I'm surprised by the PBUF statistics:
>
>  PBUF - "each pbuf is 1024 bytes"
>          avail: 30
>          used: 1
>          max: 11
>          err: 2
>          alloc_locked: 0
>          refresh_locked: 0
>
> There's something wrong here. Considering that "alloc_locked = 0", the only
> way for "err" to be incremented is if you run out of pbufs. However, the
> sign that you have run out of pbufs is that "max" equals "avail". Yet, in
> your case, max = 11, while avail = 30. So you didn't run out of pbufs, you
> only used 11 out of 30.
>
> Digging a bit more, it appears that "err" in increased when
> pbuf_pool_alloc() returns NULL. This happens when the linked-list of
> available pbufs is empty.
>
> So, how come the linked-list of available pbufs is empty when max = 11? In
> my opinion, the linked-list of available pbufs is corrupt or truncated.
>
> Are you sure that you're respecting the thread-safe requirements of lwIP?
> Are you using multiple threads? If so, make sure that the SYS_ARCH_PROTECT
> macro (in lwip/sys.h) is defined to do something useful, rather than being
> an empty definition.
>
> Regards,
>
> Michael.
>
> On 14/06/2012 06:43, Elad Yosef wrote:
>>
>> Hi Michael,
>> Thanks for the detailed reply.
>>
>> I think I have exactly the same problem that you have - the networking
>> stops working.
>>
>> I got the LwIP stats after the networking stopped working, see
>>
>>
>>
>> LINK
>>         xmit: 0
>>         rexmit: 0
>>         recv: 0
>>         fw: 0
>>         drop: 0
>>         chkerr: 0
>>         lenerr: 0
>>         memerr: 0
>>         rterr: 0
>>         proterr: 0
>>         opterr: 0
>>         err: 0
>>         cachehit: 0
>>
>> IP_FRAG
>>         xmit: 0
>>         rexmit: 0
>>         recv: 0
>>         fw: 0
>>         drop: 0
>>         chkerr: 0
>>         lenerr: 0
>>         memerr: 0
>>         rterr: 0
>>         proterr: 0
>>         opterr: 0
>>         err: 0
>>         cachehit: 0
>>
>> IP
>>         xmit: 17643
>>         rexmit: 0
>>         recv: 63100
>>         fw: 0
>>         drop: 0
>>         chkerr: 0
>>         lenerr: 0
>>         memerr: 0
>>         rterr: 0
>>         proterr: 0
>>         opterr: 0
>>         err: 0
>>         cachehit: 0
>>
>> ICMP
>>         xmit: 2775
>>         rexmit: 0
>>         recv: 2950
>>         fw: 0
>>         drop: 175
>>         chkerr: 0
>>         lenerr: 0
>>         memerr: 0
>>         rterr: 0
>>         proterr: 175
>>         opterr: 0
>>         err: 0
>>         cachehit: 0
>>
>> UDP
>>         xmit: 4714
>>         rexmit: 0
>>         recv: 53209
>>         fw: 0
>>         drop: 0
>>         chkerr: 0
>>         lenerr: 0
>>         memerr: 0
>>         rterr: 0
>>         proterr: 0
>>         opterr: 0
>>         err: 0
>>         cachehit: 0
>>
>> TCP
>>         xmit: 6715
>>         rexmit: 0
>>         recv: 6941
>>         fw: 0
>>         drop: 0
>>         chkerr: 0
>>         lenerr: 0
>>         memerr: 2705
>>         rterr: 0
>>         proterr: 0
>>         opterr: 0
>>         err: 0
>>         cachehit: 0
>>
>> PBUF - "each pbuf is 1024 bytes"
>>         avail: 30
>>         used: 1
>>         max: 11
>>         err: 2
>>         alloc_locked: 0
>>         refresh_locked: 0
>>
>>  MEM HEAP
>>         avail: 1024
>>         used: 0
>>         max: 720
>>         err: 0
>>
>>  MEM PBUF
>>         avail: 8
>>         used: 0
>>         max: 2
>>         err: 0
>>
>>  MEM RAW_PCB
>>         avail: 4
>>         used: 0
>>         max: 0
>>         err: 0
>>
>>  MEM UDP_PCB
>>         avail: 3
>>         used: 3
>>         max: 3
>>         err: 0
>>
>>  MEM TCP_PCB
>>         avail: 16
>>         used: 0
>>         max: 8
>>         err: 0
>>
>>  MEM TCP_PCB_LISTEN
>>         avail: 1
>>         used: 1
>>         max: 1
>>         err: 0
>>
>>  MEM TCP_SEG
>>         avail: 6
>>         used: 0
>>         max: 4
>>         err: 0
>>
>>  MEM NETBUF
>>         avail: 10
>>         used: 0
>>         max: 6
>>         err: 0
>>
>>  MEM NETCONN
>>         avail: 12
>>         used: 4
>>         max: 7
>>         err: 0
>>
>>  MEM API_MSG
>>         avail: 6
>>         used: 0
>>         max: 2
>>         err: 0
>>
>>  MEM TCP_MSG
>>         avail: 12
>>         used: 0
>>         max: 7
>>         err: 0
>>
>>  MEM TIMEOUT
>>         avail: 4
>>         used: 2
>>         max: 3
>>         err: 0
>>
>>
>> I would appreciate if can take a look
>>
>> Elad
>>
>>
>> On Wed, Jun 13, 2012 at 6:47 PM, Michael O'Dowd
>> <michael.odowd@kuantic.com> wrote:
>>>
>>> Hi Elad,
>>>
>>> I ran into a similar problem recently. I'm using a recent CVS checkout
>>> rather than 3.0. Also, I'm probably not using the same ethernet HW, so I
>>> don't know how well my reply corresponds to your case.
>>>
>>> The eth_drv.c file is the glue between lwIP and the underlying ethernet
>>> driver, so the issue that you are encountering may be specific to the
>>> driver. In my case, when under stress, eth_drv.c generates the error
>>> message: "cannot allocate pbuf to receive packet". Soon after that, the
>>> ethernet driver stops receiving traffic permanently, but does not crash.
>>> In
>>> your case, if I understand correctly, your system crashes.
>>>
>>> The issue is that when eth_drv_recv() fails to allocate a pbuf, it
>>> returns
>>> without calling the ethernet driver recv() function: (sc->funs->recv)().
>>> In
>>> my case, the driver requires that it's recv() function be called, in
>>> order
>>> to complete the processing of the packet reception and to free up the
>>> receive buffer(s). Failing to call it, apparently causes the receive path
>>> to
>>> cease functioning (I'm still investigating the details). In your case, I
>>> gather that it crashes the system.
>>>
>>> Note: I'm running on an NXP 1788 (Cortex-M3), using the
>>> "devs/arm/lpc2xxx/current/src/if_lpc2xxx.c" ethernet driver.
>>>
>>> There are two aspects to this problem:
>>>
>>> 1) In my opinion, there is a bug in eth_drv_recv(). If there are no pbufs
>>> available, then it should at least cause the received packet to be
>>> discarded. Otherwise, the system may fail whenever there is a minor burst
>>> of
>>> traffic on the network. It doesn't take much: there are only 16 pbufs
>>> available by default. Whether or not the system fails, depends on how the
>>> ethernet driver reacts to the failure to call it's recv() function. I
>>> hope
>>> to fix this on my platform in the near future.
>>>
>>> 2) You should also keep an eye on your pbuf usage, just to make sure that
>>> you don't have a pbuf memroy leak. You could also try to allocate more
>>> pbufs, if you have the available memory.
>>>
>>> If you are using the default lwip configuration, the pbuf memory
>>> allocation
>>> is handled by memp.[hc]. It has a fixed number of pbufs available. The
>>> default is 16 pbufs, and can be changed in the configtool under: [lwIP
>>> networking stack/Memory options/Number of memp struct pbufs].
>>>
>>> Alternatively, if you have lots of memory, you could enable the checkbox:
>>> [lwIP networking stack/Memory options/Use malloc for pool allocations].
>>> This
>>> bypasses the memp pools and their static limitations. Though this will
>>> make
>>> it harder to spot a pbuf memory leak. I haven't tried this personally.
>>>
>>> Finally, (when using memp) the pbuf usage can be monitored with
>>> lwip/stats.h. If you have access to a serial port, try calling
>>> stats_display(). Here is a snippet of the pbuf related output:
>>>
>>>>  MEM PBUF_POOL
>>>>          avail: 16
>>>>          used: 0
>>>>          max: 3
>>>>          err: 0
>>>
>>> The "err" counter increases when pbuf_alloc() fails.
>>>
>>> Hope that helps,
>>>
>>> Regards,
>>>
>>> Michael O'Dowd
>>> Kuantic SAS
>>>
>>>
>>> On 12/06/2012 22:40, Elad Yosef wrote:
>>>>
>>>> Hi all,
>>>> I'm using LwIP stack on my target and experiencing crashes under stress.
>>>>
>>>> function eth_drv_recv) from ../io/eth/v3_0/ser/lwip/eth_drv.c
>>>> calls pbuf_alloc() and this allocation fails.
>>>>
>>>> Is this result of some bad configuration?
>>>>
>>>> Thanks
>>>> Elad
>>>>
>

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss