[ECOS] "PANIC: zinit: Out of memory" when num sockets increased to 64

[ECOS] "PANIC: zinit: Out of memory" when num sockets increased to 64

[Grant Edwards]

I'm testing some application code using the pc_rltk8139 target
and the net template.  It works fine using the default
FILEIO_NFD and FILEIO_NFILE valus of 16.  But one of the tests
I need to run involves opening and closing a TCP connection
about 40 times in a row.  This test takes a while, since on the
17th iteration the network stack runs out of sockets and stalls
while the 16 zombie sockets that are all in the TIME_WAIT state
hang around until they timeout and go away.

So, I increated FILEIO_NFD and FILEIO_NFILE to 64, which means
there are 64 sockets available.

But now the app fails to start up:

  Booting '/server.elf'

   kernel /server.elf
      [Multiboot-elf, <0x108000:0x4a860:0x879f8>, shtab=0x1db1e0, entry=0x108000]
   
   [cyg_net_init] Init: mbinit(0x00000000)
   [cyg_net_init] Init: cyg_net_init_devs(0x00000000)
   Init device 'rltk8139_eth0'
   [cyg_net_init] Init: loopattach(0x00000000)
   [cyg_net_init] Init: ifinit(0x00000000)
   [cyg_net_init] Init: domaininit(0x00000000)
   [cyg_net_init] Init: cyg_net_add_domain(0x00152040)
   New domain internet at 0x00000000
   PANIC: zinit: Out of memory

The target configuration is set for 16MB of RAM, and my
application isn't all that large:

  $ size server.elf
     text    data     bss     dec     hex filename
   298992    6248  555512  860752   d2250 server.elf

So there's well over 15MB of unallocated RAM. That seems like
more than enough RAM to handle 64 sockets.
   
Why am I getting a panic on startup?

-- 
Grant Edwards                   grante             Yow! Didn't I buy a 1951
                                  at               Packard from you last March
                               visi.com            in Cairo?


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] "PANIC: zinit: Out of memory" when num sockets increased to 64

[Andrew Lunn]

> So there's well over 15MB of unallocated RAM. That seems like
> more than enough RAM to handle 64 sockets.
>    
> Why am I getting a panic on startup?

The network stack has its own memory pool. See

CYGPKG_NET_MEM_USAGE, CYGPKG_NET_MEMPOOL_SIZE, CYGPKG_NET_MBUFS_SIZE,
CYGPKG_NET_MBUFS_SIZE.

        Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: "PANIC: zinit: Out of memory" when num sockets increased to 64

[Grant Edwards]

On 2008-04-08, Andrew Lunn <andrew@lunn.ch> wrote:
>> So there's well over 15MB of unallocated RAM. That seems like
>> more than enough RAM to handle 64 sockets.
>>    
>> Why am I getting a panic on startup?
>
> The network stack has its own memory pool. See
>
> CYGPKG_NET_MEM_USAGE, CYGPKG_NET_MEMPOOL_SIZE, CYGPKG_NET_MBUFS_SIZE,
> CYGPKG_NET_MBUFS_SIZE.

I increased CYGPKG_NET_MEM_USAGE, and that fixed the problem.
The default value is (256*1024)+(CYGPKG_NET_MAXSOCKETS*1024)
What initially mislead me into thinking I should leave that
alone when I increased the number of sockets was the
description:

   Memory designated for networking buffers. This option
   controls the amount of memory pre-allocated for buffers used
   by the networking code.  The number is an upper limit, with
   at least enough space to get the stack initialized.

My reading of that is that (256*1024)+(CYGPKG_NET_MAXSOCKETS*1024)
is an upper bound on what's required for the stack to be
initialized. Apparently it isn't.

-- 
Grant Edwards                   grante             Yow! Vote for ME -- I'm
                                  at               well-tapered, half-cocked,
                               visi.com            ill-conceived and
                                                   TAX-DEFERRED!


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: "PANIC: zinit: Out of memory" when num sockets increased to 64

[Grant Edwards]

On 2008-04-08, Andrew Lunn <andrew@lunn.ch> wrote:
>> So there's well over 15MB of unallocated RAM. That seems like
>> more than enough RAM to handle 64 sockets.
>>    
>> Why am I getting a panic on startup?
>
> The network stack has its own memory pool. See
>
> CYGPKG_NET_MEM_USAGE, CYGPKG_NET_MEMPOOL_SIZE, CYGPKG_NET_MBUFS_SIZE,
> CYGPKG_NET_MBUFS_SIZE.

Several other people seem to have tripped over this bug.  IMO,
the problem is that the default value takes the number of
sockets into account, and the description claims that value is
guaranteed to allow the network stack to start up.

One presumes that the minimum value required for stack startup
could be calculated at configure time based on the max number
of sockets.  Since that isn't being done, should I submit a
patch that would change the description so that it doesn't say
that it is?

-- 
Grant Edwards                   grante             Yow! Hmmm ... a CRIPPLED
                                  at               ACCOUNTANT with a FALAFEL
                               visi.com            sandwich is HIT by a
                                                   TROLLEY-CAR ...


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: "PANIC: zinit: Out of memory" when num sockets increased to 64

[Gary Thomas]

Grant Edwards wrote:
> On 2008-04-08, Andrew Lunn <andrew@lunn.ch> wrote:
>>> So there's well over 15MB of unallocated RAM. That seems like
>>> more than enough RAM to handle 64 sockets.
>>>    
>>> Why am I getting a panic on startup?
>> The network stack has its own memory pool. See
>>
>> CYGPKG_NET_MEM_USAGE, CYGPKG_NET_MEMPOOL_SIZE, CYGPKG_NET_MBUFS_SIZE,
>> CYGPKG_NET_MBUFS_SIZE.
> 
> Several other people seem to have tripped over this bug.  IMO,
> the problem is that the default value takes the number of
> sockets into account, and the description claims that value is
> guaranteed to allow the network stack to start up.
> 
> One presumes that the minimum value required for stack startup
> could be calculated at configure time based on the max number
> of sockets.  Since that isn't being done, should I submit a
> patch that would change the description so that it doesn't say
> that it is?
> 

Or, better yet, improve the CDL so it behaves as folks expect :-)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: "PANIC: zinit: Out of memory" when num sockets increased to 64

[Grant Edwards]

On 2008-04-09, Gary Thomas <gary@mlbassoc.com> wrote:

>> Several other people seem to have tripped over this bug.  IMO,
>> the problem is that the default value takes the number of
>> sockets into account, and the description claims that value is
>> guaranteed to allow the network stack to start up.
>> 
>> One presumes that the minimum value required for stack startup
>> could be calculated at configure time based on the max number
>> of sockets.  Since that isn't being done, should I submit a
>> patch that would change the description so that it doesn't say
>> that it is?
>
> Or, better yet, improve the CDL so it behaves as folks expect :-)

That would definitely be better.  I presume if the formula for
the required amount of memory were known somebody would have
fixed the CDL already.

Is it practical to figure out how much is required by
statically examining the source code, or would it be simpler to
just do it emperically/experimentally?

-- 
Grant Edwards                   grante             Yow! ANN JILLIAN'S HAIR
                                  at               makes LONI ANDERSON'S
                               visi.com            HAIR look like RICARDO
                                                   MONTALBAN'S HAIR!


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: "PANIC: zinit: Out of memory" when num sockets increased to 64

[Gary Thomas]

Grant Edwards wrote:
> On 2008-04-09, Gary Thomas <gary@mlbassoc.com> wrote:
> 
>>> Several other people seem to have tripped over this bug.  IMO,
>>> the problem is that the default value takes the number of
>>> sockets into account, and the description claims that value is
>>> guaranteed to allow the network stack to start up.
>>>
>>> One presumes that the minimum value required for stack startup
>>> could be calculated at configure time based on the max number
>>> of sockets.  Since that isn't being done, should I submit a
>>> patch that would change the description so that it doesn't say
>>> that it is?
>> Or, better yet, improve the CDL so it behaves as folks expect :-)
> 
> That would definitely be better.  I presume if the formula for
> the required amount of memory were known somebody would have
> fixed the CDL already.
> 
> Is it practical to figure out how much is required by
> statically examining the source code, or would it be simpler to
> just do it emperically/experimentally?
> 

The formula that's in place was done by me, examining the code.
I think the problem is that since I made that analysis, additional
terms have been introduced (e.g. back then, there was no/little
notion of file descriptors).  These terms should be figured
into the result.  Of course, it gets a bit tricky as one wants
the number to "work" regardless of the configuration (e.g. you
_might_ build a system where there is no file system, so the
number of file descriptors is unknown, but you still want the
computation to be correct).

It's this sort of "moving target" (new features, variable
configurations, ...) that have made some configuration metrics
a bit hard to manage.  For example, why must one increase the
number of open files in [at least] two places, when surely one
should suffice?

Anything you can do so suggest improvements will surely be
useful and considered for adoption.


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss