[ECOS] problems with MBUFs (TCP/IP stack)

[ECOS] problems with MBUFs (TCP/IP stack)

[Marco Monguzzi]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

Hello All.

we are running eCos (with the TCP/IP package) on a MIPS R3041 based board.
The board is equipped with a CS8900A ethernet controller. The ethernet
driver has been written starting from the one that comes for the EDB7xxx
board.

Althought the Net subsystem passed all the package's tests, we are
experiencing severe problems with MBUFs handling. They are driving out of
time our project delivery date. Any help would be more than welcome...

It is possible to explain the problem using a simple program: attached
you'll find the eCos "server test" (file server_test.c) modified to show the
MBUF counter for each incoming connection. Each time we connect to the
server with telnet, the counter becames bigger (one more than the previous
iteration). Repeating the trial several times (around 60 times), we get the
signaling of "panic: eth_recv out of MBUFs" (with the net v1_0b1 package).
We also tried with a snapshot of the net package from CVS (downloaded on
June 6, 2000. We patched kapi.h and kapi.cxx to support
cyg_scheduler_safe_lock() and cyg_scheduler_read_lock()); the system now
tolerates the problem and does not halt but the behavior is the same:
"warning: eth_recv out of MBUFs".

We captured the packets exchanged between the server (192.168.0.116) the
client used to run telnet (a Linux box. 192.168.0.10). You'll find them
attached in packets.zip.

Notice: commenting the write() in server_test.c (line 115), i.e. we do not
send back the string "hello etc..", the MBUF counter does not show this
behaviour any more. It seems like the mbufs are not freed from the TCP
queue.

Anybody experienced something similar? Any suggestion about how we should
tackle this problem?

All the best,
Marco

[-- Attachment #2: packets.zip --]
[-- Type: application/zip, Size: 2222 bytes --]

[-- Attachment #3: server_test.zip --]
[-- Type: application/zip, Size: 1942 bytes --]

Re: [ECOS] problems with MBUFs (TCP/IP stack)

[Grant Edwards]

On Sat, Jul 08, 2000 at 02:41:07PM +0200, Marco Monguzzi wrote:

> Each time we connect to the server with telnet, the counter becames bigger
> (one more than the previous iteration). Repeating the trial several times
> (around 60 times), we get the signaling of "panic: eth_recv out of MBUFs"
> (with the net v1_0b1 package).

Is it the number of times a TCP connection is established that triggers the
problem?  I've been running continuous TCP traffic (about 100 packets per
second) for days without any problems -- but, I haven't been establishing and
dropping TCP connections a lot.

-- 
Grant Edwards
grante@visi.com

RE: [ECOS] problems with MBUFs (TCP/IP stack)

[Marco Monguzzi]

Hello Grant,

> Is it the number of times a TCP connection is established that
> triggers the
> problem?
It seems so. Running a web server (goahaead 2.1) over the TCP/IP stack
we observed the same behavior: the MBUFs counter becomes bigger and bigger
for
each HTTP request (i.e. a TCP connection) done. At the end, the
panic/warning
condition about the MBUFs occours.

Thanks in advance.

All the best,
Marco

RE: [ECOS] problems with MBUFs (TCP/IP stack)

[Marco Monguzzi]

Just a little addition. As from the first post, using the server_test.c,
in which the write() (line line 115) has been commented, the MBUF counter 
does not show this behaviour any more.
Even here a TCP connection is established/dropped but we do not send data 
back. 

Regards,
Marco


> -----Original Message-----
> From: Marco Monguzzi [ mailto:marco@sitek.it ]
> Sent: Saturday, July 08, 2000 8:38 PM
> To: Grant Edwards
> Cc: ecos-discuss@sourceware.cygnus.com
> Subject: RE: [ECOS] problems with MBUFs (TCP/IP stack)
> 
> 
> Hello Grant,
> 
> > Is it the number of times a TCP connection is established that 
> > triggers the
> > problem?  
> It seems so. Running a web server (goahaead 2.1) over the TCP/IP stack 
> we observed the same behavior: the MBUFs counter becomes bigger 
> and bigger for
> each HTTP request (i.e. a TCP connection) done. At the end, the 
> panic/warning
> condition about the MBUFs occours.
> 
> Thanks in advance.
> 
> All the best,
> Marco

Re: [ECOS] problems with MBUFs (TCP/IP stack)

[Grant Edwards]

On Sat, Jul 08, 2000 at 08:38:20PM +0200, Marco Monguzzi wrote:

> > Is it the number of times a TCP connection is established that
> > triggers the problem?

> It seems so. Running a web server (goahaead 2.1) over the
> TCP/IP stack we observed the same behavior: the MBUFs counter
> becomes bigger and bigger for each HTTP request (i.e. a TCP
> connection) done. At the end, the panic/warning condition about
> the MBUFs occours.

I've set up a test script that starts and stops my host-end
application, and so far have about 500 TCP/IP open/close cycles
without problems.  I haven't been printing MBUF counters, but I
haven't seen any panics or anything else odd.  After I let this
run for a while longer, I might have to try it with the program
from the test suite.

Is the source for the eCos port of the goahead publically
available?  Sometime in the next few weeks adding an HTTP
server is going to end up at the top of my list of things to
do.

-- 
Grant Edwards
grante@visi.com

Re: [ECOS] problems with MBUFs (TCP/IP stack)

[Gary Thomas]

On 10-Jul-2000 Grant Edwards wrote:
> Is the source for the eCos port of the goahead publically
> available?  Sometime in the next few weeks adding an HTTP
> server is going to end up at the top of my list of things to
> do.
> 

You should be able to get this directly from GoAhead.  I ported
this and tested it on multiple local platforms with no problems.
Let me know what your experiences are.

RE: [ECOS] problems with MBUFs (TCP/IP stack)

[Marco Monguzzi]

> I've set up a test script that starts and stops my host-end
> application, and so far have about 500 TCP/IP open/close cycles
> without problems.  I haven't been printing MBUF counters, but I
> haven't seen any panics or anything else odd.  After I let this
> run for a while longer, I might have to try it with the program
> from the test suite.

I would appreciate whether you could provide me with you test programs.
I will try them on my target. A bit more on the behavior we observe with
server_test.c: we found that the problem is in the "close connection
sequence"
(close(client); Line 128). One MBUF remains allocated because not all the
needed
"free functions" are called. In detail, close() does not call tcp_close().
The latter should call the free() for the tcp_template that we think remains
allocated.

> Is the source for the eCos port of the goahead publically
> available?  Sometime in the next few weeks adding an HTTP
> server is going to end up at the top of my list of things to
> do.

The porting is distributed with 2.1 release version by goahead software.


Regards,
Marco

Re: [ECOS] problems with MBUFs (TCP/IP stack)

[Grant Edwards]

On Mon, Jul 10, 2000 at 07:10:11PM +0200, Marco Monguzzi wrote:

> I would appreciate whether you could provide me with you test programs.
> I will try them on my target. 

The eCos application is written for our custom board, and won't
run on your target.  But, I can't see that it's doing anything
particulary odd.  It's basically something like:

  while (1)
   {
   accept()
   while (1)
     {
     select()
     n = read()
     if (n<=0)
       {
       close()
       break;
       }
     else
       {
       // handle data
       }
     }
   }
   

> The porting is distributed with 2.1 release version by goahead software.

Great!  Somebody here was looking at goahead sources a while
ago, but they must not have had a new enough version.

-- 
Grant Edwards
grante@visi.com

RE: [ECOS] problems with MBUFs (TCP/IP stack)

[Gary Thomas]

On 10-Jul-2000 Marco Monguzzi wrote:
>> I've set up a test script that starts and stops my host-end
>> application, and so far have about 500 TCP/IP open/close cycles
>> without problems.  I haven't been printing MBUF counters, but I
>> haven't seen any panics or anything else odd.  After I let this
>> run for a while longer, I might have to try it with the program
>> from the test suite.
> 
> I would appreciate whether you could provide me with you test programs.
> I will try them on my target. A bit more on the behavior we observe with
> server_test.c: we found that the problem is in the "close connection
> sequence"
> (close(client); Line 128). One MBUF remains allocated because not all the
> needed
> "free functions" are called. In detail, close() does not call tcp_close().
> The latter should call the free() for the tcp_template that we think remains
> allocated.
> 

Are you sure about this?  The 'close(client)' call should end up calling
"socket_close()" which in turn calls "tcp_close()".  Have you tried tracing
through this, e.g. using GDB, to see what happens when you call 'close(client)'?

RE: [ECOS] problems with MBUFs (TCP/IP stack)

[Marco Monguzzi]

> Are you sure about this?  The 'close(client)' call should end up calling
> "socket_close()" which in turn calls "tcp_close()".

I have:
'close(client)' ===> 'soo_close()' ==> 'soclose()' etc....

>Have you
> tried tracing
> through this, e.g. using GDB, to see what happens when you call
> 'close(client)'?

unfortunaly, we have been throught the stack code (using GDB) for several
days
(and nights... :-(

Thanks,
Marco

Re: [ECOS] problems with MBUFs (TCP/IP stack)

[Grant Edwards]

On Mon, Jul 10, 2000 at 03:50:38PM -0500, Grant Edwards wrote:

> I've set up a test script that starts and stops my host-end
> application, and so far have about 500 TCP/IP open/close cycles
> without problems.  I haven't been printing MBUF counters, but I
> haven't seen any panics or anything else odd.  After I let this
> run for a while longer, I might have to try it with the program
> from the test suite.

I'm currently running the server_test program on my target
board and have done 1000+ connections with no problems.  I'm
running eCos 1.3.1 and networking package version 1.0b1 w/ some
mutex patches from sometime in April.

I tried out a CVS snapshot a week or so ago and saw one or two
"null mbuf" panics (which the mutex patches were to have
fixed).  The CVS version and my patched 1.0b1 behaved the same
in respect the bug I was tracking down, so I continued using my
patched 1.0b1 and have not had time to look into the "null
mbuf" panics I saw.

-- 
Grant Edwards
grante@visi.com

Re: [ECOS] problems with MBUFs (TCP/IP stack)

[Hugo 'NOx' Tyson]

Grant Edwards <grante@visi.com> writes:
> On Mon, Jul 10, 2000 at 03:50:38PM -0500, Grant Edwards wrote:
> 
> > I've set up a test script that starts and stops my host-end
> > application, and so far have about 500 TCP/IP open/close cycles
> > without problems.  I haven't been printing MBUF counters, but I
> > haven't seen any panics or anything else odd.  After I let this
> > run for a while longer, I might have to try it with the program
> > from the test suite.
> 
> I'm currently running the server_test program on my target
> board and have done 1000+ connections with no problems.  I'm
> running eCos 1.3.1 and networking package version 1.0b1 w/ some
> mutex patches from sometime in April.

I've seen the out-of-mbufs issue in a variant of the ping test that I'm
playing with, but as usual, it was cured by remembering to close the socket
when I'm done with it.  That cures it anyway, it seems.

Maybe it depends on what you do with the socket even if you do close it?

Just a data point.

	- Huge

RE: [ECOS] problems with MBUFs (TCP/IP stack)

[Marco Monguzzi]

> I've seen the out-of-mbufs issue in a variant of the ping test that I'm
> playing with, but as usual, it was cured by remembering to close
> the socket
> when I'm done with it.  That cures it anyway, it seems.
>
> Maybe it depends on what you do with the socket even if you do close it?
>
> Just a data point.

Hello.
Even the below (dummy) client works (running on target server_test.c that
shows the MBUFs counter we got no problems). The problem (the MBUFs not
freed
that generate the panic/warning) occours using the telnet client via shell
(or with this code if you comment the line "close(sockfd);").

In other words, when only the server closes the connection.

We observed the same path running the Web server. The connection is
established,
the web server honours the HTTP request and it closes (two steps: shutdown()
in
SHUT_WR mode + close() ). But, at the end, the MBUF counter shows that one
MBUF has
not been freed for each request served.

Regards,
Marco



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <string.h>

#define SERVER_HOST_ADDR "192.168.0.116" //target IP
#define SERVER_TCP_PORT 7734
#define NTIMES 100

int main(int argc, char* argv[]){
    int sockfd, len, ii;
    struct sockaddr_in serv_addr;
    char out[256];
    char in[256];

    if(argc != 2){
      printf("usage: client_test \"string to be sent\"\n");
      exit(-1);
    }
    strcpy(out, argv[1]);

    bzero((char*) &serv_addr, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = inet_addr(SERVER_HOST_ADDR);
    serv_addr.sin_port = htons(SERVER_TCP_PORT);


    for(ii=0; ii<NTIMES; ii++){
      if ( (sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
    printf("cannot open stream socket!\n");
    exit(-1);
      }
      if ( connect(sockfd, (struct sockaddr_in *) &serv_addr,
sizeof(serv_addr)) < 0) {
    printf("cannot connect to the server!\n");
    exit(-1);
      }

      len = read(sockfd, in, sizeof(in));
      in[len-1] = '\0';
      printf("connected! read %d bytes from target:%s\n", len, in);
      write(sockfd, out, sizeof(out));
      close(sockfd);
    }
    exit(0);
 }