* [ECOS] problems with MBUFs (TCP/IP stack) @ 2000-07-08 5:54 Marco Monguzzi 2000-07-08 8:26 ` Grant Edwards 0 siblings, 1 reply; 13+ messages in thread From: Marco Monguzzi @ 2000-07-08 5:54 UTC (permalink / raw) To: ecos-discuss [-- Attachment #1: Type: text/plain, Size: 1717 bytes --] Hello All. we are running eCos (with the TCP/IP package) on a MIPS R3041 based board. The board is equipped with a CS8900A ethernet controller. The ethernet driver has been written starting from the one that comes for the EDB7xxx board. Althought the Net subsystem passed all the package's tests, we are experiencing severe problems with MBUFs handling. They are driving out of time our project delivery date. Any help would be more than welcome... It is possible to explain the problem using a simple program: attached you'll find the eCos "server test" (file server_test.c) modified to show the MBUF counter for each incoming connection. Each time we connect to the server with telnet, the counter becames bigger (one more than the previous iteration). Repeating the trial several times (around 60 times), we get the signaling of "panic: eth_recv out of MBUFs" (with the net v1_0b1 package). We also tried with a snapshot of the net package from CVS (downloaded on June 6, 2000. We patched kapi.h and kapi.cxx to support cyg_scheduler_safe_lock() and cyg_scheduler_read_lock()); the system now tolerates the problem and does not halt but the behavior is the same: "warning: eth_recv out of MBUFs". We captured the packets exchanged between the server (192.168.0.116) the client used to run telnet (a Linux box. 192.168.0.10). You'll find them attached in packets.zip. Notice: commenting the write() in server_test.c (line 115), i.e. we do not send back the string "hello etc..", the MBUF counter does not show this behaviour any more. It seems like the mbufs are not freed from the TCP queue. Anybody experienced something similar? Any suggestion about how we should tackle this problem? All the best, Marco [-- Attachment #2: packets.zip --] [-- Type: application/zip, Size: 2222 bytes --] [-- Attachment #3: server_test.zip --] [-- Type: application/zip, Size: 1942 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-08 5:54 [ECOS] problems with MBUFs (TCP/IP stack) Marco Monguzzi @ 2000-07-08 8:26 ` Grant Edwards 2000-07-08 11:45 ` Marco Monguzzi 0 siblings, 1 reply; 13+ messages in thread From: Grant Edwards @ 2000-07-08 8:26 UTC (permalink / raw) To: Marco Monguzzi; +Cc: ecos-discuss On Sat, Jul 08, 2000 at 02:41:07PM +0200, Marco Monguzzi wrote: > Each time we connect to the server with telnet, the counter becames bigger > (one more than the previous iteration). Repeating the trial several times > (around 60 times), we get the signaling of "panic: eth_recv out of MBUFs" > (with the net v1_0b1 package). Is it the number of times a TCP connection is established that triggers the problem? I've been running continuous TCP traffic (about 100 packets per second) for days without any problems -- but, I haven't been establishing and dropping TCP connections a lot. -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-08 8:26 ` Grant Edwards @ 2000-07-08 11:45 ` Marco Monguzzi 2000-07-08 12:05 ` Marco Monguzzi 2000-07-10 8:50 ` Grant Edwards 0 siblings, 2 replies; 13+ messages in thread From: Marco Monguzzi @ 2000-07-08 11:45 UTC (permalink / raw) To: Grant Edwards; +Cc: ecos-discuss Hello Grant, > Is it the number of times a TCP connection is established that > triggers the > problem? It seems so. Running a web server (goahaead 2.1) over the TCP/IP stack we observed the same behavior: the MBUFs counter becomes bigger and bigger for each HTTP request (i.e. a TCP connection) done. At the end, the panic/warning condition about the MBUFs occours. Thanks in advance. All the best, Marco ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-08 11:45 ` Marco Monguzzi @ 2000-07-08 12:05 ` Marco Monguzzi 2000-07-10 8:50 ` Grant Edwards 1 sibling, 0 replies; 13+ messages in thread From: Marco Monguzzi @ 2000-07-08 12:05 UTC (permalink / raw) To: grante; +Cc: ecos-discuss Just a little addition. As from the first post, using the server_test.c, in which the write() (line line 115) has been commented, the MBUF counter does not show this behaviour any more. Even here a TCP connection is established/dropped but we do not send data back. Regards, Marco > -----Original Message----- > From: Marco Monguzzi [ mailto:marco@sitek.it ] > Sent: Saturday, July 08, 2000 8:38 PM > To: Grant Edwards > Cc: ecos-discuss@sourceware.cygnus.com > Subject: RE: [ECOS] problems with MBUFs (TCP/IP stack) > > > Hello Grant, > > > Is it the number of times a TCP connection is established that > > triggers the > > problem? > It seems so. Running a web server (goahaead 2.1) over the TCP/IP stack > we observed the same behavior: the MBUFs counter becomes bigger > and bigger for > each HTTP request (i.e. a TCP connection) done. At the end, the > panic/warning > condition about the MBUFs occours. > > Thanks in advance. > > All the best, > Marco ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-08 11:45 ` Marco Monguzzi 2000-07-08 12:05 ` Marco Monguzzi @ 2000-07-10 8:50 ` Grant Edwards 2000-07-10 9:53 ` Gary Thomas ` (2 more replies) 1 sibling, 3 replies; 13+ messages in thread From: Grant Edwards @ 2000-07-10 8:50 UTC (permalink / raw) To: Marco Monguzzi; +Cc: ecos-discuss On Sat, Jul 08, 2000 at 08:38:20PM +0200, Marco Monguzzi wrote: > > Is it the number of times a TCP connection is established that > > triggers the problem? > It seems so. Running a web server (goahaead 2.1) over the > TCP/IP stack we observed the same behavior: the MBUFs counter > becomes bigger and bigger for each HTTP request (i.e. a TCP > connection) done. At the end, the panic/warning condition about > the MBUFs occours. I've set up a test script that starts and stops my host-end application, and so far have about 500 TCP/IP open/close cycles without problems. I haven't been printing MBUF counters, but I haven't seen any panics or anything else odd. After I let this run for a while longer, I might have to try it with the program from the test suite. Is the source for the eCos port of the goahead publically available? Sometime in the next few weeks adding an HTTP server is going to end up at the top of my list of things to do. -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-10 8:50 ` Grant Edwards @ 2000-07-10 9:53 ` Gary Thomas 2000-07-10 10:21 ` Marco Monguzzi 2000-07-10 13:03 ` Grant Edwards 2 siblings, 0 replies; 13+ messages in thread From: Gary Thomas @ 2000-07-10 9:53 UTC (permalink / raw) To: Grant Edwards; +Cc: ecos-discuss, Marco Monguzzi On 10-Jul-2000 Grant Edwards wrote: > Is the source for the eCos port of the goahead publically > available? Sometime in the next few weeks adding an HTTP > server is going to end up at the top of my list of things to > do. > You should be able to get this directly from GoAhead. I ported this and tested it on multiple local platforms with no problems. Let me know what your experiences are. ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-10 8:50 ` Grant Edwards 2000-07-10 9:53 ` Gary Thomas @ 2000-07-10 10:21 ` Marco Monguzzi 2000-07-10 10:33 ` Grant Edwards 2000-07-10 10:41 ` Gary Thomas 2000-07-10 13:03 ` Grant Edwards 2 siblings, 2 replies; 13+ messages in thread From: Marco Monguzzi @ 2000-07-10 10:21 UTC (permalink / raw) To: Grant Edwards; +Cc: ecos-discuss > I've set up a test script that starts and stops my host-end > application, and so far have about 500 TCP/IP open/close cycles > without problems. I haven't been printing MBUF counters, but I > haven't seen any panics or anything else odd. After I let this > run for a while longer, I might have to try it with the program > from the test suite. I would appreciate whether you could provide me with you test programs. I will try them on my target. A bit more on the behavior we observe with server_test.c: we found that the problem is in the "close connection sequence" (close(client); Line 128). One MBUF remains allocated because not all the needed "free functions" are called. In detail, close() does not call tcp_close(). The latter should call the free() for the tcp_template that we think remains allocated. > Is the source for the eCos port of the goahead publically > available? Sometime in the next few weeks adding an HTTP > server is going to end up at the top of my list of things to > do. The porting is distributed with 2.1 release version by goahead software. Regards, Marco ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-10 10:21 ` Marco Monguzzi @ 2000-07-10 10:33 ` Grant Edwards 2000-07-10 10:41 ` Gary Thomas 1 sibling, 0 replies; 13+ messages in thread From: Grant Edwards @ 2000-07-10 10:33 UTC (permalink / raw) To: Marco Monguzzi; +Cc: ecos-discuss On Mon, Jul 10, 2000 at 07:10:11PM +0200, Marco Monguzzi wrote: > I would appreciate whether you could provide me with you test programs. > I will try them on my target. The eCos application is written for our custom board, and won't run on your target. But, I can't see that it's doing anything particulary odd. It's basically something like: while (1) { accept() while (1) { select() n = read() if (n<=0) { close() break; } else { // handle data } } } > The porting is distributed with 2.1 release version by goahead software. Great! Somebody here was looking at goahead sources a while ago, but they must not have had a new enough version. -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-10 10:21 ` Marco Monguzzi 2000-07-10 10:33 ` Grant Edwards @ 2000-07-10 10:41 ` Gary Thomas 2000-07-10 11:18 ` Marco Monguzzi 1 sibling, 1 reply; 13+ messages in thread From: Gary Thomas @ 2000-07-10 10:41 UTC (permalink / raw) To: Marco Monguzzi; +Cc: ecos-discuss, Grant Edwards On 10-Jul-2000 Marco Monguzzi wrote: >> I've set up a test script that starts and stops my host-end >> application, and so far have about 500 TCP/IP open/close cycles >> without problems. I haven't been printing MBUF counters, but I >> haven't seen any panics or anything else odd. After I let this >> run for a while longer, I might have to try it with the program >> from the test suite. > > I would appreciate whether you could provide me with you test programs. > I will try them on my target. A bit more on the behavior we observe with > server_test.c: we found that the problem is in the "close connection > sequence" > (close(client); Line 128). One MBUF remains allocated because not all the > needed > "free functions" are called. In detail, close() does not call tcp_close(). > The latter should call the free() for the tcp_template that we think remains > allocated. > Are you sure about this? The 'close(client)' call should end up calling "socket_close()" which in turn calls "tcp_close()". Have you tried tracing through this, e.g. using GDB, to see what happens when you call 'close(client)'? ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-10 10:41 ` Gary Thomas @ 2000-07-10 11:18 ` Marco Monguzzi 0 siblings, 0 replies; 13+ messages in thread From: Marco Monguzzi @ 2000-07-10 11:18 UTC (permalink / raw) To: Gary Thomas; +Cc: ecos-discuss, Grant Edwards > Are you sure about this? The 'close(client)' call should end up calling > "socket_close()" which in turn calls "tcp_close()". I have: 'close(client)' ===> 'soo_close()' ==> 'soclose()' etc.... >Have you > tried tracing > through this, e.g. using GDB, to see what happens when you call > 'close(client)'? unfortunaly, we have been throught the stack code (using GDB) for several days (and nights... :-( Thanks, Marco ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-10 8:50 ` Grant Edwards 2000-07-10 9:53 ` Gary Thomas 2000-07-10 10:21 ` Marco Monguzzi @ 2000-07-10 13:03 ` Grant Edwards 2000-07-11 3:45 ` Hugo 'NOx' Tyson 2 siblings, 1 reply; 13+ messages in thread From: Grant Edwards @ 2000-07-10 13:03 UTC (permalink / raw) To: Marco Monguzzi; +Cc: ecos-discuss On Mon, Jul 10, 2000 at 03:50:38PM -0500, Grant Edwards wrote: > I've set up a test script that starts and stops my host-end > application, and so far have about 500 TCP/IP open/close cycles > without problems. I haven't been printing MBUF counters, but I > haven't seen any panics or anything else odd. After I let this > run for a while longer, I might have to try it with the program > from the test suite. I'm currently running the server_test program on my target board and have done 1000+ connections with no problems. I'm running eCos 1.3.1 and networking package version 1.0b1 w/ some mutex patches from sometime in April. I tried out a CVS snapshot a week or so ago and saw one or two "null mbuf" panics (which the mutex patches were to have fixed). The CVS version and my patched 1.0b1 behaved the same in respect the bug I was tracking down, so I continued using my patched 1.0b1 and have not had time to look into the "null mbuf" panics I saw. -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-10 13:03 ` Grant Edwards @ 2000-07-11 3:45 ` Hugo 'NOx' Tyson 2000-07-11 8:38 ` Marco Monguzzi 0 siblings, 1 reply; 13+ messages in thread From: Hugo 'NOx' Tyson @ 2000-07-11 3:45 UTC (permalink / raw) To: ecos-discuss Grant Edwards <grante@visi.com> writes: > On Mon, Jul 10, 2000 at 03:50:38PM -0500, Grant Edwards wrote: > > > I've set up a test script that starts and stops my host-end > > application, and so far have about 500 TCP/IP open/close cycles > > without problems. I haven't been printing MBUF counters, but I > > haven't seen any panics or anything else odd. After I let this > > run for a while longer, I might have to try it with the program > > from the test suite. > > I'm currently running the server_test program on my target > board and have done 1000+ connections with no problems. I'm > running eCos 1.3.1 and networking package version 1.0b1 w/ some > mutex patches from sometime in April. I've seen the out-of-mbufs issue in a variant of the ping test that I'm playing with, but as usual, it was cured by remembering to close the socket when I'm done with it. That cures it anyway, it seems. Maybe it depends on what you do with the socket even if you do close it? Just a data point. - Huge ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [ECOS] problems with MBUFs (TCP/IP stack) 2000-07-11 3:45 ` Hugo 'NOx' Tyson @ 2000-07-11 8:38 ` Marco Monguzzi 0 siblings, 0 replies; 13+ messages in thread From: Marco Monguzzi @ 2000-07-11 8:38 UTC (permalink / raw) To: Hugo 'NOx' Tyson; +Cc: ecos-discuss > I've seen the out-of-mbufs issue in a variant of the ping test that I'm > playing with, but as usual, it was cured by remembering to close > the socket > when I'm done with it. That cures it anyway, it seems. > > Maybe it depends on what you do with the socket even if you do close it? > > Just a data point. Hello. Even the below (dummy) client works (running on target server_test.c that shows the MBUFs counter we got no problems). The problem (the MBUFs not freed that generate the panic/warning) occours using the telnet client via shell (or with this code if you comment the line "close(sockfd);"). In other words, when only the server closes the connection. We observed the same path running the Web server. The connection is established, the web server honours the HTTP request and it closes (two steps: shutdown() in SHUT_WR mode + close() ). But, at the end, the MBUF counter shows that one MBUF has not been freed for each request served. Regards, Marco %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% #include <stdio.h> #include <sys/types.h> #include <sys/socket.h> #include <arpa/inet.h> #include <string.h> #define SERVER_HOST_ADDR "192.168.0.116" //target IP #define SERVER_TCP_PORT 7734 #define NTIMES 100 int main(int argc, char* argv[]){ int sockfd, len, ii; struct sockaddr_in serv_addr; char out[256]; char in[256]; if(argc != 2){ printf("usage: client_test \"string to be sent\"\n"); exit(-1); } strcpy(out, argv[1]); bzero((char*) &serv_addr, sizeof(serv_addr)); serv_addr.sin_family = AF_INET; serv_addr.sin_addr.s_addr = inet_addr(SERVER_HOST_ADDR); serv_addr.sin_port = htons(SERVER_TCP_PORT); for(ii=0; ii<NTIMES; ii++){ if ( (sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) { printf("cannot open stream socket!\n"); exit(-1); } if ( connect(sockfd, (struct sockaddr_in *) &serv_addr, sizeof(serv_addr)) < 0) { printf("cannot connect to the server!\n"); exit(-1); } len = read(sockfd, in, sizeof(in)); in[len-1] = '\0'; printf("connected! read %d bytes from target:%s\n", len, in); write(sockfd, out, sizeof(out)); close(sockfd); } exit(0); } ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2000-07-11 8:38 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2000-07-08 5:54 [ECOS] problems with MBUFs (TCP/IP stack) Marco Monguzzi 2000-07-08 8:26 ` Grant Edwards 2000-07-08 11:45 ` Marco Monguzzi 2000-07-08 12:05 ` Marco Monguzzi 2000-07-10 8:50 ` Grant Edwards 2000-07-10 9:53 ` Gary Thomas 2000-07-10 10:21 ` Marco Monguzzi 2000-07-10 10:33 ` Grant Edwards 2000-07-10 10:41 ` Gary Thomas 2000-07-10 11:18 ` Marco Monguzzi 2000-07-10 13:03 ` Grant Edwards 2000-07-11 3:45 ` Hugo 'NOx' Tyson 2000-07-11 8:38 ` Marco Monguzzi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).