[ECOS] Re: Network TCP Handler: stale socket disposal

[ECOS] Re: Network TCP Handler: stale socket disposal

[Alok Singh]

Hi John,

What is the state of this issue?

Is your patch accepted? 

Let us know the status, so that we can also pick up the changes.


Regards,
Alok


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: Network TCP Handler: stale socket disposal

[Alok Singh]

John,

Please post it in whatever form you deem as appropriate.  Let us know when you will be able to post it.

BTW, I'm seeing a similar issue in my application.  Sockets are exhausted, and web is not accessible.  Ping is going through. I've kept Max. number of sockets as 32 while configuring ecos.
The strange thing is that the issue comes after a long period of inactivity, i.e., no web activity.   

Your patch will be useful.

Regards,
Alok
 

________________________________________
From: John Mills [mailto:JMills@airdefense.net] 
Sent: Tuesday, September 25, 2007 6:56 PM
To: Alok Singh
Cc: ecos-discuss@ecos.sourceware.org
Subject: RE: [ECOS] Re: Network TCP Handler: stale socket disposal

All -

I don't believe my patch was ever accepted. The eCos sources carry a cautionary note that such a 'hung' socket may have been placed on the net stack 'accept' queue, and may be in the process of acceptance by a user thread which could then block.

I checked Linux and FreeBSD man pages on this. In Linux the blocked task would receive the next such incoming client and be unblocked. This is perfect for my application. Neither OpenBSD nor FreeBSD man pages address the issue specifically.

I didn't try to replicate the race condition myself.

eCos now uses an older version of the BSD stack, and updating it looks significant - well over a month's work, maybe 3 or 4 based on the time the original author reported for his port.

The patch solved a show-stopping problem with our application and seems to cause no problems, so we're using it. Two other users tried the approach and wrote me off-list: in one case it resolved a problem, in the other it made no difference (either way, as far as I know). I can share the patch on a "no warranty" basis, but would prefer to post it appropriately rather than circulating it randomly.

 - John Mills

-----Original Message-----
From: ecos-discuss-owner@ecos.sourceware.org on behalf of Alok Singh
Sent: Tue 9/25/2007 8:20 AM
To: johnmills@speakeasy.net
Cc: ecos-discuss@ecos.sourceware.org
Subject: [ECOS] Re: Network TCP Handler: stale socket disposal

Hi John,

What is the state of this issue?

Is your patch accepted?

Let us know the status, so that we can also pick up the changes.


Regards,
Alok


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss



--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: Network TCP Handler: stale socket disposal

[John Mills]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2467 bytes --]

All -

First, please excuse my attaching a patch here, but I would like to 
respond once (or a few times &8-) in case it's helpful.

Second, my reply to Mr. Singh failed posting because it originated from a 
reader that can't enforce text-only formatting. Sorry.

Third, I post this patch for test and discussion purposes. It seems to 
help in our limited case, but comes with no guarantee. I would appreciate 
hearing others' experience.

Thanks for your patience.

 - John Mills

On Tue, 25 Sep 2007, Alok Singh wrote:

> John,
 
> Please post it in whatever form you deem as appropriate.  Let us know
> when you will be able to post it.
 
> BTW, I'm seeing a similar issue in my application.  Sockets are
> exhausted, and web is not accessible.  Ping is going through. I've kept
> Max. number of sockets as 32 while configuring ecos. The strange thing
> is that the issue comes after a long period of inactivity, i.e., no web
> activity.   
 
> Your patch will be useful.
 
> Regards,
> Alok  

> ________________________________________
> From: John Mills [mailto:JMills@airdefense.net] 
> Sent: Tuesday, September 25, 2007 6:56 PM
> To: Alok Singh
> Cc: ecos-discuss@ecos.sourceware.org
> Subject: RE: [ECOS] Re: Network TCP Handler: stale socket disposal
 
> All -
 
> I don't believe my patch was ever accepted. The eCos sources carry a
> cautionary note that such a 'hung' socket may have been placed on the
> net stack 'accept' queue, and may be in the process of acceptance by a
> user thread which could then block.
 
> I checked Linux and FreeBSD man pages on this. In Linux the blocked task
> would receive the next such incoming client and be unblocked. This is
> perfect for my application. Neither OpenBSD nor FreeBSD man pages
> address the issue specifically.
 
> I didn't try to replicate the race condition myself.
 
> eCos now uses an older version of the BSD stack, and updating it looks
> significant - well over a month's work, maybe 3 or 4 based on the time
> the original author reported for his port.
 
> The patch solved a show-stopping problem with our application and seems
> to cause no problems, so we're using it. Two other users tried the
> approach and wrote me off-list: in one case it resolved a problem, in
> the other it made no difference (either way, as far as I know). I can
> share the patch on a "no warranty" basis, but would prefer to post it
> appropriately rather than circulating it randomly.
 
>  - John Mills

[-- Attachment #2: Patch discards stale sockets --]
[-- Type: TEXT/PLAIN, Size: 907 bytes --]

--- /opt/ecos/ecos-2.0/packages/net/bsd_tcpip/v2_0/src/sys/kern/uipc_socket.c	2003-02-12 10:29:52.000000000 -0500
+++ ./uipc_socket.c	2007-08-28 16:11:56.000000000 -0400
@@ -242,6 +246,11 @@
 			TAILQ_REMOVE(&head->so_incomp, so, so_list);
 			head->so_incqlen--;
 		} else if (so->so_state & SS_COMP) {
+			 if((so->so_error == ECONNRESET) ||
+				 (so->so_error == ECONNREFUSED)){ // forced drop if flagged
+				  TAILQ_REMOVE(&head->so_comp, so, so_list);
+				  head->so_qlen--;
+			 } else {
 			/*
 			 * We must not decommission a socket that's
 			 * on the accept(2) queue.  If we do, then
@@ -249,11 +258,13 @@
 			 * that the listening socket was ready.
 			 */
 			return;
+			 }
 		} else {
 			panic("sofree: not queued");
 		}
 		head->so_qlen--;
 		so->so_state &= ~SS_INCOMP;
+		so->so_state &= ~SS_COMP;
 		so->so_head = NULL;
 	}
 	sbrelease(&so->so_snd, so);

[-- Attachment #3: Type: text/plain, Size: 148 bytes --]

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] Re: Network TCP Handler: stale socket disposal

[Alok Singh]

Hi John/everybody,
The patch didn't work for me.  I still had all the sockets exhausted, and so the web server hangs, and doesn't accept any new connection.  The number of sockets I'm creating while configuring ECOS is 32.  Please see the dump of " cyg_kmem_print_stats()" below when the problem comes.

Test case: I've a script that opens and closes connection to web server every second. It takes around 2 hours to exhaust the SOCKETS zone of sockets. The TCP zone of sockets also comes down. Even if I stop the script, the sockets never recover. I'm currently debugging it(trying to understand  TCP by reading Comer and stevens).

Any ideas are welcome.

***************
cyg_kmem_print_stats() -
Network stack mbuf stats:
   mbufs 32, clusters 6, free clusters 6
   Failed to get 0 times
   Waited to get 0 times
   Drained queues to get 0 times
VM zone 'ripcb':
  Total: 32, Free: 32, Allocs: 0, Frees: 0, Fails: 0
VM zone 'tcpcb':
  Total: 32, Free: 1, Allocs: 3989, Frees: 3958, Fails: 0
VM zone 'udpcb':
  Total: 32, Free: 31, Allocs: 14, Frees: 13, Fails: 0
VM zone 'socket':
  Total: 32, Free: 0, Allocs: 10319, Frees: 3971, Fails: 6316
Misc mpool: total  131056, free   79008, max free block 77972
Mbufs pool: total  130944, free  128768, blocksize  128
Clust pool: total  262144, free  247808, blocksize 2048
*********************************************************** 

-Alok

-----Original Message-----
From: John Mills [mailto:johnmills@speakeasy.net] 
Sent: Tuesday, September 25, 2007 8:26 PM
To: Alok Singh
Cc: John Mills; eCos Users
Subject: Re: [ECOS] Re: Network TCP Handler: stale socket disposal

All -

First, please excuse my attaching a patch here, but I would like to 
respond once (or a few times &8-) in case it's helpful.

Second, my reply to Mr. Singh failed posting because it originated from a 
reader that can't enforce text-only formatting. Sorry.

Third, I post this patch for test and discussion purposes. It seems to 
help in our limited case, but comes with no guarantee. I would appreciate 
hearing others' experience.

Thanks for your patience.

 - John Mills

On Tue, 25 Sep 2007, Alok Singh wrote:

> John,
 
> Please post it in whatever form you deem as appropriate.  Let us know
> when you will be able to post it.
 
> BTW, I'm seeing a similar issue in my application.  Sockets are
> exhausted, and web is not accessible.  Ping is going through. I've kept
> Max. number of sockets as 32 while configuring ecos. The strange thing
> is that the issue comes after a long period of inactivity, i.e., no web
> activity.   
 
> Your patch will be useful.
 
> Regards,
> Alok  

> ________________________________________
> From: John Mills [mailto:JMills@airdefense.net] 
> Sent: Tuesday, September 25, 2007 6:56 PM
> To: Alok Singh
> Cc: ecos-discuss@ecos.sourceware.org
> Subject: RE: [ECOS] Re: Network TCP Handler: stale socket disposal
 
> All -
 
> I don't believe my patch was ever accepted. The eCos sources carry a
> cautionary note that such a 'hung' socket may have been placed on the
> net stack 'accept' queue, and may be in the process of acceptance by a
> user thread which could then block.
 
> I checked Linux and FreeBSD man pages on this. In Linux the blocked task
> would receive the next such incoming client and be unblocked. This is
> perfect for my application. Neither OpenBSD nor FreeBSD man pages
> address the issue specifically.
 
> I didn't try to replicate the race condition myself.
 
> eCos now uses an older version of the BSD stack, and updating it looks
> significant - well over a month's work, maybe 3 or 4 based on the time
> the original author reported for his port.
 
> The patch solved a show-stopping problem with our application and seems
> to cause no problems, so we're using it. Two other users tried the
> approach and wrote me off-list: in one case it resolved a problem, in
> the other it made no difference (either way, as far as I know). I can
> share the patch on a "no warranty" basis, but would prefer to post it
> appropriately rather than circulating it randomly.
 
>  - John Mills


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] Re: Network TCP Handler: stale socket disposal

[John Mills]

Alok -

Thanks for your feedback. That makes the success rate 50:50 (2 of 4 
respondents) for the patch.

The web server in our product is a somewhat secondary administrative 
function and we serve simple content that we control 100%. That allows me 
to summarily close many browser inquiries. I have added and fixed the POST 
code so it handles our binary firmware images. Either of these may have 
closed some vulnerabilities that may be affecting you - I don't know.

Operationally our problem was triggered by vulnerability scanners used by
our customers' SysAdmins. These locked up our product in the course of
their overall test scenarios. In principle that meant we were also
vlunerable to the corresponding hacker exploits. That's what I meant in an
earlier post about a specific, observed functional problem.

The lock-up is broader than just web service. When the socket-descriptor
pool ('zone') is depleted, no new net sockets can be allocated. This
affected other, primary functions in our product, making it a critical 
issue.

I traced the problem by putting 'diag_printf' lines at points where
sockets were created and deallocated, working down to find "what didn't
happen" when a socket was lost. Sounds like you have the same road ahead 
of you.

Thanks again for your reaponse.

 - John Mills

On Fri, 28 Sep 2007, Alok Singh wrote:

> Hi John/everybody,
> The patch didn't work for me.  I still had all the sockets exhausted, and so the web server hangs, and doesn't accept any new connection.  The number of sockets I'm creating while configuring ECOS is 32.  Please see the dump of " cyg_kmem_print_stats()" below when the problem comes.
> 
> Test case: I've a script that opens and closes connection to web server every second. It takes around 2 hours to exhaust the SOCKETS zone of sockets. The TCP zone of sockets also comes down. Even if I stop the script, the sockets never recover. I'm currently debugging it(trying to understand  TCP by reading Comer and stevens).
> 
> Any ideas are welcome.
> 
> ***************
> cyg_kmem_print_stats() -
> Network stack mbuf stats:
>    mbufs 32, clusters 6, free clusters 6
>    Failed to get 0 times
>    Waited to get 0 times
>    Drained queues to get 0 times
> VM zone 'ripcb':
>   Total: 32, Free: 32, Allocs: 0, Frees: 0, Fails: 0
> VM zone 'tcpcb':
>   Total: 32, Free: 1, Allocs: 3989, Frees: 3958, Fails: 0
> VM zone 'udpcb':
>   Total: 32, Free: 31, Allocs: 14, Frees: 13, Fails: 0
> VM zone 'socket':
>   Total: 32, Free: 0, Allocs: 10319, Frees: 3971, Fails: 6316
> Misc mpool: total  131056, free   79008, max free block 77972
> Mbufs pool: total  130944, free  128768, blocksize  128
> Clust pool: total  262144, free  247808, blocksize 2048
> *********************************************************** 
> 
> -Alok


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] Re: Network TCP Handler: stale socket disposal

[Lars Povlsen]

Well,

Here's one more in favour... I reported a similar problem some moths
back, seen with MSIE 6.0 interacting with ATHTTPD. MSIE 6.0 is very
liberal in its use of sockets (actually quite chaotic...) - when the
problem is seen its opening sockets, sending data ("GET /...") - and
closing the socket again before getting ack on the data or even reading
data from the server.

The patch works, at least on the problems I was seeing.

---Lars

-----Original Message-----
From: ecos-discuss-owner@ecos.sourceware.org
[mailto:ecos-discuss-owner@ecos.sourceware.org] On Behalf Of John Mills
Sent: 28. september 2007 15:31
To: eCos Users
Cc: Alok Singh
Subject: RE: [ECOS] Re: Network TCP Handler: stale socket disposal

Alok -

Thanks for your feedback. That makes the success rate 50:50 (2 of 4 
respondents) for the patch.

The web server in our product is a somewhat secondary administrative 
function and we serve simple content that we control 100%. That allows
me 
to summarily close many browser inquiries. I have added and fixed the
POST 
code so it handles our binary firmware images. Either of these may have 
closed some vulnerabilities that may be affecting you - I don't know.

Operationally our problem was triggered by vulnerability scanners used
by
our customers' SysAdmins. These locked up our product in the course of
their overall test scenarios. In principle that meant we were also
vlunerable to the corresponding hacker exploits. That's what I meant in
an
earlier post about a specific, observed functional problem.

The lock-up is broader than just web service. When the socket-descriptor
pool ('zone') is depleted, no new net sockets can be allocated. This
affected other, primary functions in our product, making it a critical 
issue.

I traced the problem by putting 'diag_printf' lines at points where
sockets were created and deallocated, working down to find "what didn't
happen" when a socket was lost. Sounds like you have the same road ahead

of you.

Thanks again for your reaponse.

 - John Mills

On Fri, 28 Sep 2007, Alok Singh wrote:

> Hi John/everybody,
> The patch didn't work for me.  I still had all the sockets exhausted,
and so the web server hangs, and doesn't accept any new connection.  The
number of sockets I'm creating while configuring ECOS is 32.  Please see
the dump of " cyg_kmem_print_stats()" below when the problem comes.
> 
> Test case: I've a script that opens and closes connection to web
server every second. It takes around 2 hours to exhaust the SOCKETS zone
of sockets. The TCP zone of sockets also comes down. Even if I stop the
script, the sockets never recover. I'm currently debugging it(trying to
understand  TCP by reading Comer and stevens).
> 
> Any ideas are welcome.
> 
> ***************
> cyg_kmem_print_stats() -
> Network stack mbuf stats:
>    mbufs 32, clusters 6, free clusters 6
>    Failed to get 0 times
>    Waited to get 0 times
>    Drained queues to get 0 times
> VM zone 'ripcb':
>   Total: 32, Free: 32, Allocs: 0, Frees: 0, Fails: 0
> VM zone 'tcpcb':
>   Total: 32, Free: 1, Allocs: 3989, Frees: 3958, Fails: 0
> VM zone 'udpcb':
>   Total: 32, Free: 31, Allocs: 14, Frees: 13, Fails: 0
> VM zone 'socket':
>   Total: 32, Free: 0, Allocs: 10319, Frees: 3971, Fails: 6316
> Misc mpool: total  131056, free   79008, max free block 77972
> Mbufs pool: total  130944, free  128768, blocksize  128
> Clust pool: total  262144, free  247808, blocksize 2048
> *********************************************************** 
> 
> -Alok


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: Network TCP Handler: stale socket disposal

[Weili Yao]

Hello John, Gary, and Andrew:
   It seemed that John's patch proposal in the following post 
      https://www.sourceware.org/ml/ecos-patches/2007-08/msg00059.html 
   was not adopted by ECOS, but the patch indeed works for us. I put two devices using ECOS and put in our cooperation LAN side by side, one device with the patch and the other device without the patch. The device without the patch lost socketIDs, and the device with the patch has been running great. The backtrace shows as follows. I am not sure how to reproduce the problem, but when the devices are put in a big network, the patch code will be triggered and helped the free of socketIDs. Without the patch, the system slowly runs out of sockets and caused the system unaccessable.  
    My question is when the patch will be officially adopted by ECOS?
    Thank you a lot!

Weili
    
(ECONNRESET is the so->error value) when reaching the patch code below:

(gdb) list *0x0010C8F0
0x10c8f0 is in debug_socket_patch (ecos/web/web_netinfo.c:501).
496     #include <sys/socketvar.h>
497     void debug_socket_patch(struct socket *so)
498     {
499       static int count = 0;
500       count++;
501       app_warning_error(fe_expro(), (count << 16) + so->so_error);
502     }
503
504     // end of the file
505
(gdb) list *0x003141D4
0x3141d4 is in cyg_sofree (/ecos_build/ecos_sources/packages/net/bsd_tcpip/current/src/sys/kern/uipc_socket.c:260).
255                             head->so_incqlen--;
256                     } else if (so->so_state & SS_COMP) {
257
258                 if((so->so_error == ECONNRESET) ||
259                    (so->so_error == ECONNREFUSED)){ // forced drop if flagged
260                       debug_socket_patch(so);
261                       TAILQ_REMOVE(&head->so_comp, so, so_list);
262                       head->so_qlen--;
263                 }
264                 else
(gdb) list *0x003048F8
0x3048f8 is in cyg_in_pcbdetach (/ecos_build/ecos_sources/packages/net/bsd_tcpip/current/src/sys/netinet/in_pcb.c:528).
523             ipsec4_delete_pcbpolicy(inp);
524     #endif /*IPSEC*/
525             inp->inp_gencnt = ++ipi->ipi_gencnt;
526             in_pcbremlists(inp);
527             so->so_pcb = 0;
528             sofree(so);
529             if (inp->inp_options)
530                     (void)m_free(inp->inp_options);
531             if (rt) {
532                     /*
(gdb) list *0x00310FDC
0x310fdc is in cyg_tcp_close (/ecos_build/ecos_sources/packages/net/bsd_tcpip/current/src/sys/netinet/tcp_subr.c:748).
743     #ifdef INET6
744             if (INP_CHECK_SOCKAF(so, AF_INET6))
745                     in6_pcbdetach(inp);
746             else
747     #endif /* INET6 */
748             in_pcbdetach(inp);
749             tcpstat.tcps_closed++;
750             return ((struct tcpcb *)0);
751     }
752
(gdb) list *0x0030DB58
0x30db58 is in cyg_tcp_input (/ecos_build/ecos_sources/packages/net/bsd_tcpip/current/src/sys/netinet/tcp_input.c:2074).
2069                     * If our FIN is now acknowledged, delete the TCB,
2070                     * enter the closed state and return.
2071                     */
2072                    case TCPS_LAST_ACK:
2073                            if (ourfinisacked) {
2074                                    tp = tcp_close(tp);
2075                                    goto drop;
2076                            }
2077                            break;
2078
(gdb) list *0x00308700
0x308700 is in cyg_ip_input (/ecos_build/ecos_sources/packages/net/bsd_tcpip/current/src/sys/netinet/ip_input.c:951).
946               ip->ip_len -= hlen; // subtract the hlen
947         }
948
949     upper_layer:
950
951             (*inetsw[ip_protox[ip->ip_p]].pr_input)(m, off);
952     #ifdef  IPFIREWALL_FORWARD
953             ip_fw_fwd_addr = NULL;  /* tcp needed it */
954     #endif
955             return;
(gdb) list *0x003087A8
0x3087a8 is in ipintr (/ecos_build/ecos_sources/packages/net/bsd_tcpip/current/src/sys/netinet/ip_input.c:979).
974                     s = splimp();
975                     IF_DEQUEUE(&ipintrq, m);
976                     splx(s);
977                     if (m == 0)
978                             return;
979                     ip_input(m);
980             }
981     }
982
983     /*
(gdb) list *0x002F987C
0x2f987c is in cyg_netint (/ecos_build/ecos_sources/packages/net/bsd_tcpip/current/src/ecos/support.c:757).
752                                    CYG_FLAG_WAITMODE_OR|CYG_FLAG_WAITMODE_CLR);
753             spl = splsoftnet(); // Prevent any overlapping "stack" processing
754             for (lvl = NETISR_MIN;  lvl <= NETISR_MAX;  lvl++) {
755                 if (curisr & (1<<lvl)) {
756                     if (NULL != _netisr_handlers[lvl])
757                         (*_netisr_handlers[lvl])();
758                 }
759             }
760             splx(spl);
761         }
(gdb) list *0x002D14B4
0x2d14b4 is in Cyg_HardwareThread::thread_entry(Cyg_Thread*) (/ecos_build/ecos_sources/packages/kernel/current/src/common/thread.cxx:94).
89          Cyg_Scheduler::scheduler.thread_entry( thread );
90
91          // Call entry point in a loop.
92          for(;;)
93          {
94              thread->entry_point(thread->entry_data);
95              thread->exit();
96          }
97      }
98
(gdb) list *0x002D147C
0x2d147c is in Cyg_Thread::exit() (/ecos_build/ecos_sources/packages/kernel/current/src/common/thread.cxx:781).
776
777             Cyg_Scheduler::scheduler.rem_thread(self);
778         }
779
780         Cyg_Scheduler::reschedule();
781     }
782
783     // -------------------------------------------------------------------------
784     // Kill thread. Force the thread into EXITED state externally, or
785     // make it wake up and call exit().
(gdb)

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss