* [ECOS] TCP Transactions @ 2003-10-20 23:27 Matt Jerdonek 2003-10-21 7:16 ` Andrew Lunn ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Matt Jerdonek @ 2003-10-20 23:27 UTC (permalink / raw) To: Discussion eCos Hello again, I'm developing an application that sends small transactions via TCP. The basic flow is 1) Bring up the connection 2) Send request 3) Get response 4) Bring down the connection. Simple enough ... But, when running any volume of transactions, all my socket file descriptors get stuck in TIME_WAIT state. When this occurs, my application is effectively disabled until a file descriptor times out. I see at least 3 possible fixes: 1) Turn on RFC1644 support (TCP extensions for transactions). But it appears as if RFC1644 support in the FreeBSD stack is not complete. I'm also skeptical of support of RFC1644 in other OSs that my application will be taking to. 2) Allocate more than enough sockets (CYGPKG_NET_MAXSOCKETS) so that I'll always have some available. However, this is wasteful of RAM space. 3) Change the MSL timeout from 30sec to something smaller. Is there any interest in making the MSL timer something configurable? Any other thoughts are greatly appreciated, -- Matt __________________________________ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-20 23:27 [ECOS] TCP Transactions Matt Jerdonek @ 2003-10-21 7:16 ` Andrew Lunn 2003-10-21 15:42 ` Matt Jerdonek 2003-10-21 16:06 ` David Vrabel 2003-10-21 19:11 ` Andrew Lunn 2 siblings, 1 reply; 10+ messages in thread From: Andrew Lunn @ 2003-10-21 7:16 UTC (permalink / raw) To: Matt Jerdonek; +Cc: Discussion eCos On Mon, Oct 20, 2003 at 04:27:11PM -0700, Matt Jerdonek wrote: > Hello again, > > I'm developing an application that sends small > transactions via TCP. The basic flow is 1) Bring up > the connection 2) Send request 3) Get response 4) > Bring down the connection. Simple enough ... > > But, when running any volume of transactions, all my > socket file descriptors get stuck in TIME_WAIT state. > When this occurs, my application is effectively > disabled until a file descriptor times out. Is this the client or the server end that has the problem? Are you doing a setsocketopt() for SO_REUSEADDR? Andrew -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-21 7:16 ` Andrew Lunn @ 2003-10-21 15:42 ` Matt Jerdonek 0 siblings, 0 replies; 10+ messages in thread From: Matt Jerdonek @ 2003-10-21 15:42 UTC (permalink / raw) To: Andrew Lunn; +Cc: Discussion eCos --- Andrew Lunn <andrew@lunn.ch> wrote: > On Mon, Oct 20, 2003 at 04:27:11PM -0700, Matt > Jerdonek wrote: > > Hello again, > > > > I'm developing an application that sends small > > transactions via TCP. The basic flow is 1) Bring > up > > the connection 2) Send request 3) Get response 4) > > Bring down the connection. Simple enough ... > > > > But, when running any volume of transactions, all > my > > socket file descriptors get stuck in TIME_WAIT > state. > > When this occurs, my application is effectively > > disabled until a file descriptor times out. > > Is this the client or the server end that has the > problem? > > Are you doing a setsocketopt() for SO_REUSEADDR? > > > Andrew > Andrew, Thanks for the reply. My code is the client end of the connection. Initially, my code was a call only to socket() and a call to connect(). I changed it to a call to socket(), setsockopt(REUSEADDR), setsockopt(REUSEPORT), bind(), and connect(). In both cases, it failed. Please look at the following code from bsd_tcpip/ver/src/sys/netinet/tcp_usrreq.c ------------------------------------- /* * Cannot simply call in_pcbconnect, because there * might be an * earlier incarnation of this same connection still in * TIME_WAIT state, creating an ADDRINUSE error. */ oinp = in_pcblookup_hash(...); if (oinp) { if (...RFC1644 check...) otp = tcp_close(otp); else return EADDRINUSE; } ----------------------------------- I think I have to find a method to shorten the TIME_WAIT state. The TIME_WAIT state is defined as 2 * MSL (or 60 seconds). If I make the MSL configurable, I can shorten the TIME_WAIT. Is this reasonable? Thanks, -- Matt __________________________________ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-20 23:27 [ECOS] TCP Transactions Matt Jerdonek 2003-10-21 7:16 ` Andrew Lunn @ 2003-10-21 16:06 ` David Vrabel 2003-10-21 17:49 ` Matt Jerdonek 2003-10-21 19:11 ` Andrew Lunn 2 siblings, 1 reply; 10+ messages in thread From: David Vrabel @ 2003-10-21 16:06 UTC (permalink / raw) To: Discussion eCos Matt Jerdonek wrote: > > I'm developing an application that sends small > transactions via TCP. The basic flow is 1) Bring up > the connection 2) Send request 3) Get response 4) > Bring down the connection. Simple enough ... Have you considered using UDP and manually handling any required retries? David Vrabel -- David Vrabel, Design Engineer Arcom Tel: +44 (0)1223 411200 ext. 3233 Clifton Road Fax: +44 (0)1223 403400 Cambridge CB1 7EA E-mail: dvrabel@arcom.com UK Web: http://www.arcom.com/ _____________________________________________________________________ The message in this transmission is sent in confidence for the attention of the addressee only and should not be disclosed to any other party. Unauthorised recipients are requested to preserve this confidentiality. Please advise the sender if the addressee is not resident at the receiving end. This message has been checked for all viruses by MessageLabs Virus Control Centre. -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-21 16:06 ` David Vrabel @ 2003-10-21 17:49 ` Matt Jerdonek 0 siblings, 0 replies; 10+ messages in thread From: Matt Jerdonek @ 2003-10-21 17:49 UTC (permalink / raw) To: David Vrabel, Discussion eCos I'd love to use UDP, but the project requirements specify TCP. Does anyone know how other systems handle the TIME_WAIT situation? Thanks, -- Matt --- David Vrabel <dvrabel@arcom.com> wrote: > Matt Jerdonek wrote: > > > > I'm developing an application that sends small > > transactions via TCP. The basic flow is 1) Bring > up > > the connection 2) Send request 3) Get response 4) > > Bring down the connection. Simple enough ... > > Have you considered using UDP and manually handling > any required retries? > > David Vrabel > -- > David Vrabel, Design Engineer __________________________________ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-20 23:27 [ECOS] TCP Transactions Matt Jerdonek 2003-10-21 7:16 ` Andrew Lunn 2003-10-21 16:06 ` David Vrabel @ 2003-10-21 19:11 ` Andrew Lunn 2003-10-23 18:32 ` Matt Jerdonek 2 siblings, 1 reply; 10+ messages in thread From: Andrew Lunn @ 2003-10-21 19:11 UTC (permalink / raw) To: Matt Jerdonek; +Cc: Discussion eCos > 3) Change the MSL timeout from 30sec to something > smaller. Is there any interest in making the MSL > timer something configurable? It already is configurable. 1) Uses sysctl(). Its something like net.inet.tcp.msl. You can probably find its proper name using the sysctl test program. 2) The network stack makes a lot of its variables global. tcp_msl is no exception. But there is a bit of jiggery-pokery going on, so its actually called cyg_tcp_msl. So just do something like... extern int cyg_tcp_msl; /* 'Orrible hack warning. Play around with the internal timer variables of the network stack. Yuck. */ cyg_tcp_msl = 10; I would spend some time to see what else uses this timer. This might change other things are well as how long sockets take to close. Andrew -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-21 19:11 ` Andrew Lunn @ 2003-10-23 18:32 ` Matt Jerdonek 2003-10-23 20:15 ` Andrew Lunn 0 siblings, 1 reply; 10+ messages in thread From: Matt Jerdonek @ 2003-10-23 18:32 UTC (permalink / raw) To: Andrew Lunn; +Cc: Discussion eCos > > 1) Uses sysctl(). Its something like > net.inet.tcp.msl. You can probably find its > proper name using the sysctl test program. I downloaded the current FreeBSD stack to use your sysctl updates (Thanks for adding this!). Anyway, msl isn't an option. So, I'm stuck using global memory for now until I extend the sysctl interface. Is there any standard sysctl template? Can I just add it to the end of the list in tcp_var.h? Also, I need to control the TCP retransmissions, which is set to 12, and is way too slow for transactions. Since rexmits is controlled by TCP_MAXRXTSHIFT (tcp_timer.h) and is not accessible via global memory, is my best option to make it configurable via the cdl? Thanks, -- Matt > > 2) The network stack makes a lot of its variables > global. tcp_msl is > no exception. But there is a bit of > jiggery-pokery going on, so its > actually called cyg_tcp_msl. So just do something > like... > > extern int cyg_tcp_msl; > > /* 'Orrible hack warning. Play around with the > internal timer > variables of the network stack. Yuck. */ > cyg_tcp_msl = 10; > __________________________________ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-23 18:32 ` Matt Jerdonek @ 2003-10-23 20:15 ` Andrew Lunn 2003-10-24 16:38 ` Matt Jerdonek 0 siblings, 1 reply; 10+ messages in thread From: Andrew Lunn @ 2003-10-23 20:15 UTC (permalink / raw) To: Matt Jerdonek; +Cc: Discussion eCos On Thu, Oct 23, 2003 at 11:32:41AM -0700, Matt Jerdonek wrote: > > > > 1) Uses sysctl(). Its something like > > net.inet.tcp.msl. You can probably find its > > proper name using the sysctl test program. > > I downloaded the current FreeBSD stack to use your > sysctl updates (Thanks for adding this!). Anyway, msl > isn't an option. Humm. packages/net/bsd_tcpip/current/src/sys/netinet/tcp_timer.125: int tcp_msl; SYSCTL_PROC(_net_inet_tcp, OID_AUTO, msl, CTLTYPE_INT|CTLFLAG_RW, &tcp_msl, 0, sysctl_msec_to_ticks, "I", "Maximum segment lifetime"); So it should be there. Running the test program ./install/tests/net/bsd_tcpip/current/tests/sysctl1 INFO:<4 2 6 106 = net.inet.tcp.local_slowstart_flightsize (INT Read Write ){6}> INFO:<4 2 6 107 = net.inet.tcp.newreno (INT Read Write ){6}> INFO:<4 2 6 108 = net.inet.tcp.msl (INT Read Write ){6}> INFO:<4 2 6 109 = net.inet.tcp.always_keepalive (INT Read Write ){6}> INFO:<4 2 6 110 = net.inet.tcp.tcbhashsize (INT Read ){6}> > So, I'm stuck using global memory > for now until I extend the sysctl interface. Is there > any standard sysctl template? Can I just add it to > the end of the list in tcp_var.h? > > Also, I need to control the TCP retransmissions, which > is set to 12, and is way too slow for transactions. > Since rexmits is controlled by TCP_MAXRXTSHIFT > (tcp_timer.h) and is not accessible via global memory, > is my best option to make it configurable via the cdl? Hummm, i don't think it works like that. I think a retransmit occurs about 2-4 times the round trip time. There is not much you can do about this. Also, the timer resolution is probably 50ms. So don't expect anything faster than that. You might want to get hold of the book: The Implementation (TCP/IP Illustrated, Volume 2) by Gary R. Wright (Author), W. Richard Stevens (Author) Its the best book i know about for the insides of *BSD stacks. Andrew -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-23 20:15 ` Andrew Lunn @ 2003-10-24 16:38 ` Matt Jerdonek 2003-10-28 12:50 ` Andrew Lunn 0 siblings, 1 reply; 10+ messages in thread From: Matt Jerdonek @ 2003-10-24 16:38 UTC (permalink / raw) To: Andrew Lunn; +Cc: Discussion eCos Andrew, Thanks for the clarification on the sysctl. Regarding the retransmissions ... I'm not trying to change the retransmit interval, but I'm trying to limit the retransmit attempts. For example, if the retransmit interval is 1sec, then the retransmit attempts will occur at 1s, 2s, 4s, 8s, 16s, 32s, 64s, 64s, 64s, 64s etc ... (12 times). This is obviously way too long to wait for a transaction timeout. (Imagine standing in line at a checkout for 10 minutes for your transaction to finish). In my company's existing products, we limit the TCP retry attempts to 4, which provides a more reasonable timeout. So, I think I need to change the TCP_MAXRXTSHIFT from 12 to another value. Maybe I can assign this to a global variable and use the sysctl interface to manipulate it? Any thoughts are appreciated, -- Matt __________________________________ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ECOS] TCP Transactions 2003-10-24 16:38 ` Matt Jerdonek @ 2003-10-28 12:50 ` Andrew Lunn 0 siblings, 0 replies; 10+ messages in thread From: Andrew Lunn @ 2003-10-28 12:50 UTC (permalink / raw) To: Matt Jerdonek; +Cc: Discussion eCos > So, I think I need to change the TCP_MAXRXTSHIFT from > 12 to another value. Maybe I can assign this to a > global variable and use the sysctl interface to > manipulate it? Sounds reasonable. I would first look at the latest FreeBSD code and see if they have added anything like that. If so i would copy it. Otherwise make it up yourself. Andrew -- Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos and search the list archive: http://sources.redhat.com/ml/ecos-discuss ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2003-10-28 12:50 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2003-10-20 23:27 [ECOS] TCP Transactions Matt Jerdonek 2003-10-21 7:16 ` Andrew Lunn 2003-10-21 15:42 ` Matt Jerdonek 2003-10-21 16:06 ` David Vrabel 2003-10-21 17:49 ` Matt Jerdonek 2003-10-21 19:11 ` Andrew Lunn 2003-10-23 18:32 ` Matt Jerdonek 2003-10-23 20:15 ` Andrew Lunn 2003-10-24 16:38 ` Matt Jerdonek 2003-10-28 12:50 ` Andrew Lunn
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).