[ECOS] TCP Transactions

[ECOS] TCP Transactions

[Matt Jerdonek]

Hello again,

I'm developing an application that sends small
transactions via TCP.  The basic flow is 1) Bring up
the connection 2) Send request 3) Get response 4)
Bring down the connection.  Simple enough ...

But, when running any volume of transactions, all my
socket file descriptors get stuck in TIME_WAIT state. 
When this occurs, my application is effectively
disabled until a file descriptor times out.

I see at least 3 possible fixes:
1) Turn on RFC1644 support (TCP extensions for
transactions).  But it appears as if RFC1644 support
in the FreeBSD stack is not complete.  I'm also
skeptical of support of RFC1644 in other OSs that my
application will be taking to.
2) Allocate more than enough sockets
(CYGPKG_NET_MAXSOCKETS) so that I'll always have some
available.  However, this is wasteful of RAM space.
3) Change the MSL timeout from 30sec to something
smaller.  Is there any interest in making the MSL
timer something configurable?

Any other thoughts are greatly appreciated,
-- Matt

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Andrew Lunn]

On Mon, Oct 20, 2003 at 04:27:11PM -0700, Matt Jerdonek wrote:
> Hello again,
> 
> I'm developing an application that sends small
> transactions via TCP.  The basic flow is 1) Bring up
> the connection 2) Send request 3) Get response 4)
> Bring down the connection.  Simple enough ...
> 
> But, when running any volume of transactions, all my
> socket file descriptors get stuck in TIME_WAIT state. 
> When this occurs, my application is effectively
> disabled until a file descriptor times out.

Is this the client or the server end that has the problem?

Are you doing a setsocketopt() for SO_REUSEADDR?


    Andrew

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Matt Jerdonek]

--- Andrew Lunn <andrew@lunn.ch> wrote:
> On Mon, Oct 20, 2003 at 04:27:11PM -0700, Matt
> Jerdonek wrote:
> > Hello again,
> > 
> > I'm developing an application that sends small
> > transactions via TCP.  The basic flow is 1) Bring
> up
> > the connection 2) Send request 3) Get response 4)
> > Bring down the connection.  Simple enough ...
> > 
> > But, when running any volume of transactions, all
> my
> > socket file descriptors get stuck in TIME_WAIT
> state. 
> > When this occurs, my application is effectively
> > disabled until a file descriptor times out.
> 
> Is this the client or the server end that has the
> problem?
> 
> Are you doing a setsocketopt() for SO_REUSEADDR?
> 
> 
>     Andrew
> 
Andrew,

Thanks for the reply.  My code is the client end of
the connection.  Initially, my code was a call only to
socket() and a call to connect().  I changed it to a
call to socket(), setsockopt(REUSEADDR),
setsockopt(REUSEPORT), bind(), and connect().  In both
cases,  it failed.  Please look at the following code
from bsd_tcpip/ver/src/sys/netinet/tcp_usrreq.c
-------------------------------------
/*
 * Cannot simply call in_pcbconnect, because there
 * might be an
 * earlier incarnation of this same connection still
in
 * TIME_WAIT state, creating an ADDRINUSE error.
 */
oinp = in_pcblookup_hash(...);
if (oinp) {
	if (...RFC1644 check...)
	        otp = tcp_close(otp);
	else
		return EADDRINUSE;
	} 
-----------------------------------
I think I have to find a method to shorten the
TIME_WAIT state.  The TIME_WAIT state is defined as 2
* MSL (or 60 seconds).  If I make the MSL
configurable, I can shorten the TIME_WAIT.  Is this
reasonable?

Thanks,
-- Matt


__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[David Vrabel]

Matt Jerdonek wrote:
> 
> I'm developing an application that sends small
> transactions via TCP.  The basic flow is 1) Bring up
> the connection 2) Send request 3) Get response 4)
> Bring down the connection.  Simple enough ...

Have you considered using UDP and manually handling any required retries?

David Vrabel
-- 
David Vrabel, Design Engineer

Arcom                         Tel: +44 (0)1223 411200 ext. 3233
Clifton Road                  Fax: +44 (0)1223 403400
Cambridge CB1 7EA             E-mail: dvrabel@arcom.com
UK                            Web: http://www.arcom.com/


_____________________________________________________________________
The message in this transmission is sent in confidence for the attention of the addressee only and should not be disclosed to any other party. Unauthorised recipients are requested to preserve this confidentiality. Please advise the sender if the addressee is not resident at the receiving end.

This message has been checked for all viruses by MessageLabs Virus Control Centre.

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Matt Jerdonek]

I'd love to use UDP, but the project requirements
specify TCP.  

Does anyone know how other systems handle the
TIME_WAIT situation?

Thanks,
-- Matt


--- David Vrabel <dvrabel@arcom.com> wrote:
> Matt Jerdonek wrote:
> > 
> > I'm developing an application that sends small
> > transactions via TCP.  The basic flow is 1) Bring
> up
> > the connection 2) Send request 3) Get response 4)
> > Bring down the connection.  Simple enough ...
> 
> Have you considered using UDP and manually handling
> any required retries?
> 
> David Vrabel
> -- 
> David Vrabel, Design Engineer


__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Andrew Lunn]

> 3) Change the MSL timeout from 30sec to something
> smaller.  Is there any interest in making the MSL
> timer something configurable?

It already is configurable.

1) Uses sysctl(). Its something like net.inet.tcp.msl. You can
   probably find its proper name using the sysctl test program.

2) The network stack makes a lot of its variables global. tcp_msl is
   no exception. But there is a bit of jiggery-pokery going on, so its
   actually called cyg_tcp_msl. So just do something like...

extern int cyg_tcp_msl;

/* 'Orrible hack warning. Play around with the internal timer
variables of the network stack. Yuck. */
cyg_tcp_msl = 10;

I would spend some time to see what else uses this timer. This might
change other things are well as how long sockets take to close.

       Andrew


-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Matt Jerdonek]

> 
> 1) Uses sysctl(). Its something like
> net.inet.tcp.msl. You can probably find its 
> proper name using the sysctl test program.

I downloaded the current FreeBSD stack to use your
sysctl updates (Thanks for adding this!).  Anyway, msl
isn't an option.  So, I'm stuck using global memory
for now until I extend the sysctl interface.  Is there
any standard sysctl template?  Can I just add it to
the end of the list in tcp_var.h?

Also, I need to control the TCP retransmissions, which
is set to 12, and is way too slow for transactions. 
Since rexmits is controlled by TCP_MAXRXTSHIFT
(tcp_timer.h) and is not accessible via global memory,
is my best option to make it configurable via the cdl?

Thanks,
-- Matt
> 
> 2) The network stack makes a lot of its variables
> global. tcp_msl is
>    no exception. But there is a bit of
> jiggery-pokery going on, so its
>    actually called cyg_tcp_msl. So just do something
> like...
> 
> extern int cyg_tcp_msl;
> 
> /* 'Orrible hack warning. Play around with the
> internal timer
> variables of the network stack. Yuck. */
> cyg_tcp_msl = 10;
> 
 


__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Andrew Lunn]

On Thu, Oct 23, 2003 at 11:32:41AM -0700, Matt Jerdonek wrote:
> > 
> > 1) Uses sysctl(). Its something like
> > net.inet.tcp.msl. You can probably find its 
> > proper name using the sysctl test program.
> 
> I downloaded the current FreeBSD stack to use your
> sysctl updates (Thanks for adding this!).  Anyway, msl
> isn't an option.

Humm. 

packages/net/bsd_tcpip/current/src/sys/netinet/tcp_timer.125:
int	tcp_msl;
SYSCTL_PROC(_net_inet_tcp, OID_AUTO, msl, CTLTYPE_INT|CTLFLAG_RW,
    &tcp_msl, 0, sysctl_msec_to_ticks, "I", "Maximum segment lifetime");

So it should be there.

Running the test program 
./install/tests/net/bsd_tcpip/current/tests/sysctl1

INFO:<4 2 6 106 = net.inet.tcp.local_slowstart_flightsize (INT Read Write ){6}>
INFO:<4 2 6 107 = net.inet.tcp.newreno (INT Read Write ){6}>
INFO:<4 2 6 108 = net.inet.tcp.msl (INT Read Write ){6}>
INFO:<4 2 6 109 = net.inet.tcp.always_keepalive (INT Read Write ){6}>
INFO:<4 2 6 110 = net.inet.tcp.tcbhashsize (INT Read ){6}>


> So, I'm stuck using global memory
> for now until I extend the sysctl interface.  Is there
> any standard sysctl template?  Can I just add it to
> the end of the list in tcp_var.h?
> 
> Also, I need to control the TCP retransmissions, which
> is set to 12, and is way too slow for transactions. 
> Since rexmits is controlled by TCP_MAXRXTSHIFT
> (tcp_timer.h) and is not accessible via global memory,
> is my best option to make it configurable via the cdl?

Hummm, i don't think it works like that. I think a retransmit occurs
about 2-4 times the round trip time. There is not much you can do
about this. Also, the timer resolution is probably 50ms. So don't
expect anything faster than that. 

You might want to get hold of the book:

The Implementation (TCP/IP Illustrated, Volume 2)
by Gary R. Wright (Author), W. Richard Stevens (Author) 

Its the best book i know about for the insides of *BSD stacks.

    Andrew

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Matt Jerdonek]

Andrew,

Thanks for the clarification on the sysctl.

Regarding the retransmissions ... I'm not trying to
change the retransmit interval, but I'm trying to
limit the retransmit attempts.

For example, if the retransmit interval is 1sec, then
the retransmit attempts will occur at 
1s, 2s, 4s, 8s, 16s, 32s, 64s, 64s, 64s, 64s etc ...
(12 times).

This is obviously way too long to wait for a
transaction timeout. (Imagine standing in line at a
checkout for 10 minutes for your transaction to
finish).  In my company's existing products, we limit
the TCP retry attempts to 4, which provides a more
reasonable timeout.

So, I think I need to change the TCP_MAXRXTSHIFT from
12 to another value.  Maybe I can assign this to a
global variable and use the sysctl interface to
manipulate it?

Any thoughts are appreciated,
-- Matt

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP Transactions

[Andrew Lunn]

> So, I think I need to change the TCP_MAXRXTSHIFT from
> 12 to another value.  Maybe I can assign this to a
> global variable and use the sysctl interface to
> manipulate it?

Sounds reasonable. I would first look at the latest FreeBSD code and
see if they have added anything like that. If so i would copy
it. Otherwise make it up yourself.

    Andrew

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss