RE: [ECOS] Not responding to TCP SYN

RE: [ECOS] Not responding to TCP SYN

[Dan Jakubiec]

Hi Lars,

Whenever a TCP connection shuts down, one end of the connection always
"hangs around" in a TCP FIN-WAIT state.  This state is part of the TCP
state machine and is required to make sure that everything has shut
down properly.  It is meant to deal with lost acknowledgments during
the TCP shutdown handshake.

The host that initiates the shutdown is usually the one that remains in
FIN-WAIT.  The FIN-WAIT time is implementation-defined, but is always a
fixed timeout (generally on the order of ~70 seconds).

While a socket is in FIN-WAIT it can't be reused for new connections. 
So if you have lots of connections coming and going, you can often run
out of socket resources.  There are a couple of things you can do about
this:

1) Increase the number of socket resources to meet your maximum
connection rate (with respect to the FIN_WAIT timeout).

2) Reduce the FIN-WAIT timeout in the TCP stack to release the socket
resources sooner.

3) Initiate the shutdown from the other host and let IT deal with the
FIN-WAITs.  This is a good option if the other host is running a
full-fledged OS with "unlimited" resources.

4) Certain network APIs will let you do an "abortive disconnect" which
lets you shut down a socket without going through the graceful
handshake that leads to FIN_WAIT.  This uses the TCP "RST" flag to
"reset" the connection immediately.  The TLI network API lets you do
this, but I'm not aware of an equivalent method via BSD sockets. 
[Unfortunately for you though, eCos doesn't include a TLI API.]


I hope this helps,

--
Dan Jakubiec
Systech Corp



Original message:
-----------------
Hi

Thanks! This seems to be the problem. I'm not exactly a TCP shark :)

Is there any way I can avoid having these sockets hanging in the
FIN_WAIT
state? As far as I can see, it requires that clients close their end of
the socket gracefully. There is no way that I can guarantee that they
will
(I don't write the clients).  Otherwise, this seems to me like an
opportunity to perform DoS attacks.

Increasing the number of sockets doesn't seem like an option. I don't
need
many concurrent connections. I just need to be able to handle a large
amount of connections within a short time.

I'm sorry if this is not eCos-specific.

Lars

On Thu, 11 Dec 2003, Jay Foster wrote:

> Sounds like you probabaly have exhausted your supply of sockets
(they're all
> in the FIN_WAIT state).  You can try increasing the number of
sockets.
>
> Jay
>
> -----Original Message-----
> From: Lars Haugaard Kristensen [mailto:larsk@cs.auc.dk]
> Sent: Thursday, December 11, 2003 12:59 PM
> To: ecos-discuss@sources.redhat.com
> Subject: [ECOS] Not responding to TCP SYN
>
>
> Hi list!
>
> I've been working with eCos for a month or so, and I really like it
so
> far. Unfortunately, I'm having problems with TCP connections. I use
eCos
> from cvs (checkout from december 2nd). I have the same problem in
both of
> the following setups:
>
> 1) Linux synthetic target, FreeBSD TCP/IP stack
> 2) I386 PC target, I82559 NIC, OpenBSD TCP/IP stack
>
> I run the server_test.c test (not modified) on the target machine. On
the
> client (a linux 2.4 machine) I run the following:
>
> $ while true; do echo a | netcat target 7734; done
>
> It works fine. The server sends a message like "Hello
10.0.0.2:31234",
> reads "a\n", and closes the connection. However, after some number of
> iterations the server stops responding. Then, after a while, it
starts
> responding again.
>
> This has been tested in both setups. I have used tcpdump to try to
figure
> out what goes wrong. I've attached a trace file. I used the synthetic
> target (10.0.0.3) and a linux host (10.0.0.2) when doing this trace.
It
> seems that after 15 succesfull connections, the target does not
respond to
> the 16th SYN packet. The client retries with an increasing delay,
until it
> finally gets a SYN ACK reply.
>
> When running on i386 and using the OpenBSD stack I can connect 62
times
> before it stops responding. Otherwise, the behaviour is the same. I
tried
> increasing the CYGPKG_NET_MEM_USAGE option with no effect.
>
> The target seems to still respond to icmp echo and arp requests while
it
> refuses to answer TCP SYN.
>
> I have the same problem with my own applications. I have been unable
to
> make this work smoothly in any configuration.
>
> Any suggestions to what is wrong? Could this be an eCos bug, or am I
doing
> something stupid? :)
>
> Lars Kristensen
>
>


__________________________________
Do you Yahoo!?
New Yahoo! Photos - easier uploading and sharing.
http://photos.yahoo.com/

__________________________________
Do you Yahoo!?
New Yahoo! Photos - easier uploading and sharing.
http://photos.yahoo.com/

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

RE: [ECOS] Not responding to TCP SYN

[Matt Jerdonek]

You can decrease the maximum segment lifetime for the
TCP session.  Once 2*msl expires, the session exits
FIN_WAIT state and becomes available again.

There's two ways to do it ...
1) tcp_msl is a global variable that defaults to 30sec
(I think!!).  Since tcp_msl is a global variable, you
can have your application set it to a smaller value.

2) If you get the eCos from the CVS repository, Andrew
Lunn put in a sysctl interface that will allow you to
manipulate tcp_msl in a more standard way.

-- Matt


--- Lars Haugaard Kristensen <larsk@cs.auc.dk> wrote:
> Hi
> 
> Thanks! This seems to be the problem. I'm not
> exactly a TCP shark :)
> 
> Is there any way I can avoid having these sockets
> hanging in the FIN_WAIT
> state? As far as I can see, it requires that clients
> close their end of
> the socket gracefully. There is no way that I can
> guarantee that they will
> (I don't write the clients).  Otherwise, this seems
> to me like an
> opportunity to perform DoS attacks.
> 
> Increasing the number of sockets doesn't seem like
> an option. I don't need
> many concurrent connections. I just need to be able
> to handle a large
> amount of connections within a short time.
> 
> I'm sorry if this is not eCos-specific.
> 
> Lars
> 
> On Thu, 11 Dec 2003, Jay Foster wrote:
> 
> > Sounds like you probabaly have exhausted your
> supply of sockets (they're all
> > in the FIN_WAIT state).  You can try increasing
> the number of sockets.
> >
> > Jay
> >
> > -----Original Message-----
> > From: Lars Haugaard Kristensen
> [mailto:larsk@cs.auc.dk]
> > Sent: Thursday, December 11, 2003 12:59 PM
> > To: ecos-discuss@sources.redhat.com
> > Subject: [ECOS] Not responding to TCP SYN
> >
> >
> > Hi list!
> >
> > I've been working with eCos for a month or so, and
> I really like it so
> > far. Unfortunately, I'm having problems with TCP
> connections. I use eCos
> > from cvs (checkout from december 2nd). I have the
> same problem in both of
> > the following setups:
> >
> > 1) Linux synthetic target, FreeBSD TCP/IP stack
> > 2) I386 PC target, I82559 NIC, OpenBSD TCP/IP
> stack
> >
> > I run the server_test.c test (not modified) on the
> target machine. On the
> > client (a linux 2.4 machine) I run the following:
> >
> > $ while true; do echo a | netcat target 7734; done
> >
> > It works fine. The server sends a message like
> "Hello 10.0.0.2:31234",
> > reads "a\n", and closes the connection. However,
> after some number of
> > iterations the server stops responding. Then,
> after a while, it starts
> > responding again.
> >
> > This has been tested in both setups. I have used
> tcpdump to try to figure
> > out what goes wrong. I've attached a trace file. I
> used the synthetic
> > target (10.0.0.3) and a linux host (10.0.0.2) when
> doing this trace. It
> > seems that after 15 succesfull connections, the
> target does not respond to
> > the 16th SYN packet. The client retries with an
> increasing delay, until it
> > finally gets a SYN ACK reply.
> >
> > When running on i386 and using the OpenBSD stack I
> can connect 62 times
> > before it stops responding. Otherwise, the
> behaviour is the same. I tried
> > increasing the CYGPKG_NET_MEM_USAGE option with no
> effect.
> >
> > The target seems to still respond to icmp echo and
> arp requests while it
> > refuses to answer TCP SYN.
> >
> > I have the same problem with my own applications.
> I have been unable to
> > make this work smoothly in any configuration.
> >
> > Any suggestions to what is wrong? Could this be an
> eCos bug, or am I doing
> > something stupid? :)
> >
> > Lars Kristensen
> >
> >
> 
> -- 
> Before posting, please read the FAQ:
> http://sources.redhat.com/fom/ecos
> and search the list archive:
> http://sources.redhat.com/ml/ecos-discuss
> 


__________________________________
Do you Yahoo!?
New Yahoo! Photos - easier uploading and sharing.
http://photos.yahoo.com/

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

RE: [ECOS] Not responding to TCP SYN

[Lars Haugaard Kristensen]

Hi

Thanks! This seems to be the problem. I'm not exactly a TCP shark :)

Is there any way I can avoid having these sockets hanging in the FIN_WAIT
state? As far as I can see, it requires that clients close their end of
the socket gracefully. There is no way that I can guarantee that they will
(I don't write the clients).  Otherwise, this seems to me like an
opportunity to perform DoS attacks.

Increasing the number of sockets doesn't seem like an option. I don't need
many concurrent connections. I just need to be able to handle a large
amount of connections within a short time.

I'm sorry if this is not eCos-specific.

Lars

On Thu, 11 Dec 2003, Jay Foster wrote:

> Sounds like you probabaly have exhausted your supply of sockets (they're all
> in the FIN_WAIT state).  You can try increasing the number of sockets.
>
> Jay
>
> -----Original Message-----
> From: Lars Haugaard Kristensen [mailto:larsk@cs.auc.dk]
> Sent: Thursday, December 11, 2003 12:59 PM
> To: ecos-discuss@sources.redhat.com
> Subject: [ECOS] Not responding to TCP SYN
>
>
> Hi list!
>
> I've been working with eCos for a month or so, and I really like it so
> far. Unfortunately, I'm having problems with TCP connections. I use eCos
> from cvs (checkout from december 2nd). I have the same problem in both of
> the following setups:
>
> 1) Linux synthetic target, FreeBSD TCP/IP stack
> 2) I386 PC target, I82559 NIC, OpenBSD TCP/IP stack
>
> I run the server_test.c test (not modified) on the target machine. On the
> client (a linux 2.4 machine) I run the following:
>
> $ while true; do echo a | netcat target 7734; done
>
> It works fine. The server sends a message like "Hello 10.0.0.2:31234",
> reads "a\n", and closes the connection. However, after some number of
> iterations the server stops responding. Then, after a while, it starts
> responding again.
>
> This has been tested in both setups. I have used tcpdump to try to figure
> out what goes wrong. I've attached a trace file. I used the synthetic
> target (10.0.0.3) and a linux host (10.0.0.2) when doing this trace. It
> seems that after 15 succesfull connections, the target does not respond to
> the 16th SYN packet. The client retries with an increasing delay, until it
> finally gets a SYN ACK reply.
>
> When running on i386 and using the OpenBSD stack I can connect 62 times
> before it stops responding. Otherwise, the behaviour is the same. I tried
> increasing the CYGPKG_NET_MEM_USAGE option with no effect.
>
> The target seems to still respond to icmp echo and arp requests while it
> refuses to answer TCP SYN.
>
> I have the same problem with my own applications. I have been unable to
> make this work smoothly in any configuration.
>
> Any suggestions to what is wrong? Could this be an eCos bug, or am I doing
> something stupid? :)
>
> Lars Kristensen
>
>

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

RE: [ECOS] Not responding to TCP SYN

[Jay Foster]

Sounds like you probabaly have exhausted your supply of sockets (they're all
in the FIN_WAIT state).  You can try increasing the number of sockets.

Jay

-----Original Message-----
From: Lars Haugaard Kristensen [mailto:larsk@cs.auc.dk]
Sent: Thursday, December 11, 2003 12:59 PM
To: ecos-discuss@sources.redhat.com
Subject: [ECOS] Not responding to TCP SYN


Hi list!

I've been working with eCos for a month or so, and I really like it so
far. Unfortunately, I'm having problems with TCP connections. I use eCos
from cvs (checkout from december 2nd). I have the same problem in both of
the following setups:

1) Linux synthetic target, FreeBSD TCP/IP stack
2) I386 PC target, I82559 NIC, OpenBSD TCP/IP stack

I run the server_test.c test (not modified) on the target machine. On the
client (a linux 2.4 machine) I run the following:

$ while true; do echo a | netcat target 7734; done

It works fine. The server sends a message like "Hello 10.0.0.2:31234",
reads "a\n", and closes the connection. However, after some number of
iterations the server stops responding. Then, after a while, it starts
responding again.

This has been tested in both setups. I have used tcpdump to try to figure
out what goes wrong. I've attached a trace file. I used the synthetic
target (10.0.0.3) and a linux host (10.0.0.2) when doing this trace. It
seems that after 15 succesfull connections, the target does not respond to
the 16th SYN packet. The client retries with an increasing delay, until it
finally gets a SYN ACK reply.

When running on i386 and using the OpenBSD stack I can connect 62 times
before it stops responding. Otherwise, the behaviour is the same. I tried
increasing the CYGPKG_NET_MEM_USAGE option with no effect.

The target seems to still respond to icmp echo and arp requests while it
refuses to answer TCP SYN.

I have the same problem with my own applications. I have been unable to
make this work smoothly in any configuration.

Any suggestions to what is wrong? Could this be an eCos bug, or am I doing
something stupid? :)

Lars Kristensen

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

[ECOS] Not responding to TCP SYN

[Lars Haugaard Kristensen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1695 bytes --]

Hi list!

I've been working with eCos for a month or so, and I really like it so
far. Unfortunately, I'm having problems with TCP connections. I use eCos
from cvs (checkout from december 2nd). I have the same problem in both of
the following setups:

1) Linux synthetic target, FreeBSD TCP/IP stack
2) I386 PC target, I82559 NIC, OpenBSD TCP/IP stack

I run the server_test.c test (not modified) on the target machine. On the
client (a linux 2.4 machine) I run the following:

$ while true; do echo a | netcat target 7734; done

It works fine. The server sends a message like "Hello 10.0.0.2:31234",
reads "a\n", and closes the connection. However, after some number of
iterations the server stops responding. Then, after a while, it starts
responding again.

This has been tested in both setups. I have used tcpdump to try to figure
out what goes wrong. I've attached a trace file. I used the synthetic
target (10.0.0.3) and a linux host (10.0.0.2) when doing this trace. It
seems that after 15 succesfull connections, the target does not respond to
the 16th SYN packet. The client retries with an increasing delay, until it
finally gets a SYN ACK reply.

When running on i386 and using the OpenBSD stack I can connect 62 times
before it stops responding. Otherwise, the behaviour is the same. I tried
increasing the CYGPKG_NET_MEM_USAGE option with no effect.

The target seems to still respond to icmp echo and arp requests while it
refuses to answer TCP SYN.

I have the same problem with my own applications. I have been unable to
make this work smoothly in any configuration.

Any suggestions to what is wrong? Could this be an eCos bug, or am I doing
something stupid? :)

Lars Kristensen

[-- Attachment #2: Type: APPLICATION/OCTET-STREAM, Size: 36211 bytes --]

[-- Attachment #3: Type: text/plain, Size: 146 bytes --]

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss