[ECOS] TCP stack not delaying/piggybacking ACKs?

[ECOS] TCP stack not delaying/piggybacking ACKs?

[Grant Edwards]

I'm working with an application that exchanges a continuous
stream of data with a host.  Each sends a TCP packet full of
data every 10ms or so.

The eCos stack is not delaying/piggybacking ACKs, so the eCos
app is sending roughly twice as many packets as it needs to.
[I'm running out of CPU throughput, and cutting the number of
TCP packets sent by 50% would really help]. The other end
(Linux host) never sends a lone ACK, since there's never more
than 10ms between data packets. This is an eCos CVS snapshot
from a while back (early 2001).

Have others noticed this behavior?

Has it been fixed?

-- 
Grant Edwards
grante@visi.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Gary Thomas]

On Mon, 2002-07-08 at 17:16, Grant Edwards wrote:
> 
> I'm working with an application that exchanges a continuous
> stream of data with a host.  Each sends a TCP packet full of
> data every 10ms or so.
> 
> The eCos stack is not delaying/piggybacking ACKs, so the eCos
> app is sending roughly twice as many packets as it needs to.
> [I'm running out of CPU throughput, and cutting the number of
> TCP packets sent by 50% would really help]. The other end
> (Linux host) never sends a lone ACK, since there's never more
> than 10ms between data packets. This is an eCos CVS snapshot
> from a while back (early 2001).
> 
> Have others noticed this behavior?
> 
> Has it been fixed?

This has never been mentioned before.  Which is not to say that
it's working properly :-)

A couple of things to try:
  * The new OpenBSD stack.  This would require updating to the
    latest CVS.
  * There may be a flag/switch in the FreeBSD stack (the one you
    are using) which controls this, but it would require some time
    to investigate.

How are you testing this (creating such an environment)?  Perhaps
a simple setup can be created which exhibits the behaviour and then
we can look into why (and how to fix it)


-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Grant Edwards]

On Mon, Jul 08, 2002 at 05:18:54PM -0600, Gary Thomas wrote:

> > I'm working with an application that exchanges a continuous
> > stream of data with a host.  Each sends a TCP packet full of
> > data every 10ms or so.
> > 
> > The eCos stack is not delaying/piggybacking ACKs, so the eCos
> > app is sending roughly twice as many packets as it needs to.
> 
> This has never been mentioned before.  Which is not to say that
> it's working properly :-)

It's also possible I've broken something while mucking with the
stack.  I've replaced the IP checksum routine with assembly
language and made the TCP keepalive timer configurable at run
time.  I don't think either should have affected the ACK delay
mechanism, but that's what they all say...

> A couple of things to try:
>   * The new OpenBSD stack.  This would require updating to the
>     latest CVS.

I may try that, but it will have to wait a bit until I find and
fix the unrelated application bug I was looking for when I
noticed this.

>   * There may be a flag/switch in the FreeBSD stack (the one you
>     are using) which controls this, but it would require some time
>     to investigate.

I've been browsing around the sources and haven't quite figured
out how it's supposed to work.  I do have the TCP_NODELAY flag
set on the socket, but I don't know if that has anything to do
with it.  That is easy enough to test.

> How are you testing this (creating such an environment)?

It is, of course, rather complicated: There's a custom app and
device driver driver on the host end and so on.  I'll see if I
can come up with a simpler setup that also demonstrates the
behavior.

-- 
Grant Edwards
grante@visi.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Mark Salter]

>>>>> Grant Edwards writes:

> I've been browsing around the sources and haven't quite figured
> out how it's supposed to work.  I do have the TCP_NODELAY flag
> set on the socket, but I don't know if that has anything to do
> with it.  That is easy enough to test.

Try turning it off. TCP_NODELAY tells the stack to send ACKs without
waiting a bit for a possible packet to piggyback on.

--Mark

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Grant Edwards]

On Mon, Jul 08, 2002 at 09:36:24PM -0400, Mark Salter wrote:

> > I've been browsing around the sources and haven't quite figured
> > out how it's supposed to work.  I do have the TCP_NODELAY flag
> > set on the socket, but I don't know if that has anything to do
> > with it.  That is easy enough to test.
> 
> Try turning it off.

I'll try it, but that's not an acceptible solution since not
setting NODELAY creates huge data transfer latencies.

> TCP_NODELAY tells the stack to send ACKs without
> waiting a bit for a possible packet to piggyback on.

Really? I just re-read that section of Stevens and it didn't
sound like it was supposed to do that.  I left the book in the
office, but it sounded to me like NODELAY tells the stack to go
ahead and send additional data packets without waiting for the
first one to be ACKed (that's certainly why I have it set).

I don't see why the two features should be linked.  :/




-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Wolfgang Heppner]

Sorry, but now i'm confused:

I thought the OpenBSD stack is the old and the FreeBSD stack is the
newer stack!
see:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~/ecos/packages/NEWS?content-type=text/plain&cvsroot=ecos

Wolfgang


Gary Thomas wrote:
> 
> 
> A couple of things to try:
>   * The new OpenBSD stack.  This would require updating to the
>     latest CVS.
>   * There may be a flag/switch in the FreeBSD stack (the one you
>     are using) which controls this, but it would require some time
>     to investigate.
> 
-- 
Wolfgang Heppner

High Speed Kamerasysteme  - Abt. Bildsensorik (ESY-B)
High Speed Camera Systems - Dept. Electronic Imaging (ESY-B)

Fraunhofer                            
Institut Integrierte Schaltungen
Angewandte Elektronik

Am Wolfsmantel 33 
91058 Erlangen       
(Germany)

Tel:   +49 (0) 9131 776 528
Fax:   +49 (0) 9131 776 598

email: hep@iis.fhg.de
web:   http://www.iis.fhg.de

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Gary Thomas]

On Tue, 2002-07-09 at 08:49, Wolfgang Heppner wrote:
> Sorry, but now i'm confused:
> 
> I thought the OpenBSD stack is the old and the FreeBSD stack is the
> newer stack!
> see:
> http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~/ecos/packages/NEWS?content-type=text/plain&cvsroot=ecos
> 

Indeed, you have it right (my brain was still on holiday!)
  .../net/tcpip/ - OpenBSD stack from early 2000
  .../net/bsd_tcpip - FreeBSD stack from late 2001

> Wolfgang
> 
> 
> Gary Thomas wrote:
> > 
> > 
> > A couple of things to try:
> >   * The new OpenBSD stack.  This would require updating to the
> >     latest CVS.
> >   * There may be a flag/switch in the FreeBSD stack (the one you
> >     are using) which controls this, but it would require some time
> >     to investigate.
> > 
> -- 
> Wolfgang Heppner
> 
> High Speed Kamerasysteme  - Abt. Bildsensorik (ESY-B)
> High Speed Camera Systems - Dept. Electronic Imaging (ESY-B)
> 
> Fraunhofer                            
> Institut Integrierte Schaltungen
> Angewandte Elektronik
> 
> Am Wolfsmantel 33 
> 91058 Erlangen       
> (Germany)
> 
> Tel:   +49 (0) 9131 776 528
> Fax:   +49 (0) 9131 776 598
> 
> email: hep@iis.fhg.de
> web:   http://www.iis.fhg.de



-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Grant Edwards]

On Mon, Jul 08, 2002 at 09:36:24PM -0400, Mark Salter wrote:
> >>>>> Grant Edwards writes:
> 
> > I've been browsing around the sources and haven't quite figured
> > out how it's supposed to work.  I do have the TCP_NODELAY flag
> > set on the socket, but I don't know if that has anything to do
> > with it.  That is easy enough to test.
> 
> Try turning it off. TCP_NODELAY tells the stack to send ACKs without
> waiting a bit for a possible packet to piggyback on.

Turning off TCP_NODELAY doesn't make any difference.  ACKs are
still sent out immediately.

I've read through the TCP stack sources and AFAICT, the only
place where NODELAY is checked is when tcp_output is trying to
figure out how much data to send.  [This agrees with my reading
of Stevens.]

Why ACKs aren't being delayed is apparently a different issue...

-- 
Grant Edwards
grante@visi.com

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Mark Salter]

>>>>> Grant Edwards writes:

> On Mon, Jul 08, 2002 at 09:36:24PM -0400, Mark Salter wrote:
>> >>>>> Grant Edwards writes:
>> 
>> > I've been browsing around the sources and haven't quite figured
>> > out how it's supposed to work.  I do have the TCP_NODELAY flag
>> > set on the socket, but I don't know if that has anything to do
>> > with it.  That is easy enough to test.
>> 
>> Try turning it off. TCP_NODELAY tells the stack to send ACKs without
>> waiting a bit for a possible packet to piggyback on.

> Turning off TCP_NODELAY doesn't make any difference.  ACKs are
> still sent out immediately.

> I've read through the TCP stack sources and AFAICT, the only
> place where NODELAY is checked is when tcp_output is trying to
> figure out how much data to send.  [This agrees with my reading
> of Stevens.]

> Why ACKs aren't being delayed is apparently a different issue...

Maybe there's something else at play with eCos, but TCP_NODELAY
certainly does control immediate sending of ACKs (and other packets)
on Solaris, Linux, and Windows. It turns off the Nagle algorithm.
In general, the Nagle algorithm causes small packets (including
ACK-only packets) to be held back in the hopes that they can be
coalesced with following packets. The delay is usually in the
neighborhood of a few 10s of milliseconds although I've seen WNT
wait ~200ms to send an ACK.

--Mark


-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss

Re: [ECOS] TCP stack not delaying/piggybacking ACKs?

[Martin Buck]

Grant Edwards wrote:
> Why ACKs aren't being delayed is apparently a different issue...

Maybe your host is setting the PUSH bit in packets it sends? From
tcp_input.c:

                        if (th->th_flags & TH_PUSH)
                                tp->t_flags |= TF_ACKNOW;
                        else
                                tp->t_flags |= TF_DELACK;


i.e. packets with the PUSH bit set are ACKed immediately, others (as
long as they are in sequence) get a delayed ACK.

Martin

-- 
Before posting, please read the FAQ: http://sources.redhat.com/fom/ecos
and search the list archive: http://sources.redhat.com/ml/ecos-discuss