[ECOS] What causes interrupted network system calls?

[ECOS] What causes interrupted network system calls?

[Grant Edwards]

Somebody working on a network-intensive eCos application
reported to me that in situations where there are frequent
connection open/close operations, the various socket-related
system calls (send, connect, etc.) are frequently returning
-EINTR.

This application was recently ported from the old NetBSD
network stack to the FreeBSD stack, and the -EINTR return
values were never seen with the old stack.

Is the frequent -EINTR return expected behavior for the new
stack?

Is there a way to get the stack to restart system calls, or is
application code required to check for -EINTR and retry system
calls?

-- 
Grant Edwards                   grante             Yow! I'm continually AMAZED
                                  at               at th'breathtaking effects
                               visi.com            of WIND EROSION!!


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] What causes interrupted network system calls?

[Carruth, Rusty]

In my experience with Solaris and Linux, you should ALWAYS check for
EINTR and retry the operation.  As I remember, you wouldn't get them
often, but you could (and of course, it was always at the worst possible
moment!).

If I remember right (which may not be the case!).

I don't suppose you have the Stevens book?  What does it say about this?

Rusty

Rusty Carruth | CSE
SSD Storage Products Division | SMART Modular Technologies
Adtron Corporation | 4415 E. Cotton Center Blvd | Phoenix, AZ 85040
Office:  602-735-0300 | Fax: 602-735-0349 | Email:
Rusty.Carruth@smartm.com | www.smartm.com


 
> -----Original Message-----
> From: ecos-discuss-owner@ecos.sourceware.org [mailto:ecos-discuss-
> owner@ecos.sourceware.org] On Behalf Of Grant Edwards
> Sent: Wednesday, October 28, 2009 9:06 AM
> To: ecos-discuss@sources.redhat.com
> Subject: [ECOS] What causes interrupted network system calls?
> 
> Somebody working on a network-intensive eCos application
> reported to me that in situations where there are frequent
> connection open/close operations, the various socket-related
> system calls (send, connect, etc.) are frequently returning
> -EINTR.
> 
> This application was recently ported from the old NetBSD
> network stack to the FreeBSD stack, and the -EINTR return
> values were never seen with the old stack.
> 
> Is the frequent -EINTR return expected behavior for the new
> stack?
> 
> Is there a way to get the stack to restart system calls, or is
> application code required to check for -EINTR and retry system
> calls?
> 
> --


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: What causes interrupted network system calls?

[Grant Edwards]

On 2009-10-28, Carruth, Rusty <Rusty.Carruth@smartm.com> wrote:

> In my experience with Solaris and Linux, you should ALWAYS check for
> EINTR and retry the operation.

That's generally true.

> As I remember, you wouldn't get them often, but you could (and
> of course, it was always at the worst possible moment!).
>
> If I remember right (which may not be the case!).

I agree that re-trying the operation is probably the correct
solution, but I'm a little puzzled because

 1) in the same situation we never saw EINTR with the previous
    network stack.

 2) EINTR isn't listed as one of the possible return values for
    send(), but we seem to be getting it:
    http://ecos.sourceware.org/docs-3.0/ref/net-common-tcpip-manpages-send.html

> I don't suppose you have the Stevens book?  What does it say
> about this?

I have TCP/IP Illustrated v1, but it doesn't address eCos
system call behavior.

-- 
Grant Edwards                   grante             Yow! ANN JILLIAN'S HAIR
                                  at               makes LONI ANDERSON'S
                               visi.com            HAIR look like RICARDO
                                                   MONTALBAN'S HAIR!


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: What causes interrupted network system calls?

[Nick Garnett]

 Grant Edwards <grant.b.edwards@gmail.com> writes:

>  1) in the same situation we never saw EINTR with the previous
>     network stack.

It is probable that the OpenBSD stack totally ignored signals and
therefore wouldn't necessarily do the right thing. The FreeBSD stack
is better integrated with POSIX support.


>  2) EINTR isn't listed as one of the possible return values for
>     send(), but we seem to be getting it:
>     http://ecos.sourceware.org/docs-3.0/ref/net-common-tcpip-manpages-send.html
>

Very weird. The current FreeBSD documentation also does not mention
EINTR. However Linux man pages do, as does the POSIX standard.

In general net stack calls return EINTR only if the thread is broken
out of a wait by cyg_thread_release(), and only POSIX signal handling
generally does that. If the application is using signals, then maybe
the signal masks need to be set up differently.

-- 
Nick Garnett                                       eCos Kernel Architect
eCosCentric Limited    http://www.eCosCentric.com       The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales:                         Reg No: 4422071


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: What causes interrupted network system calls?

[Grant Edwards]

On 2009-10-28, Nick Garnett <nickg@ecoscentric.com> wrote:
>  Grant Edwards <grant.b.edwards@gmail.com> writes:
>
>>  1) in the same situation we never saw EINTR with the previous
>>     network stack.
>
> It is probable that the OpenBSD stack totally ignored signals
> and therefore wouldn't necessarily do the right thing. The
> FreeBSD stack is better integrated with POSIX support.

We're not using POSIX support.

>>  2) EINTR isn't listed as one of the possible return values for
>>     send(), but we seem to be getting it:
>>     http://ecos.sourceware.org/docs-3.0/ref/net-common-tcpip-manpages-send.html
>
> Very weird. The current FreeBSD documentation also does not
> mention EINTR. However Linux man pages do, as does the POSIX
> standard.
>
> In general net stack calls return EINTR only if the thread is
> broken out of a wait by cyg_thread_release(), and only POSIX
> signal handling generally does that.

We don't have any of the POSIX support enabled.

> If the application is using signals,

AFAIK, it isn't.

> then maybe the signal masks need to be set up differently.

I've just been told that retrying doesn't seem to help.  Once
you get an -EINTR, it will happen 5-10 times in a row.

-- 
Grant Edwards                   grante             Yow! Did you move a lot of
                                  at               KOREAN STEAK KNIVES this
                               visi.com            trip, Dingy?


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: What causes interrupted network system calls?

[Nick Garnett]

 Grant Edwards <grant.b.edwards@gmail.com> writes:

> On 2009-10-28, Nick Garnett <nickg@ecoscentric.com> wrote:
> >  Grant Edwards <grant.b.edwards@gmail.com> writes:
> >
> >>  1) in the same situation we never saw EINTR with the previous
> >>     network stack.
> >
> > It is probable that the OpenBSD stack totally ignored signals
> > and therefore wouldn't necessarily do the right thing. The
> > FreeBSD stack is better integrated with POSIX support.
> 
> We're not using POSIX support.

In which case there is something else going on. The only place that
the FreeBSD stack generates an EINTR result is in cyg_tsleep() when
the call to cyg_semaphore_timed_wait() returns false, and it is not a
timeout. Something must either be zapping that semaphore in some way,
or the thread is being released by something else. The OpenBSD stack
has similar code, so it is not immediately clear why one stack is
doing this and the other not.

Are you sure that the EINTR result is coming from the stack itself and
is not simply a leftover in errno from a previous call? Try zeroing
errno before the call.

Also, in the original message you mentioned -EINTR. While negative
error codes are returned from some drivers, network drivers don't do
this. Negative driver error codes are also negated before being put
into errno.

So, I don't know where -EINTR is coming from, but it should never find
its way into errno. I think your best approach is to track down where
this error code is actually coming from.

-- 
Nick Garnett                                       eCos Kernel Architect
eCosCentric Limited    http://www.eCosCentric.com       The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales:                         Reg No: 4422071


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: What causes interrupted network system calls?

[Grant Edwards]

On 2009-10-28, Nick Garnett <nickg@ecoscentric.com> wrote:
>  Grant Edwards <grant.b.edwards@gmail.com> writes:
>> On 2009-10-28, Nick Garnett <nickg@ecoscentric.com> wrote:
>>>  Grant Edwards <grant.b.edwards@gmail.com> writes:
>>>
>>>>  1) in the same situation we never saw EINTR with the previous
>>>>     network stack.
>>>
>>> It is probable that the OpenBSD stack totally ignored signals
>>> and therefore wouldn't necessarily do the right thing. The
>>> FreeBSD stack is better integrated with POSIX support.
>> 
>> We're not using POSIX support.
>
> In which case there is something else going on. The only place
> that the FreeBSD stack generates an EINTR result is in
> cyg_tsleep() when the call to cyg_semaphore_timed_wait()
> returns false, and it is not a timeout.

Yep, that's the only place I could find.

> Something must either be zapping that semaphore in some way,
> or the thread is being released by something else. The OpenBSD
> stack has similar code, so it is not immediately clear why one
> stack is doing this and the other not.
>
> Are you sure that the EINTR result is coming from the stack
> itself and is not simply a leftover in errno from a previous
> call? Try zeroing errno before the call.

We'll try that.

> Also, in the original message you mentioned -EINTR.
>
> While negative error codes are returned from some drivers,
> network drivers don't do this. Negative driver error codes are
> also negated before being put into errno.
>
> So, I don't know where -EINTR is coming from, but it should
> never find its way into errno. I think your best approach is
> to track down where this error code is actually coming from.

I may have misunderstood the description of the send()
return/errno values.  I'll double-check that.

-- 
Grant Edwards                   grante             Yow! Oh my GOD -- the
                                  at               SUN just fell into YANKEE
                               visi.com            STADIUM!!


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: What causes interrupted network system calls?

[Grant Edwards]

On 2009-10-28, Nick Garnett <nickg@ecoscentric.com> wrote:

>> We're not using POSIX support.
>
> In which case there is something else going on. The only place that
> the FreeBSD stack generates an EINTR result is in cyg_tsleep() when
> the call to cyg_semaphore_timed_wait() returns false, and it is not a
> timeout. Something must either be zapping that semaphore in some way,
> or the thread is being released by something else. The OpenBSD stack
> has similar code, so it is not immediately clear why one stack is
> doing this and the other not.
>
> Are you sure that the EINTR result is coming from the stack itself and
> is not simply a leftover in errno from a previous call? Try zeroing
> errno before the call.
>
> Also, in the original message you mentioned -EINTR.

I was wrong.  send() is returning -1, and then errno contains
EINTR (4).

Setting errno to 0 before calling send results in the same
thing.  This only seems to happen with heavy network traffic on
several sockets and connections being opened and close every
second or two.  Some more experimention has shown that the
first rety will succeed most of the time, but it will
occasionally fail repeatedly (as many as 10 times in a row).

We do have the SNMP agent running, but I didn't find any
alarm() or raise() calls.  What else can cause a signal in eCos?

Are signals thread-specific?

-- 
Grant Edwards                   grante             Yow! Were these parsnips
                                  at               CORRECTLY MARINATED in
                               visi.com            TACO SAUCE?


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: What causes interrupted network system calls?

[Nick Garnett]

 Grant Edwards <grant.b.edwards@gmail.com> writes:

> 
> We do have the SNMP agent running, but I didn't find any
> alarm() or raise() calls.  What else can cause a signal in eCos?
> 
> Are signals thread-specific?


If you don't have the POSIX package then there won't be any signal
deliver of the sort that could affect the net stack.

The net stack interprets any return from an event signal that is not a
proper signal or a timeout as an interrupt and returns EINTR. If POSIX
is not included then something else must be forcing that thread out of
the semaphore wait. Perhaps there are other parts of the system that
call cyg_thread_release() and may be zapping the wrong thread. Or
perhaps the right thread at the wrong time.


-- 
Nick Garnett                                       eCos Kernel Architect
eCosCentric Limited    http://www.eCosCentric.com       The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales:                         Reg No: 4422071


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: What causes interrupted network system calls?

[Grant Edwards]

On 2009-10-29, Nick Garnett <nickg@ecoscentric.com> wrote:
>  Grant Edwards <grant.b.edwards@gmail.com> writes:
>
> If you don't have the POSIX package then there won't be any
> signal deliver of the sort that could affect the net stack.

That's what I thought.

> The net stack interprets any return from an event signal that
> is not a proper signal or a timeout as an interrupt and
> returns EINTR. If POSIX is not included then something else
> must be forcing that thread out of the semaphore wait. Perhaps
> there are other parts of the system that call
> cyg_thread_release()

Yep, I just realized that any call to cyg_thread_release() on a
thread that's doing network stuff could cause an EINTR.  I
don't think "signals" (in the POSIX/libc sense) have anything
to do with it.

> and may be zapping the wrong thread. Or perhaps the right
> thread at the wrong time.

I'm now betting it's going to turn out to be the latter.  It
looks like there is a case where one task calls
cyg_thread_release() to wake up another task (which I think is
the task that's seeing the EINTR).  That bit of synchronization
should probably be done with an event flag or something...

-- 
Grant Edwards                   grante             Yow! Is my fallout shelter
                                  at               termite proof?
                               visi.com            


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: What causes interrupted network system calls?

[Grant Edwards]

On 2009-10-28, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> On 2009-10-28, Nick Garnett <nickg@ecoscentric.com> wrote:
>>  Grant Edwards <grant.b.edwards@gmail.com> writes:
>>
>>>  1) in the same situation we never saw EINTR with the previous
>>>     network stack.
>>
>> It is probable that the OpenBSD stack totally ignored signals
>> and therefore wouldn't necessarily do the right thing. The
>> FreeBSD stack is better integrated with POSIX support.
>
> We're not using POSIX support.

We do have CYGPKG_LIBC_SIGNALS (it comes in with one of the
network templates), but nobody ever calls raise(), and I'm
pretty sure nobody ever calls signal() either.

We're getting pretty good results with retrying the send()
once, and if it fails a second time then closing and reopening
the connection.  Just retrying doesn't seem to help.

Our copy of the network stack is mostly a few years old, so I'm
going to see if I can build with a current snapshot of the
FreeBSD net package.

-- 
Grant Edwards                   grante             Yow! Uh-oh!!  I'm having
                                  at               TOO MUCH FUN!!
                               visi.com            


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: What causes interrupted network system calls?

[Nick Garnett]

 Grant Edwards <grant.b.edwards@gmail.com> writes:

> We do have CYGPKG_LIBC_SIGNALS (it comes in with one of the
> network templates), but nobody ever calls raise(), and I'm
> pretty sure nobody ever calls signal() either.

I don't think that package does any real signal handling. It is just
there to provide the minimal signal mechanism required by the C
library. It wouldn't cause these problems.

> Our copy of the network stack is mostly a few years old, so I'm
> going to see if I can build with a current snapshot of the
> FreeBSD net package.

I doubt that there will be much difference, but it is worth a try.


-- 
Nick Garnett                                       eCos Kernel Architect
eCosCentric Limited    http://www.eCosCentric.com       The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales:                         Reg No: 4422071


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] Re: What causes interrupted network system calls?

[Carruth, Rusty]

> .....
> 
> >  2) EINTR isn't listed as one of the possible return values for
> >     send(), but we seem to be getting it:
> >
http://ecos.sourceware.org/docs-3.0/ref/net-common-tcpip-manpages-
> send.html
> >
> 
> Very weird. The current FreeBSD documentation also does not mention
> EINTR. However Linux man pages do, as does the POSIX standard.
> 
> In general net stack calls return EINTR only if the thread is broken
> out of a wait by cyg_thread_release(), and only POSIX signal handling
> generally does that. If the application is using signals, then maybe
> the signal masks need to be set up differently.
> 

Yes, now I remember - signals.  Yes, signals (and threads) are the most
likely culprit(s) (again, using Solaris and Linux as my base of
reference).

Signals, by the way, are used by asynch I/O (at least for Solaris and
Linux), and probably other non-obvious things.  So just because your
application doesn't explicitly use signals doesn't mean it isn't using
them.

Also, using threads GREATLY increases your chance of EINTR.  (Again,
Solaris and Linux)

I just saw Grant's other email, where he says that they don't think they
are using signals.  However, I've seen cases where signals get used by
the underlying code even though you don't ask for it.  

Does your software (explicitly) use threads?

Grant also said:

> I've just been told that retrying doesn't seem to help.  Once you get
an -> EINTR, it will happen 5-10 times in a row.

That's interesting.  I was going to ask if you could get some sort of
snapshot of what was happening, and came up with a silly idea:

When you get the first EINTR, immediately set up a signal handler to
capture all signals, and see what signal is causing the interrupted
system call.  (I assume eCOS can do that - sorry I'm debugging from a
different frame of reference and with no eCOS experience - but I do
remember having 'much fun' with EINTR, and hope I can help even without
the eCOS knowledge).

Anyway, finding out what signal(s) is (are) causing the interrupt(s) may
give a hint as to where it's (they're) coming from.

Rusty


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss