Help: shutdown(..., SHUT_WR) on TCP sockets

Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Hello,

I've got a few problems properly closing TCP sockets on GLIBC/Linux/ARM,
and getting to the manual only added to the overall confusion.

First, the manual[*] doesn't even mention SHUT_WR, etc. constants,
instead it has raw numbers:

     The argument HOW specifies what action to perform:

     '0'
          Stop receiving data for this socket.  If further data arrives,
          reject it.

          [...]


That seems quite outdated.

2. The SHUT_WR (=1) specification seems to be plain wrong:

     '1'
          Stop trying to transmit data from this socket.  Discard any
          data waiting to be sent.  Stop looking for acknowledgement of
          data already sent; don't retransmit it if it is lost.


Discarding data contradicts both the intended usage of the feature and
the actual behavior (as observed by testing the state of the socket
write queue using 'ioctl(fd, TIOCOUTQ, &val)' call before and after
shutdown()).

Given the above, could please somebody clarify what is the actual
intended behavior of shutdown(..., SHUT_WR) on GLIBC/Linux nowadays?

1. Should it actually discard the data, or instead achieve something
useful? Like, say, what is described in Free-BSD manual[**]:

"Send queued data, wait for ACK, then send FIN."

2. Should it obey SO_LINGER and actually wait for data to be sent? To be
acknowledged[***]? 

3. Should it send 'FIN'?

Footnotes:

[*]  https://www.gnu.org/software/libc/manual/html_node/Closing-a-Socket.html#Closing-a-Socket

[**] https://gsp.com/cgi-bin/man.cgi?section=2&topic=shutdown

[***] My observation is that it rather neither discards the data, nor
waits for anything, no matter what SO_LINGER and ~O_NONBLOCK say, that
is not that bad as discarding the data, but still makes it virtually
impossible to properly wait for data sending or delivery before closing
the socket.

-- Sergey

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Florian Weimer]

* Sergey Organov:

> Hello,
>
> I've got a few problems properly closing TCP sockets on GLIBC/Linux/ARM,
> and getting to the manual only added to the overall confusion.
>
> First, the manual[*] doesn't even mention SHUT_WR, etc. constants,
> instead it has raw numbers:
>
>      The argument HOW specifies what action to perform:
>
>      '0'
>           Stop receiving data for this socket.  If further data arrives,
>           reject it.
>
>           [...]
>
>
> That seems quite outdated.
>
> 2. The SHUT_WR (=1) specification seems to be plain wrong:
>
>      '1'
>           Stop trying to transmit data from this socket.  Discard any
>           data waiting to be sent.  Stop looking for acknowledgement of
>           data already sent; don't retransmit it if it is lost.
>
>
> Discarding data contradicts both the intended usage of the feature and
> the actual behavior (as observed by testing the state of the socket
> write queue using 'ioctl(fd, TIOCOUTQ, &val)' call before and after
> shutdown()).
>
> Given the above, could please somebody clarify what is the actual
> intended behavior of shutdown(..., SHUT_WR) on GLIBC/Linux nowadays?

Thanks for raising the issue.  I posted a patch to libc-alpha.

What you saw is the expected behavior for the mainline Linux kernel.
Since this is what distributions ship, I think it makes sense to
document it.

Thanks,
Florian

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Florian Weimer <fweimer@redhat.com> writes:

[...]

>> Given the above, could please somebody clarify what is the actual
>> intended behavior of shutdown(..., SHUT_WR) on GLIBC/Linux nowadays?
>
> Thanks for raising the issue.  I posted a patch to libc-alpha.
>
> What you saw is the expected behavior for the mainline Linux kernel.
> Since this is what distributions ship, I think it makes sense to
> document it.

Thanks! While we are at this part of the manual, I've found that:

"When you have finished using a socket, you can simply close its file
 descriptor with close; see Opening and Closing Files. If there is still
 data waiting to be transmitted over the connection, normally close tries
 to complete this transmission. You can control this behavior using the
 SO_LINGER socket option to specify a timeout period; see Socket
 Options."

has its own problem.

It looks like SO_LINGER is disabled by default in Linux, so "_normally_"
in "normally lose tries to complete this transmission" is probably
wrong.

As for the actual behavior, is close()/shutdown() on sockets is more
Linux kernel question than GLIBC one? I.e., is GLIBC only a thin wrapper
over calls to kernel and does nothing special by itself? If so, do you
have some reference where I could get information on behavior details
and/or check relevant discussions? Just ask in generic Linux kernel
mailing list?

I mean, what you wrote in the patch to GLIBC documentation is close to
my observations, yet Linux manual page on SO_LINGER seems to document
entirely different behavior, see below. That means you've likely got
your information elsewhere, from more recent/reliable source.

SOCKET(7)                 Linux Programmer's Manual                SOCKET(7)

[...]
       SO_LINGER
              Sets or gets the SO_LINGER option.  The argument is a linger
              structure.

                  struct linger {
                      int l_onoff;    /* linger active */
                      int l_linger;   /* how many seconds to linger for */
                  };

              When enabled, a close(2) or shutdown(2) will not return until
              all queued messages for the socket have been successfully sent
              or the linger timeout has been reached.  Otherwise, the call
              returns immediately and the closing is done in the background.
              When the socket is closed as part of exit(2), it always
              lingers in the background.

-- Sergey

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Florian Weimer]

* Sergey Organov:

> Florian Weimer <fweimer@redhat.com> writes:
>
> [...]
>
>>> Given the above, could please somebody clarify what is the actual
>>> intended behavior of shutdown(..., SHUT_WR) on GLIBC/Linux nowadays?
>>
>> Thanks for raising the issue.  I posted a patch to libc-alpha.
>>
>> What you saw is the expected behavior for the mainline Linux kernel.
>> Since this is what distributions ship, I think it makes sense to
>> document it.
>
> Thanks! While we are at this part of the manual, I've found that:
>
> "When you have finished using a socket, you can simply close its file
>  descriptor with close; see Opening and Closing Files. If there is still
>  data waiting to be transmitted over the connection, normally close tries
>  to complete this transmission. You can control this behavior using the
>  SO_LINGER socket option to specify a timeout period; see Socket
>  Options."
>
> has its own problem.
>
> It looks like SO_LINGER is disabled by default in Linux, so "_normally_"
> in "normally lose tries to complete this transmission" is probably
> wrong.

Good point.  Do you think that “normally close tries to deliver data to
the peer in the background” captures the Linux behavior more accurately?

> As for the actual behavior, is close()/shutdown() on sockets is more
> Linux kernel question than GLIBC one? I.e., is GLIBC only a thin wrapper
> over calls to kernel and does nothing special by itself? If so, do you
> have some reference where I could get information on behavior details
> and/or check relevant discussions? Just ask in generic Linux kernel
> mailing list?

It's only a very thin wrapper, and the transmission/blocking behavior in
particular is not changed at all.

> I mean, what you wrote in the patch to GLIBC documentation is close to
> my observations, yet Linux manual page on SO_LINGER seems to document
> entirely different behavior, see below. That means you've likely got
> your information elsewhere, from more recent/reliable source.

To be honest, I didn't bother looking this up. 8-/  I was pretty sure
that shutdown is not blocking/waiting, and a quick peek at the kernel
sources seems to confirm that.

Thanks,
Florian

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Florian Weimer <fweimer@redhat.com> writes:

> * Sergey Organov:
>
>> Florian Weimer <fweimer@redhat.com> writes:
>>
>> [...]
>>
>>>> Given the above, could please somebody clarify what is the actual
>>>> intended behavior of shutdown(..., SHUT_WR) on GLIBC/Linux nowadays?
>>>
>>> Thanks for raising the issue.  I posted a patch to libc-alpha.
>>>
>>> What you saw is the expected behavior for the mainline Linux kernel.
>>> Since this is what distributions ship, I think it makes sense to
>>> document it.
>>
>> Thanks! While we are at this part of the manual, I've found that:
>>
>> "When you have finished using a socket, you can simply close its file
>>  descriptor with close; see Opening and Closing Files. If there is still
>>  data waiting to be transmitted over the connection, normally close tries
>>  to complete this transmission. You can control this behavior using the
>>  SO_LINGER socket option to specify a timeout period; see Socket
>>  Options."
>>
>> has its own problem.
>>
>> It looks like SO_LINGER is disabled by default in Linux, so "_normally_"
>> in "normally lose tries to complete this transmission" is probably
>> wrong.
>
> Good point.  Do you think that “normally close tries to deliver data to
> the peer in the background” captures the Linux behavior more
> accurately?

I have no opinion, as I really encounter weird behavior on my system. To
me it looks like this part of GLIBC manual currently describes sane BSD
approach, while Linux simply gone wild, at least for me (I loose data on
program exit no matter how I do closing, and shutdown(...,SHUT_WR) only
makes it worse, even if I put long pause after it before exit()).

>
>> As for the actual behavior, is close()/shutdown() on sockets is more
>> Linux kernel question than GLIBC one? I.e., is GLIBC only a thin wrapper
>> over calls to kernel and does nothing special by itself? If so, do you
>> have some reference where I could get information on behavior details
>> and/or check relevant discussions? Just ask in generic Linux kernel
>> mailing list?
>
> It's only a very thin wrapper, and the transmission/blocking behavior in
> particular is not changed at all.

Shouldn't GLIBC manual then rather have some very generic description,
and then tell the details are system-specific?

I now even think that your original patch to the manual could be too
much, as neither Linux manual page (shame), nor POSIX (somewhat
expected) specify any details about the topic:

"
SHUTDOWN(2)               Linux Programmer's Manual              SHUTDOWN(2)

[...]
      
      The shutdown() call causes all or part of a full-duplex connection on
      the socket associated with sockfd to be shut down.  If how is
      SHUT_RD, further receptions will be disallowed.  If how is SHUT_WR,
      further transmissions will be disallowed.  If how is SHUT_RDWR,
      further receptions and transmissions will be disallowed.
"


-- Sergey

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Florian Weimer]

* Sergey Organov:

> Florian Weimer <fweimer@redhat.com> writes:
>
>> * Sergey Organov:
>>
>>> Florian Weimer <fweimer@redhat.com> writes:
>>>
>>> [...]
>>>
>>>>> Given the above, could please somebody clarify what is the actual
>>>>> intended behavior of shutdown(..., SHUT_WR) on GLIBC/Linux nowadays?
>>>>
>>>> Thanks for raising the issue.  I posted a patch to libc-alpha.
>>>>
>>>> What you saw is the expected behavior for the mainline Linux kernel.
>>>> Since this is what distributions ship, I think it makes sense to
>>>> document it.
>>>
>>> Thanks! While we are at this part of the manual, I've found that:
>>>
>>> "When you have finished using a socket, you can simply close its file
>>>  descriptor with close; see Opening and Closing Files. If there is still
>>>  data waiting to be transmitted over the connection, normally close tries
>>>  to complete this transmission. You can control this behavior using the
>>>  SO_LINGER socket option to specify a timeout period; see Socket
>>>  Options."
>>>
>>> has its own problem.
>>>
>>> It looks like SO_LINGER is disabled by default in Linux, so "_normally_"
>>> in "normally lose tries to complete this transmission" is probably
>>> wrong.
>>
>> Good point.  Do you think that “normally close tries to deliver data to
>> the peer in the background” captures the Linux behavior more
>> accurately?
>
> I have no opinion, as I really encounter weird behavior on my system. To
> me it looks like this part of GLIBC manual currently describes sane BSD
> approach, while Linux simply gone wild, at least for me (I loose data on
> program exit no matter how I do closing, and shutdown(...,SHUT_WR) only
> makes it worse, even if I put long pause after it before exit()).

Maybe it's related to the full-duplex problem I tried to describe?

>>> As for the actual behavior, is close()/shutdown() on sockets is more
>>> Linux kernel question than GLIBC one? I.e., is GLIBC only a thin wrapper
>>> over calls to kernel and does nothing special by itself? If so, do you
>>> have some reference where I could get information on behavior details
>>> and/or check relevant discussions? Just ask in generic Linux kernel
>>> mailing list?
>>
>> It's only a very thin wrapper, and the transmission/blocking behavior in
>> particular is not changed at all.
>
> Shouldn't GLIBC manual then rather have some very generic description,
> and then tell the details are system-specific?

I think it's not wrong to focus on Linux and TCP/IP these days.

> I now even think that your original patch to the manual could be too
> much, as neither Linux manual page (shame), nor POSIX (somewhat
> expected) specify any details about the topic:
>
> "
> SHUTDOWN(2)               Linux Programmer's Manual              SHUTDOWN(2)
>
> [...]
>       
>       The shutdown() call causes all or part of a full-duplex connection on
>       the socket associated with sockfd to be shut down.  If how is
>       SHUT_RD, further receptions will be disallowed.  If how is SHUT_WR,
>       further transmissions will be disallowed.  If how is SHUT_RDWR,
>       further receptions and transmissions will be disallowed.
> "

Well, we could improve the (non-POSIX) manual pages, too.

Thanks,
Florian

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Florian Weimer <fweimer@redhat.com> writes:

> * Sergey Organov:
>
>> Florian Weimer <fweimer@redhat.com> writes:
>>
>>> * Sergey Organov:
>>>
>>>> Florian Weimer <fweimer@redhat.com> writes:
>>>>
>>>> [...]
>>>>
>>>>>> Given the above, could please somebody clarify what is the actual
>>>>>> intended behavior of shutdown(..., SHUT_WR) on GLIBC/Linux nowadays?
>>>>>
>>>>> Thanks for raising the issue.  I posted a patch to libc-alpha.
>>>>>
>>>>> What you saw is the expected behavior for the mainline Linux kernel.
>>>>> Since this is what distributions ship, I think it makes sense to
>>>>> document it.
>>>>
>>>> Thanks! While we are at this part of the manual, I've found that:
>>>>
>>>> "When you have finished using a socket, you can simply close its file
>>>>  descriptor with close; see Opening and Closing Files. If there is still
>>>>  data waiting to be transmitted over the connection, normally close tries
>>>>  to complete this transmission. You can control this behavior using the
>>>>  SO_LINGER socket option to specify a timeout period; see Socket
>>>>  Options."
>>>>
>>>> has its own problem.
>>>>
>>>> It looks like SO_LINGER is disabled by default in Linux, so "_normally_"
>>>> in "normally lose tries to complete this transmission" is probably
>>>> wrong.
>>>
>>> Good point.  Do you think that “normally close tries to deliver data to
>>> the peer in the background” captures the Linux behavior more
>>> accurately?
>>
>> I have no opinion, as I really encounter weird behavior on my system. To
>> me it looks like this part of GLIBC manual currently describes sane BSD
>> approach, while Linux simply gone wild, at least for me (I loose data on
>> program exit no matter how I do closing, and shutdown(...,SHUT_WR) only
>> makes it worse, even if I put long pause after it before exit()).
>
> Maybe it's related to the full-duplex problem I tried to describe?
>

Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
anywhere before 'close(fd)' can do things worse from the POV of data
delivery to the other end.

What I observe is that either:

sleep(1);
close(fd);
exit(0);

or:

sleep(1);
shutdown(fd, SHUT_WR);
close(fd);
exit(0);

deliver all the data, while:

shutdown(fd, SHUT_WR);
sleep(1);
close(fd);
exit(0);

cuts some of the data (read() returns 0 on the other end indicating
closed socket).

Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
above cases when put at the beginning of the above sequences[*],
indicating that there are no pended data, so there should be nothing to
loose in the first place, one way or another.

[*] TIOCOUTQ does give 1 right after 'shutdown(fd, SHUT_WR)', and
matching amount of data if I do write() something right before calling
it, so it does work as expected.

-- Sergey

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Florian Weimer]

* Sergey Organov:

> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
> anywhere before 'close(fd)' can do things worse from the POV of data
> delivery to the other end.
>
> What I observe is that either:
>
> sleep(1);
> close(fd);
> exit(0);
>
> or:
>
> sleep(1);
> shutdown(fd, SHUT_WR);
> close(fd);
> exit(0);
>
> deliver all the data, while:
>
> shutdown(fd, SHUT_WR);
> sleep(1);
> close(fd);
> exit(0);
>
> cuts some of the data (read() returns 0 on the other end indicating
> closed socket).

Wait, you get less data with the last sequence?  And you do not write to
fd after the shutdown call?

That should definitely not happen.

> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
> above cases when put at the beginning of the above sequences[*],
> indicating that there are no pended data, so there should be nothing to
> loose in the first place, one way or another.

Yes, I agree that data loss should not happen in these cases.  There has
to be something else that triggers an RST segment, or perhaps there is a
middlebox with a broken TCP implementation.

Thanks,
Florian

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Florian Weimer <fweimer@redhat.com> writes:

> * Sergey Organov:
>
>> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
>> anywhere before 'close(fd)' can do things worse from the POV of data
>> delivery to the other end.
>>
>> What I observe is that either:
>>
>> sleep(1);
>> close(fd);
>> exit(0);
>>
>> or:
>>
>> sleep(1);
>> shutdown(fd, SHUT_WR);
>> close(fd);
>> exit(0);
>>
>> deliver all the data, while:
>>
>> shutdown(fd, SHUT_WR);
>> sleep(1);
>> close(fd);
>> exit(0);
>>
>> cuts some of the data (read() returns 0 on the other end indicating
>> closed socket).
>
> Wait, you get less data with the last sequence?  And you do not write to
> fd after the shutdown call?

Yes, and this is single-threaded application so I'm very sure I don't
write anything myself.

However, as I already mentioned, ioctl(fd, TIOCOUTQ, &v) gives 0 before
shutdown() and 1 right  after shutdown(), that suggests shutdown()
itself writes something. Dunno if it's expected.

>
> That should definitely not happen.

I also think it should not, and I overcame it at upper-level of my
application for now, as I didn't have time to get down to kernel
internals to figure what actually happens, nor do I have enough TCP
stack knowledge to figure these things easily.

>
>> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
>> above cases when put at the beginning of the above sequences[*],
>> indicating that there are no pended data, so there should be nothing to
>> loose in the first place, one way or another.
>
> Yes, I agree that data loss should not happen in these cases.

It looks like shutdown() does something that is transferred to receiver
and makes it drop some amount of recently received data that receiving
application didn't read yet. I.e., it's probably receiving end that
drops the data?

> There has to be something else that triggers an RST segment, or
> perhaps there is a middlebox with a broken TCP implementation.

As far as I recall, I did try direct cable connection from dedicated
Ethernet controller on my PC running the application under:

Linux 4.9.0-0.bpo.6-686 #1 SMP Debian 4.9.88-1+deb9u1~bpo8+1 (2018-05-13) i686 GNU/Linux

to embedded ARM/Linux board receiving the data, running Linux 4.9.x,
4.9.145 being most likely version.

If it looks like an unknown bug, I think I can resurrect the tests and
investigate further.

-- Sergey

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Florian Weimer]

* Sergey Organov:

> Florian Weimer <fweimer@redhat.com> writes:
>
>> * Sergey Organov:
>>
>>> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
>>> anywhere before 'close(fd)' can do things worse from the POV of data
>>> delivery to the other end.
>>>
>>> What I observe is that either:
>>>
>>> sleep(1);
>>> close(fd);
>>> exit(0);
>>>
>>> or:
>>>
>>> sleep(1);
>>> shutdown(fd, SHUT_WR);
>>> close(fd);
>>> exit(0);
>>>
>>> deliver all the data, while:
>>>
>>> shutdown(fd, SHUT_WR);
>>> sleep(1);
>>> close(fd);
>>> exit(0);
>>>
>>> cuts some of the data (read() returns 0 on the other end indicating
>>> closed socket).
>>
>> Wait, you get less data with the last sequence?  And you do not write to
>> fd after the shutdown call?
>
> Yes, and this is single-threaded application so I'm very sure I don't
> write anything myself.
>
> However, as I already mentioned, ioctl(fd, TIOCOUTQ, &v) gives 0 before
> shutdown() and 1 right  after shutdown(), that suggests shutdown()
> itself writes something. Dunno if it's expected.

It needs to queue the FIN segment, for the connection teardown
handshake.

>>> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
>>> above cases when put at the beginning of the above sequences[*],
>>> indicating that there are no pended data, so there should be nothing to
>>> loose in the first place, one way or another.
>>
>> Yes, I agree that data loss should not happen in these cases.
>
> It looks like shutdown() does something that is transferred to receiver
> and makes it drop some amount of recently received data that receiving
> application didn't read yet. I.e., it's probably receiving end that
> drops the data?

Do you see any RST segments in packet captures?  As I said, RST is not
expected to be ordered with respect to data delivery, so if an RST
segment is generated, then it would explain the data loss.

Thanks,
Florian

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Florian Weimer <fweimer@redhat.com> writes:

> * Sergey Organov:
>
>> Florian Weimer <fweimer@redhat.com> writes:
>>
>>> * Sergey Organov:
>>>
>>>> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
>>>> anywhere before 'close(fd)' can do things worse from the POV of data
>>>> delivery to the other end.
>>>>
>>>> What I observe is that either:
>>>>
>>>> sleep(1);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> or:
>>>>
>>>> sleep(1);
>>>> shutdown(fd, SHUT_WR);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> deliver all the data, while:
>>>>
>>>> shutdown(fd, SHUT_WR);
>>>> sleep(1);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> cuts some of the data (read() returns 0 on the other end indicating
>>>> closed socket).
>>>
>>> Wait, you get less data with the last sequence?  And you do not write to
>>> fd after the shutdown call?
>>
>> Yes, and this is single-threaded application so I'm very sure I don't
>> write anything myself.
>>
>> However, as I already mentioned, ioctl(fd, TIOCOUTQ, &v) gives 0 before
>> shutdown() and 1 right  after shutdown(), that suggests shutdown()
>> itself writes something. Dunno if it's expected.
>
> It needs to queue the FIN segment, for the connection teardown
> handshake.
>
>>>> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
>>>> above cases when put at the beginning of the above sequences[*],
>>>> indicating that there are no pended data, so there should be nothing to
>>>> loose in the first place, one way or another.
>>>
>>> Yes, I agree that data loss should not happen in these cases.
>>
>> It looks like shutdown() does something that is transferred to receiver
>> and makes it drop some amount of recently received data that receiving
>> application didn't read yet. I.e., it's probably receiving end that
>> drops the data?
>
> Do you see any RST segments in packet captures?  As I said, RST is not
> expected to be ordered with respect to data delivery, so if an RST
> segment is generated, then it would explain the data loss.

I didn't even try to capture/analyze packets at the time of debugging,
as I didn't know what to expect and had no time to learn.

I'll try to get time to repeat the tests and do some tcpdumps, paying
special attention to RSTs.

Thanks,

-- Sergey

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Florian Weimer <fweimer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:

> * Sergey Organov:
>
>> Florian Weimer <fweimer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:
>>
>>> * Sergey Organov:
>>>
>>>> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
>>>> anywhere before 'close(fd)' can do things worse from the POV of data
>>>> delivery to the other end.
>>>>
>>>> What I observe is that either:
>>>>
>>>> sleep(1);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> or:
>>>>
>>>> sleep(1);
>>>> shutdown(fd, SHUT_WR);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> deliver all the data, while:
>>>>
>>>> shutdown(fd, SHUT_WR);
>>>> sleep(1);
>>>> close(fd);
>>>> exit(0);
>>>>
>>>> cuts some of the data (read() returns 0 on the other end indicating
>>>> closed socket).
>>>
>>> Wait, you get less data with the last sequence?  And you do not write to
>>> fd after the shutdown call?
>>
>> Yes, and this is single-threaded application so I'm very sure I don't
>> write anything myself.
>>
>> However, as I already mentioned, ioctl(fd, TIOCOUTQ, &v) gives 0 before
>> shutdown() and 1 right  after shutdown(), that suggests shutdown()
>> itself writes something. Dunno if it's expected.
>
> It needs to queue the FIN segment, for the connection teardown
> handshake.
>
>>>> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
>>>> above cases when put at the beginning of the above sequences[*],
>>>> indicating that there are no pended data, so there should be nothing to
>>>> loose in the first place, one way or another.
>>>
>>> Yes, I agree that data loss should not happen in these cases.
>>
>> It looks like shutdown() does something that is transferred to receiver
>> and makes it drop some amount of recently received data that receiving
>> application didn't read yet. I.e., it's probably receiving end that
>> drops the data?
>
> Do you see any RST segments in packet captures?  As I said, RST is not
> expected to be ordered with respect to data delivery, so if an RST
> segment is generated, then it would explain the data loss.

I've got time to get back to the issue, and what I see is the following.

When shutdown(fd, SHUT_WR) is called without delay on the sending end, the sending
terminates early like this:

[...]
08:14:25.741054 IP (tos 0x0, ttl 64, id 23482, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e17 (correct), seq 540:541, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741331 IP (tos 0x0, ttl 64, id 23483, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e16 (correct), seq 541:542, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741728 IP (tos 0x0, ttl 64, id 23484, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e15 (correct), seq 542:543, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.743013 IP (tos 0x0, ttl 64, id 23485, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [F.], cksum 0x641b (correct), seq 543, ack 16504, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0
08:14:25.743644 IP (tos 0x0, ttl 64, id 23486, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [.], cksum 0x6414 (correct), seq 544, ack 16511, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0

whereas in the "good" case, where shutdown(fd, SHUT_WR) is called after
1-second delay, sending terminates much later, like this:

[...]
08:04:50.446074 IP (tos 0x0, ttl 64, id 34857, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc795 (correct), seq 697:698, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446339 IP (tos 0x0, ttl 64, id 34858, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc794 (correct), seq 698:699, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446738 IP (tos 0x0, ttl 64, id 34859, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc793 (correct), seq 699:700, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.459717 IP (tos 0x0, ttl 64, id 34860, offset 0, flags [DF], proto TCP (6), length 64)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [.], cksum 0x0fed (correct), ack 16510, win 1385, options [nop,nop,TS val 188923386 ecr 1249313311,nop,nop,sack 1 {16509:16510}], length 0
08:04:51.384289 IP (tos 0x0, ttl 64, id 34861, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [F.], cksum 0xcd35 (correct), seq 700, ack 16510, win 1385, options [nop,nop,TS val 188923479 ecr 1249313311], length 0

To me it looks like it's in fact the sending end of the connection that
drops a lot of yet unsent packets (packets 543--700) when shutdown() is
called.

What is also surprising is that TIOCOUTQ requests put like this:

  int v1 = -1, v2 = -1;
  ioctl(fd, TIOCOUTQ, &v1);
  shutdown(fd, SHUT_WR);
  ioctl(fd, TIOCOUTQ, &v2);
  DPRINT("$$OUT %d %d\n", v1, v2);
  sleep(1);
  close(fd);
  exit(0);

result in

$$OUT 3 2

debug output in the "bad" case that doesn't match the number of dropped
data at all.

Any thoughts?

Thanks,
-- Sergey

Re: Help: shutdown(..., SHUT_WR) on TCP sockets

[Sergey Organov]

Hello Florian,

I've sent this to the mailing list only the first time, sorry, --
resend.

Sergey Organov <sorganov@gmail.com> writes:

> Florian Weimer <fweimer@redhat.com> writes:
>
>> * Sergey Organov:
>>
>>> Florian Weimer <fweimer@redhat.com> writes:
>>>
>>>> * Sergey Organov:
>>>>
>>>>> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
>>>>> anywhere before 'close(fd)' can do things worse from the POV of data
>>>>> delivery to the other end.
>>>>>
>>>>> What I observe is that either:
>>>>>
>>>>> sleep(1);
>>>>> close(fd);
>>>>> exit(0);
>>>>>
>>>>> or:
>>>>>
>>>>> sleep(1);
>>>>> shutdown(fd, SHUT_WR);
>>>>> close(fd);
>>>>> exit(0);
>>>>>
>>>>> deliver all the data, while:
>>>>>
>>>>> shutdown(fd, SHUT_WR);
>>>>> sleep(1);
>>>>> close(fd);
>>>>> exit(0);
>>>>>
>>>>> cuts some of the data (read() returns 0 on the other end indicating
>>>>> closed socket).
>>>>
>>>> Wait, you get less data with the last sequence?  And you do not write to
>>>> fd after the shutdown call?
>>>
>>> Yes, and this is single-threaded application so I'm very sure I don't
>>> write anything myself.
>>>
>>> However, as I already mentioned, ioctl(fd, TIOCOUTQ, &v) gives 0 before
>>> shutdown() and 1 right  after shutdown(), that suggests shutdown()
>>> itself writes something. Dunno if it's expected.
>>
>> It needs to queue the FIN segment, for the connection teardown
>> handshake.
>>
>>>>> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
>>>>> above cases when put at the beginning of the above sequences[*],
>>>>> indicating that there are no pended data, so there should be nothing to
>>>>> loose in the first place, one way or another.
>>>>
>>>> Yes, I agree that data loss should not happen in these cases.
>>>
>>> It looks like shutdown() does something that is transferred to receiver
>>> and makes it drop some amount of recently received data that receiving
>>> application didn't read yet. I.e., it's probably receiving end that
>>> drops the data?
>>
>> Do you see any RST segments in packet captures?  As I said, RST is not
>> expected to be ordered with respect to data delivery, so if an RST
>> segment is generated, then it would explain the data loss.

I've got time to get back to the issue, and what I see is the following.

When shutdown(fd, SHUT_WR) is called without delay on the sending end, the sending
terminates early like this:

[...]
08:14:25.741054 IP (tos 0x0, ttl 64, id 23482, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e17 (correct), seq 540:541, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741331 IP (tos 0x0, ttl 64, id 23483, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e16 (correct), seq 541:542, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741728 IP (tos 0x0, ttl 64, id 23484, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e15 (correct), seq 542:543, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.743013 IP (tos 0x0, ttl 64, id 23485, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [F.], cksum 0x641b (correct), seq 543, ack 16504, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0
08:14:25.743644 IP (tos 0x0, ttl 64, id 23486, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43076: Flags [.], cksum 0x6414 (correct), seq 544, ack 16511, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0

whereas in the "good" case, where shutdown(fd, SHUT_WR) is called after
1-second delay, sending terminates much later, like this:

[...]
08:04:50.446074 IP (tos 0x0, ttl 64, id 34857, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc795 (correct), seq 697:698, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446339 IP (tos 0x0, ttl 64, id 34858, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc794 (correct), seq 698:699, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446738 IP (tos 0x0, ttl 64, id 34859, offset 0, flags [DF], proto TCP (6), length 53)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc793 (correct), seq 699:700, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.459717 IP (tos 0x0, ttl 64, id 34860, offset 0, flags [DF], proto TCP (6), length 64)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [.], cksum 0x0fed (correct), ack 16510, win 1385, options [nop,nop,TS val 188923386 ecr 1249313311,nop,nop,sack 1 {16509:16510}], length 0
08:04:51.384289 IP (tos 0x0, ttl 64, id 34861, offset 0, flags [DF], proto TCP (6), length 52)
    omega-51da5d.gnss.ru.8002 > osv.43048: Flags [F.], cksum 0xcd35 (correct), seq 700, ack 16510, win 1385, options [nop,nop,TS val 188923479 ecr 1249313311], length 0

To me it looks like it's in fact the sending end of the connection that
drops a lot of yet unsent packets (packets 543--700) when shutdown() is
called.

What is also surprising is that TIOCOUTQ requests put like this:

  int v1 = -1, v2 = -1;
  ioctl(fd, TIOCOUTQ, &v1);
  shutdown(fd, SHUT_WR);
  ioctl(fd, TIOCOUTQ, &v2);
  printf("$$OUT %d %d\n", v1, v2);
  sleep(1);
  close(fd);
  exit(0);

result in

$$OUT 3 2

debug output in the "bad" case (that doesn't match the number of dropped
data at all).

Any thoughts? Does it look like a bug to be reported?

Thanks,
-- Sergey