Re: cURL uploads 0 length file for sftp transfer.

[Cary Lewis <cary.lewis@gmail.com>]

Curl under cygwin - TCP keepalive

I need to download a file from a remote server via a https request, and the
remote server takes several minutes to prepare the file.

Without any keepalives, the remote end will terminate the socket, but
it doesn't close it (which I know is weird).

Running curl with --keepalive 45 on a linux centos7 machine, enables the
transfer to work. With tcpdump, I see the nop packet every 45 seconds.

But on cygwin, the --keepalive parameter doesn't seem to have effect.
Running wireshark, I don't see any keepalives.

Running the request in a browser, chrome or firefox, works fine, and I can
see keepalives in wireshark, so clearly they are either setting the socket
options correctly, or they are sending their own nop.

Running on a vm, vmware workstation /  centos8 , on the same machine where
I am running curl, I can see the keepalives.

I have downloaded and rebuilt curl and that did not resolve the issue.



 Can someone shed some light on this?

On Sat, Sep 7, 2019 at 9:39 PM ALbert via cygwin <cygwin@cygwin.com> wrote:

> I upgraded to the latest version curl 7.65.3 with libcurl 7.65.3, same
> issue.
> curl 7.65.3 (i686-pc-cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11
> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
> Thanks.
>
>     On Friday, September 6, 2019, 3:44:13 PM EDT, ALbert <
> albert_76@yahoo.com> wrote:
>
>  We recently upgraded cURL from version:curl 7.50.3 (i686-pc-cygwin)
> libcurl/7.50.3 OpenSSL/1.0.2j zlib/1.2.8 libidn/1.29 libpsl/0.14.0
> (+libidn/1.29) libssh2/1.7.0 nghttp2/1.14.0tocurl 7.64.1 (i686-pc-cygwin)
> libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0
> libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0
> Since the upgrade, we've seen problems uploading text files to sftp
> servers. Here's the call that we make:
> curl--trace-ascii trace.txt -k sftp://<user>:<pass>@<server>/file.txt-T
> file.txt
>
> On initial call, a 0-length file is uploaded, with successful return code.
> On immediate rerun, a full file is uploaded (if we wait few minutes,
> 0-length file will get uploaded). Below are the traces from initial and
> subsequent calls. We also tried using same version curl on a linux box,
> without any issues. We also downloaded latest version of curl from curl for
> Windows, "curl 7.65.3 (i386-pc-win32) libcurl/7.65.3 OpenSSL/1.1.1c
> (Schannel) zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.39.2",
> and its also working correctly.
> Unfortunately, the server we're uploading to belongs to a client, and we
> can't see if this is an issue on a server side. However, even if it was,
> why would linux curl and windows curl (from curl.haxx.se) would work
> correctly?
> Any help or pointers would be appreciated. Thank you.
>
> Here's the trace files:
> From initial call:== Info: STATE: INIT => CONNECT handle 0x80068340; line
> 1356 (connection #-5000)
> == Info: Added connection 0. The cache now contains 1 members
> == Info: STATE: CONNECT => WAITRESOLVE handle 0x80068340; line 1397
> (connection #0)
> == Info:   Trying XXX.XX.XX.XX...== Info: TCP_NODELAY set
> == Info: STATE: WAITRESOLVE => WAITCONNECT handle 0x80068340; line 1476
> (connection #0)
> == Info: Connected to XXXXXXX  ( XXX.XX.XX.XX) port 22 (#0)== Info: STATE:
> WAITCONNECT => SENDPROTOCONNECT handle 0x80068340; line 1532 (connection #0)
> == Info: Marked for [keep alive]: SSH default
> == Info: User: Opus_efg
> == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_INIT (line 2115)
> == Info: SSH 0x80048f90 state change from SSH_INIT to SSH_S_STARTUP (line
> 582)
> == Info: STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x80068340; line
> 1547 (connection #0)
> == Info: SSH 0x80048f90 state change from SSH_S_STARTUP to SSH_HOSTKEY
> (line 595)
> == Info: SSH 0x80048f90 state change from SSH_HOSTKEY to SSH_AUTHLIST
> (line 605)
> == Info: SSH 0x80048f90 state change from SSH_AUTHLIST to
> SSH_AUTH_PKEY_INIT (line 628)
> == Info: Authentication using SSH public key file
> == Info: SSH 0x80048f90 state change from SSH_AUTH_PKEY_INIT to
> SSH_AUTH_KEY_INIT (line 697)
> == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY_INIT to
> SSH_AUTH_KEY (line 743)
> == Info: completed keyboard interactive authentication
> == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY to SSH_AUTH_DONE
> (line 761)
> == Info: Authentication complete
> == Info: SSH 0x80048f90 state change from SSH_AUTH_DONE to SSH_SFTP_INIT
> (line 807)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_INIT to
> SSH_SFTP_REALPATH (line 833)
> == Info: SSH CONNECT phase done
> == Info: SSH 0x80048f90 state change from SSH_SFTP_REALPATH to SSH_STOP
> (line 850)
> == Info: STATE: PROTOCONNECT => DO handle 0x80068340; line 1566
> (connection #0)
> == Info: DO phase starts
> == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_QUOTE_INIT
> (line 2332)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_QUOTE_INIT to
> SSH_SFTP_GETINFO (line 868)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_GETINFO to
> SSH_SFTP_TRANS_INIT (line 1060)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_TRANS_INIT to
> SSH_SFTP_UPLOAD_INIT (line 1080)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_UPLOAD_INIT to
> SSH_SFTP_CLOSE (line 1143)
> == Info: STATE: DO => DOING handle 0x80068340; line 1608 (connection #0)
> == Info: SFTP DONE done
> == Info: SSH 0x80048f90 state change from SSH_SFTP_CLOSE to SSH_STOP (line
> 1651)
> == Info: DO phase is complete
> == Info: STATE: DOING => DO_DONE handle 0x80068340; line 1693 (connection
> #0)
> == Info: STATE: DO_DONE => PERFORM handle 0x80068340; line 1743
> (connection #0)
> == Info: STATE: PERFORM => DONE handle 0x80068340; line 1933 (connection
> #0)
> == Info: multi_done
> == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_CLOSE (line
> 2390)
> == Info: SFTP DONE done
> == Info: SSH 0x80048f90 state change from SSH_SFTP_CLOSE to SSH_STOP (line
> 1651)
> == Info: Connection #0 to host XXXXXXX left intact
>
>
> Trace from subsequent call:== Info: STATE: INIT => CONNECT handle
> 0x80068340; line 1356 (connection #-5000)
> == Info: Added connection 0. The cache now contains 1 members
> == Info: STATE: CONNECT => WAITRESOLVE handle 0x80068340; line 1397
> (connection #0)
> == Info:   Trying XXX.XX.XX.XX...== Info: TCP_NODELAY set
> == Info: STATE: WAITRESOLVE => WAITCONNECT handle 0x80068340; line 1476
> (connection #0)
> == Info: Connected to XXXXXXX (XXX.XX.XX.XX) port 22 (#0)
> == Info: STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x80068340; line
> 1532 (connection #0)
> == Info: Marked for [keep alive]: SSH default
> == Info: User: Opus_efg
> == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_INIT (line 2115)
> == Info: SSH 0x80048f90 state change from SSH_INIT to SSH_S_STARTUP (line
> 582)
> == Info: STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x80068340; line
> 1547 (connection #0)
> == Info: SSH 0x80048f90 state change from SSH_S_STARTUP to SSH_HOSTKEY
> (line 595)
> == Info: SSH 0x80048f90 state change from SSH_HOSTKEY to SSH_AUTHLIST
> (line 605)
> == Info: SSH 0x80048f90 state change from SSH_AUTHLIST to
> SSH_AUTH_PKEY_INIT (line 628)
> == Info: Authentication using SSH public key file
> == Info: SSH 0x80048f90 state change from SSH_AUTH_PKEY_INIT to
> SSH_AUTH_KEY_INIT (line 697)
> == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY_INIT to
> SSH_AUTH_KEY (line 743)
> == Info: completed keyboard interactive authentication
> == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY to SSH_AUTH_DONE
> (line 761)
> == Info: Authentication complete
> == Info: SSH 0x80048f90 state change from SSH_AUTH_DONE to SSH_SFTP_INIT
> (line 807)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_INIT to
> SSH_SFTP_REALPATH (line 833)
> == Info: SSH CONNECT phase done
> == Info: SSH 0x80048f90 state change from SSH_SFTP_REALPATH to SSH_STOP
> (line 850)
> == Info: STATE: PROTOCONNECT => DO handle 0x80068340; line 1566
> (connection #0)
> == Info: DO phase starts
> == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_QUOTE_INIT
> (line 2332)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_QUOTE_INIT to
> SSH_SFTP_GETINFO (line 868)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_GETINFO to
> SSH_SFTP_TRANS_INIT (line 1060)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_TRANS_INIT to
> SSH_SFTP_UPLOAD_INIT (line 1080)
> == Info: SSH 0x80048f90 state change from SSH_SFTP_UPLOAD_INIT to SSH_STOP
> (line 1222)
> == Info: DO phase is complete
> == Info: STATE: DO => DO_DONE handle 0x80068340; line 1621 (connection #0)
> == Info: STATE: DO_DONE => PERFORM handle 0x80068340; line 1743
> (connection #0)
> => Send data, 10688 bytes (0x29c0)
>   SENSITIVE DATA DELETED
> == Info: We are completely uploaded and fine
> == Info: STATE: PERFORM => DONE handle 0x80068340; line 1933 (connection
> #0)
> == Info: multi_done
> == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_CLOSE (line
> 2390)
> == Info: SFTP DONE done
> == Info: SSH 0x80048f90 state change from SSH_SFTP_CLOSE to SSH_STOP (line
> 1651)
> == Info: Connection #0 to host XXXXXXX left intact
> == Info: Expire cleared (transfer 0x80068340)
>
>
>
>
>
>
>
>
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>
>