Building cygwin1.dll

Building cygwin1.dll

[Johan van den Berg]

Any pointers on how to compile cygwin1.dll? I installed cygwin latest onto a windows machine, installed all the prerequisites as per faq (gcc perl et al), downloaded a snapshot, configured and maked but keep running into compile errors.

There is no mention in the docs regarding version of gcc required, possibility of using gcc as supplied by cygwin or mingw, etc.
I am trying to build a custom cygwin1.dll to test http://cygwin.com/ml/cygwin/2011-12/msg00337.html
I have set the send and receive buffers to 256k so that I can get transfer speeds up on a high latency rsync copy from one cygwin windows to another cygwin windows box with over 300ms latency between the two, but a 100mbit bandwidth
I tested with a previous version (very old) of cygwin that had the buffers at 8k and that caused the transfer speeds to go down to next to nothing, so the theory seems sound that increasing the buffers should increase the amount of packets that can be sent without receiving an ack.
I have already patched the code, now I just need to figure out how to actually build cygwin, and 3 hours worth of google + reading of cygwin mailing lists have been fruitless.

The error I get when trying to compile using the default cygwin install of gcc and tools:

... lots of successful compiles, and then:
make[5]: Entering directory `/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/libc/locale'
gcc -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup/cygwin -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup/w32api/lib -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/cygwin/include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/w32api/include -B/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/ -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/targ-include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/newlib/libc/include    -I/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/cygwin/include -DPACKAGE_NAME=\"newlib\" -DPACKAGE_TARNAME=\"newlib\" -DPACKAGE_VERSION=\"1.20.0\" -DPACKAGE_STRING=\"newlib\ 1.20.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I. -I../../../../../newlib/libc/locale -DHAVE_OPENDIR -DHAVE_RENAME -DSIGNAL_PROVIDED -D_COMPILING_NEWLIB -DHAVE_BLKSIZE -DHAVE_FCNTL -DMALLOC_PROVIDED -fno-builtin      -g -O2 -c -o lib_a-nl_langinfo.o `test -f 'nl_langinfo.c' || echo '../../../../../newlib/libc/locale/'`nl_langinfo.c
../../../../../newlib/libc/locale/nl_langinfo.c:66: error: parse error before "struct"
../../../../../newlib/libc/locale/nl_langinfo.c:66: error: `outdigits' undeclared here (not in a function)
../../../../../newlib/libc/locale/nl_langinfo.c:66: error: initializer element is not constant
../../../../../newlib/libc/locale/nl_langinfo.c:66: error: (near initialization for `nl_ext[0].offset')
../../../../../newlib/libc/locale/nl_langinfo.c:66: error: parse error before ')' token
../../../../../newlib/libc/locale/nl_langinfo.c:66: error: initializer element is not constant
../../../../../newlib/libc/locale/nl_langinfo.c:66: error: (near initialization for `nl_ext[0]')
.... and many more similar complaints.

Johan van den Berg
Play Safe IT Services South Africa (Pty) Ltd
Email: johan.vdberg@playsafesa.com
Skype: johantheitguy




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Building cygwin1.dll

[marco atzeri]

On 1/4/2012 2:49 PM, Johan van den Berg wrote:
> Any pointers on how to compile cygwin1.dll? I installed cygwin latest onto a windows machine, installed all the prerequisites as per faq (gcc perl et al), downloaded a snapshot, configured and maked but keep running into compile errors.
>
> There is no mention in the docs regarding version of gcc required, possibility of using gcc as supplied by cygwin or mingw, etc.
> I am trying to build a custom cygwin1.dll to test http://cygwin.com/ml/cygwin/2011-12/msg00337.html
> I have set the send and receive buffers to 256k so that I can get transfer speeds up on a high latency rsync copy from one cygwin windows to another cygwin windows box with over 300ms latency between the two, but a 100mbit bandwidth
> I tested with a previous version (very old) of cygwin that had the buffers at 8k and that caused the transfer speeds to go down to next to nothing, so the theory seems sound that increasing the buffers should increase the amount of packets that can be sent without receiving an ack.
> I have already patched the code, now I just need to figure out how to actually build cygwin, and 3 hours worth of google + reading of cygwin mailing lists have been fruitless.
>
> The error I get when trying to compile using the default cygwin install of gcc and tools:
>
> ... lots of successful compiles, and then:
> make[5]: Entering directory `/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/libc/locale'
> gcc -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup/cygwin -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup/w32api/lib -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/cygwin/include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/w32api/include -B/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/ -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/targ-include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/newlib/libc/include    -I/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/cygwin/include -DPACKAGE_NAME=\"newlib\" -DPACKAGE_TARNAME=\"newlib\" -DPACKAGE_VERSION=\"1.20.0\" -DPACKAGE_STRING=\"newlib\ 1.20.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKA
GE_URL=\"\" -I. -I../../../../../newlib/libc/locale -DHAVE_OPENDIR -DHAVE_RENAME -DSIGNAL_PROVIDED -D_COMPILING_NEWLIB -DHAVE_BLKSIZE -DHAVE_FCNTL -DMALLOC_PROVIDED -fno-builtin      -g -O2 -c -o lib_a-nl_langinfo.o `test -f 'nl_langinfo.c' || echo '../../../../../newlib/libc/locale/'`nl_langinfo.c
> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: parse error before "struct"
> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: `outdigits' undeclared here (not in a function)
> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: initializer element is not constant
> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: (near initialization for `nl_ext[0].offset')
> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: parse error before ')' token
> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: initializer element is not constant
> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: (near initialization for `nl_ext[0]')
> .... and many more similar complaints.
>
> Johan van den Berg

hi John,

gcc and g++ version 4.5.3 (GCC)
as provided by cygwin packages

gcc4-core                      4.5.3-3
gcc4-g++                       4.5.3-3

usually this is enough for building

  <source_path>/configure
  make

Regards
Marco



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Building cygwin1.dll

[Johan van den Berg]

I am very happy to report that increasing the send and receive buffers has done the job (at least, on a 10MBit link but will be testing a 100Mbit in a few days). I calculated the ideal size as per http://www.ibm.com/developerworks/linux/library/l-hisock/index.html

$ diff -u cygwin-snapshot-20111222-1/winsup/cygwin/net.cc.orig cygwin-snapshot-20111222-1/winsup/cygwin/net.cc
--- cygwin-snapshot-20111222-1/winsup/cygwin/net.cc.orig        2012-01-04 06:56:22.094250000 -0500
+++ cygwin-snapshot-20111222-1/winsup/cygwin/net.cc     2012-01-04 07:10:14.031750000 -0500
@@ -569,8 +569,12 @@
      be nice, though.

      (*) Maximum normal TCP window size.  Coincidence?  */
-  ((fhandler_socket *) fd)->rmem () = 65535;
-  ((fhandler_socket *) fd)->wmem () = 65535;
+
+  /* JVDB - 64K is too small for our 100mb link with 200ms latency. Bumped to 2MB
+     See http://www.ibm.com/developerworks/linux/library/l-hisock/index.html */
+
+  ((fhandler_socket *) fd)->rmem () = 2097152;
+  ((fhandler_socket *) fd)->wmem () = 2097152;
   if (::setsockopt (soc, SOL_SOCKET, SO_RCVBUF,
                    (char *) &((fhandler_socket *) fd)->rmem (), sizeof (int)))
     {

Thank you very much for the help!

Kind regards
Johan


On 04 Jan 2012, at 4:24 PM, marco atzeri wrote:

> On 1/4/2012 2:49 PM, Johan van den Berg wrote:
>> Any pointers on how to compile cygwin1.dll? I installed cygwin latest onto a windows machine, installed all the prerequisites as per faq (gcc perl et al), downloaded a snapshot, configured and maked but keep running into compile errors.
>> 
>> There is no mention in the docs regarding version of gcc required, possibility of using gcc as supplied by cygwin or mingw, etc.
>> I am trying to build a custom cygwin1.dll to test http://cygwin.com/ml/cygwin/2011-12/msg00337.html
>> I have set the send and receive buffers to 256k so that I can get transfer speeds up on a high latency rsync copy from one cygwin windows to another cygwin windows box with over 300ms latency between the two, but a 100mbit bandwidth
>> I tested with a previous version (very old) of cygwin that had the buffers at 8k and that caused the transfer speeds to go down to next to nothing, so the theory seems sound that increasing the buffers should increase the amount of packets that can be sent without receiving an ack.
>> I have already patched the code, now I just need to figure out how to actually build cygwin, and 3 hours worth of google + reading of cygwin mailing lists have been fruitless.
>> 
>> The error I get when trying to compile using the default cygwin install of gcc and tools:
>> 
>> ... lots of successful compiles, and then:
>> make[5]: Entering directory `/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/libc/locale'
>> gcc -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup/cygwin -L/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/winsup/w32api/lib -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/cygwin/include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/w32api/include -B/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/ -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/build/i686-pc-cygwin/newlib/targ-include -isystem /home/DBA/cygwin-build/cygwin-snapshot-20111222-1/newlib/libc/include    -I/home/DBA/cygwin-build/cygwin-snapshot-20111222-1/winsup/cygwin/include -DPACKAGE_NAME=\"newlib\" -DPACKAGE_TARNAME=\"newlib\" -DPACKAGE_VERSION=\"1.20.0\" -DPACKAGE_STRING=\"newlib\ 1.20.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKA
> GE_URL=\"\" -I. -I../../../../../newlib/libc/locale -DHAVE_OPENDIR -DHAVE_RENAME -DSIGNAL_PROVIDED -D_COMPILING_NEWLIB -DHAVE_BLKSIZE -DHAVE_FCNTL -DMALLOC_PROVIDED -fno-builtin      -g -O2 -c -o lib_a-nl_langinfo.o `test -f 'nl_langinfo.c' || echo '../../../../../newlib/libc/locale/'`nl_langinfo.c
>> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: parse error before "struct"
>> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: `outdigits' undeclared here (not in a function)
>> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: initializer element is not constant
>> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: (near initialization for `nl_ext[0].offset')
>> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: parse error before ')' token
>> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: initializer element is not constant
>> ../../../../../newlib/libc/locale/nl_langinfo.c:66: error: (near initialization for `nl_ext[0]')
>> .... and many more similar complaints.
>> 
>> Johan van den Berg
> 
> hi John,
> 
> gcc and g++ version 4.5.3 (GCC)
> as provided by cygwin packages
> 
> gcc4-core                      4.5.3-3
> gcc4-g++                       4.5.3-3
> 
> usually this is enough for building
> 
>  <source_path>/configure
>  make
> 
> Regards
> Marco
> 
> 
> 
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
> 


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

Johan,

please don't http://cygwin.com/acronyms/#TOFU.  Thanks.

On Jan  4 21:25, Johan van den Berg wrote:
> I am very happy to report that increasing the send and receive buffers
> has done the job (at least, on a 10MBit link but will be testing a
> 100Mbit in a few days). I calculated the ideal size as per
> http://www.ibm.com/developerworks/linux/library/l-hisock/index.html

it's nice to know that you could increase the performance by increasing
the buffer sizes.  However, I'm reluctant to implement this as a generic
option.  As far as I know the socket buffers are taken from nonpaged pool,
so generically using 2 Meg buffers will take a lot of precious resources.

I made a test in a local LAN between Linux and a W7 64 bit machine, and
I didn't see a lot of difference between 64K, 2 Megs, or letting the OS
decide.  So I'm wondering if it's not the best option to let the OS
decide starting with Vista and later.

How's the performance in your scenario when applying the below patch
instead of yours?


Corinna


Index: net.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/net.cc,v
retrieving revision 1.294
diff -u -p -r1.294 net.cc
--- net.cc	17 Dec 2011 23:39:46 -0000	1.294
+++ net.cc	9 Jan 2012 13:41:23 -0000
@@ -569,26 +569,46 @@ fdsock (cygheap_fdmanip& fd, const devic
      be nice, though.
 
      (*) Maximum normal TCP window size.  Coincidence?  */
-  ((fhandler_socket *) fd)->rmem () = 65535;
-  ((fhandler_socket *) fd)->wmem () = 65535;
-  if (::setsockopt (soc, SOL_SOCKET, SO_RCVBUF,
-		    (char *) &((fhandler_socket *) fd)->rmem (), sizeof (int)))
+  if (wincap.has_sendmsg ())	/* FIXME.  Invent another wincap. */
     {
-      debug_printf ("setsockopt(SO_RCVBUF) failed, %lu", WSAGetLastError ());
       if (::getsockopt (soc, SOL_SOCKET, SO_RCVBUF,
 			(char *) &((fhandler_socket *) fd)->rmem (),
 			(size = sizeof (int), &size)))
 	system_printf ("getsockopt(SO_RCVBUF) failed, %lu", WSAGetLastError ());
-    }
-  if (::setsockopt (soc, SOL_SOCKET, SO_SNDBUF,
-		    (char *) &((fhandler_socket *) fd)->wmem (), sizeof (int)))
-    {
-      debug_printf ("setsockopt(SO_SNDBUF) failed, %lu", WSAGetLastError ());
       if (::getsockopt (soc, SOL_SOCKET, SO_SNDBUF,
 			(char *) &((fhandler_socket *) fd)->wmem (),
 			(size = sizeof (int), &size)))
 	system_printf ("getsockopt(SO_SNDBUF) failed, %lu", WSAGetLastError ());
     }
+  else
+    {
+      ((fhandler_socket *) fd)->rmem () = 65535;
+      ((fhandler_socket *) fd)->wmem () = 65535;
+      if (::setsockopt (soc, SOL_SOCKET, SO_RCVBUF,
+			(char *) &((fhandler_socket *) fd)->rmem (),
+			sizeof (int)))
+	{
+	  debug_printf ("setsockopt(SO_RCVBUF) failed, %lu",
+			WSAGetLastError ());
+	  if (::getsockopt (soc, SOL_SOCKET, SO_RCVBUF,
+			    (char *) &((fhandler_socket *) fd)->rmem (),
+			    (size = sizeof (int), &size)))
+	    system_printf ("getsockopt(SO_RCVBUF) failed, %lu",
+			   WSAGetLastError ());
+	}
+      if (::setsockopt (soc, SOL_SOCKET, SO_SNDBUF,
+			(char *) &((fhandler_socket *) fd)->wmem (),
+			sizeof (int)))
+	{
+	  debug_printf ("setsockopt(SO_SNDBUF) failed, %lu",
+			WSAGetLastError ());
+	  if (::getsockopt (soc, SOL_SOCKET, SO_SNDBUF,
+			    (char *) &((fhandler_socket *) fd)->wmem (),
+			    (size = sizeof (int), &size)))
+	    system_printf ("getsockopt(SO_SNDBUF) failed, %lu",
+			   WSAGetLastError ());
+	}
+    }
 
   return true;
 }
Index: fhandler_socket.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/fhandler_socket.cc,v
retrieving revision 1.284
diff -u -p -r1.284 fhandler_socket.cc
--- fhandler_socket.cc	4 Dec 2011 17:58:24 -0000	1.284
+++ fhandler_socket.cc	9 Jan 2012 13:41:23 -0000
@@ -1598,7 +1598,7 @@ fhandler_socket::send_internal (struct _
       /* CV 2009-12-02: Don't split datagram messages. */
       /* FIXME: Look for a way to split a message into the least number of
 		pieces to minimize the number of WsaSendTo calls. */
-      if (get_socket_type () == SOCK_STREAM)
+      if (!wincap.has_sendmsg () && get_socket_type () == SOCK_STREAM)
 	{
 	  buf.buf = wsamsg->lpBuffers[i].buf + off;
 	  buf.len = wsamsg->lpBuffers[i].len - off;
@@ -1611,7 +1611,7 @@ fhandler_socket::send_internal (struct _
 	{
 	  if (use_sendmsg)
 	    res = WSASendMsg (get_socket (), wsamsg, flags, &ret, NULL, NULL);
-	  else if (get_socket_type () == SOCK_STREAM)
+	  else if (!wincap.has_sendmsg () && get_socket_type () == SOCK_STREAM)
 	    res = WSASendTo (get_socket (), &buf, 1, &ret, flags,
 			     wsamsg->name, wsamsg->namelen, NULL, NULL);
 	  else

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Václav Zeman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/09/2012 02:43 PM, Corinna Vinschen wrote:
> Johan,
> 
> please don't http://cygwin.com/acronyms/#TOFU.  Thanks.
> 
> On Jan  4 21:25, Johan van den Berg wrote:
>> I am very happy to report that increasing the send and receive
>> buffers has done the job (at least, on a 10MBit link but will be
>> testing a 100Mbit in a few days). I calculated the ideal size as
>> per 
>> http://www.ibm.com/developerworks/linux/library/l-hisock/index.html
>
>> 
> it's nice to know that you could increase the performance by
> increasing the buffer sizes.  However, I'm reluctant to implement
> this as a generic option.  As far as I know the socket buffers are
> taken from nonpaged pool, so generically using 2 Meg buffers will
> take a lot of precious resources.
> 
> I made a test in a local LAN between Linux and a W7 64 bit machine,
> and I didn't see a lot of difference between 64K, 2 Megs, or
> letting the OS decide.  So I'm wondering if it's not the best
> option to let the OS decide starting with Vista and later.
Testing it on LAN will not show much. The buffer/TCP window size is
important for high latency * bandwidth product connections.

> 
> How's the performance in your scenario when applying the below
> patch instead of yours?

- -- 
VZ
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk8LF40ACgkQbJlIwZz1Ood/rwEAojdJKMJFtpmjKfOelJxa0p5L
s8aSKELVKao7/IN0WAIA/0Z7osFYBOw4plvQ7ToDLHgquhbKBdI+9FfDSIta7PIP
=wpBf
-----END PGP SIGNATURE-----

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance

[Henry S. Thompson]

Corinna Vinschen writes:

> it's nice to know that you could increase the performance by increasing
> the buffer sizes.  However, I'm reluctant to implement this as a generic
> option.  As far as I know the socket buffers are taken from nonpaged pool,
> so generically using 2 Meg buffers will take a lot of precious resources.

And contribute to bufferbloat problems [1] [2] elsewhere!

ht

[1] http://queue.acm.org/detail.cfm?id=2076798
[2] http://queue.acm.org/detail.cfm?id=2071893
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: ht@inf.ed.ac.uk
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

On 1/9/12 11:22 PM, Henry S. Thompson wrote:
> Corinna Vinschen writes:
> 
>> it's nice to know that you could increase the performance by increasing
>> the buffer sizes.  However, I'm reluctant to implement this as a generic
>> option.  As far as I know the socket buffers are taken from nonpaged pool,
>> so generically using 2 Meg buffers will take a lot of precious resources.
> 
> And contribute to bufferbloat problems [1] [2] elsewhere!

Large socket buffers don't contribute bufferbloat: a socket's
send-buffer holds bytes, not packets. It sits above TCP on the
networking stack, thus doesn't affect TCP's flow control decisions.
Bufferbloat happens when deep IP-packet buffers confuse TCP's
congestion avoidance, not when the OS feeds bytes to TCP more efficiently.

Of course, using a larger socket buffer _can_ increase the
send(2)-to-wire latency, but that's not quite the same thing as
bufferbloat.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 235 bytes --]

Re: socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

On Jan  9 17:36, Václav Zeman wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 01/09/2012 02:43 PM, Corinna Vinschen wrote:
> > Johan,
> > 
> > please don't http://cygwin.com/acronyms/#TOFU.  Thanks.
> > 
> > On Jan  4 21:25, Johan van den Berg wrote:
> >> I am very happy to report that increasing the send and receive
> >> buffers has done the job (at least, on a 10MBit link but will be
> >> testing a 100Mbit in a few days). I calculated the ideal size as
> >> per 
> >> http://www.ibm.com/developerworks/linux/library/l-hisock/index.html
> >
> >> 
> > it's nice to know that you could increase the performance by
> > increasing the buffer sizes.  However, I'm reluctant to implement
> > this as a generic option.  As far as I know the socket buffers are
> > taken from nonpaged pool, so generically using 2 Meg buffers will
> > take a lot of precious resources.
> > 
> > I made a test in a local LAN between Linux and a W7 64 bit machine,
> > and I didn't see a lot of difference between 64K, 2 Megs, or
> > letting the OS decide.  So I'm wondering if it's not the best
> > option to let the OS decide starting with Vista and later.
> Testing it on LAN will not show much. The buffer/TCP window size is
> important for high latency * bandwidth product connections.

Right, but that's not the default scenario.  Cygwin's job in this case
is to provide a POSIX socket API.  It neither takes over the job of
the TCP stack, nor does it know the circumstances in which the socket
will be used.  How should it decide what the ideal buffer sizes are?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Johan van den Berg]

On 09 Jan 2012, at 3:43 PM, Corinna Vinschen wrote:

> How's the performance in your scenario when applying the below patch
> instead of yours?

I have to run back with my tails between my legs. I implemented your patch, and the transfer speed on a 200ms latency, 10mbit max link went down to 5-6mbit using rsync. I then rolled back to my version, and suddenly also got 5-6mbit. I started another rsync and I was able to max the 10mbit line, hence, I think my original patch never had the effect I hoped for.

Checking further, I noticed that stopping a task in windows task scheduler doesn't actually stop the rsync, so the only reason why I then must have seen that 10mbit max on my patch was simply because another rsync was already running ;(

I am now however back to the drawing board. With your patch on both ends of the line, with a client rsync option of "--sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000" I still only get 5-6mbit max. I installed iperf on both ends, and no combination of settings (higher window size, higher MSS) will give me more than 5-6mbit transfer rate, except when I add the -P option which does parallel transfers. As soon as I do parallel, I can max the line. I then tested with a 100mbit link, and got similar results.

Thinking outside the box, I started up iperf on a linux box on the other end of a 100mbit line:

Cygwin to cygwin = 5mbit
Cygwin to linux = 5mbit
Linux to linux = 28mbit

In all cases, adjusting the window size had no effect other than making the client "think" it can transfer faster if the buffer is bigger than the total amount of data to send.

Any advice while I carry on trying to figure this out?

Kind regards
Johan

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

On Jan 10 14:45, Johan van den Berg wrote:
> 
> On 09 Jan 2012, at 3:43 PM, Corinna Vinschen wrote:
> 
> > How's the performance in your scenario when applying the below patch
> > instead of yours?
> 
> I have to run back with my tails between my legs. I implemented your patch, and the transfer speed on a 200ms latency, 10mbit max link went down to 5-6mbit using rsync. I then rolled back to my version, and suddenly also got 5-6mbit. I started another rsync and I was able to max the 10mbit line, hence, I think my original patch never had the effect I hoped for.
> 
> Checking further, I noticed that stopping a task in windows task scheduler doesn't actually stop the rsync, so the only reason why I then must have seen that 10mbit max on my patch was simply because another rsync was already running ;(
> 
> I am now however back to the drawing board. With your patch on both ends of the line, with a client rsync option of "--sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000" I still only get 5-6mbit max. I installed iperf on both ends, and no combination of settings (higher window size, higher MSS) will give me more than 5-6mbit transfer rate, except when I add the -P option which does parallel transfers. As soon as I do parallel, I can max the line. I then tested with a 100mbit link, and got similar results.
> 
> Thinking outside the box, I started up iperf on a linux box on the other end of a 100mbit line:
> 
> Cygwin to cygwin = 5mbit
> Cygwin to linux = 5mbit
> Linux to linux = 28mbit
> 
> In all cases, adjusting the window size had no effect other than making the client "think" it can transfer faster if the buffer is bigger than the total amount of data to send.
> 
> Any advice while I carry on trying to figure this out?

What Windows versions are we talking about?  Is that pre-Vista?  XP,
for instance?  If so, setting the buffer size > 64K should have no effect.

I really don't know why the performance should be so much worse than
under Linux in your scenario, sorry.  Cygwin is not trying to do
anything fancy.  The speed should be basically in the same range as on
Linux.

At least it is for me when using sftp.  When using scp I just found that
I get a similar bad performance, only 6.9 MB/s instead of 35 MB/s.

Is it possible that the limiting factor is not the socket, but the pipes
between rsync and ssh, assuming you are using rsync over ssh?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Steven Hartland]

If your running Windows 7 or 2k8 are you running the following hotfix, if not
you should try that too, just in case you machine has got a degraded tcp
stack.

http://support.microsoft.com/kb/983528

    Regards
    Steve

----- Original Message ----- 
From: "Corinna Vinschen"
Sent: Tuesday, January 10, 2012 2:45 PM
Subject: Re: socket performance (was Re: Building cygwin1.dll)


> On Jan 10 14:45, Johan van den Berg wrote:
>>
>> On 09 Jan 2012, at 3:43 PM, Corinna Vinschen wrote:
>>
>> > How's the performance in your scenario when applying the below patch
>> > instead of yours?
>>
>> I have to run back with my tails between my legs. I implemented your patch, and the transfer speed on a 200ms latency, 10mbit 
>> max link went down to 5-6mbit using rsync. I then rolled back to my version, and suddenly also got 5-6mbit. I started another 
>> rsync and I was able to max the 10mbit line, hence, I think my original patch never had the effect I hoped for.
>>
>> Checking further, I noticed that stopping a task in windows task scheduler doesn't actually stop the rsync, so the only reason 
>> why I then must have seen that 10mbit max on my patch was simply because another rsync was already running ;(
>>
>> I am now however back to the drawing board. With your patch on both ends of the line, with a client rsync option of 
>> "--sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000" I still only get 5-6mbit max. I installed iperf on both ends, and no 
>> combination of settings (higher window size, higher MSS) will give me more than 5-6mbit transfer rate, except when I add the -P 
>> option which does parallel transfers. As soon as I do parallel, I can max the line. I then tested with a 100mbit link, and got 
>> similar results.
>>
>> Thinking outside the box, I started up iperf on a linux box on the other end of a 100mbit line:
>>
>> Cygwin to cygwin = 5mbit
>> Cygwin to linux = 5mbit
>> Linux to linux = 28mbit
>>
>> In all cases, adjusting the window size had no effect other than making the client "think" it can transfer faster if the buffer 
>> is bigger than the total amount of data to send.
>>
>> Any advice while I carry on trying to figure this out?
>
> What Windows versions are we talking about?  Is that pre-Vista?  XP,
> for instance?  If so, setting the buffer size > 64K should have no effect.
>
> I really don't know why the performance should be so much worse than
> under Linux in your scenario, sorry.  Cygwin is not trying to do
> anything fancy.  The speed should be basically in the same range as on
> Linux.
>
> At least it is for me when using sftp.  When using scp I just found that
> I get a similar bad performance, only 6.9 MB/s instead of 35 MB/s.
>
> Is it possible that the limiting factor is not the socket, but the pipes
> between rsync and ssh, assuming you are using rsync over ssh?
>
>
> Corinna
>
> -- 
> Corinna Vinschen                  Please, send mails regarding Cygwin to
> Cygwin Project Co-Leader          cygwin AT cygwin DOT com
> Red Hat
>
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>
> 



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

On Jan 10 15:24, Steven Hartland wrote:
> If your running Windows 7 or 2k8 are you running the following hotfix, if not
> you should try that too, just in case you machine has got a degraded tcp
> stack.
> 
> http://support.microsoft.com/kb/983528

I tried that, but it doesn't install.  The installer tells me "The
update is not applicable to your computer."  This is W7 64 bit on a
Lenovo laptop.

What's also puzzeling is this.  I tried this on three virtual machines
as well, W7 32, W7 64, 2008R2 64.  In all three VMs scp was almost just
as fast as sftp, about 13 MB/s. 

So why is scp (and rsync, too, btw) half as fast on the real HW as on
the VMs while sftp is three times faster?!?  That doesn't make sense.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

On Jan 10 17:28, Corinna Vinschen wrote:
> On Jan 10 15:24, Steven Hartland wrote:
> > If your running Windows 7 or 2k8 are you running the following hotfix, if not
> > you should try that too, just in case you machine has got a degraded tcp
> > stack.
> > 
> > http://support.microsoft.com/kb/983528
> 
> I tried that, but it doesn't install.  The installer tells me "The
> update is not applicable to your computer."  This is W7 64 bit on a
> Lenovo laptop.
> 
> What's also puzzeling is this.  I tried this on three virtual machines
> as well, W7 32, W7 64, 2008R2 64.  In all three VMs scp was almost just
> as fast as sftp, about 13 MB/s. 
> 
> So why is scp (and rsync, too, btw) half as fast on the real HW as on
> the VMs while sftp is three times faster?!?  That doesn't make sense.

...and just for kicks I replaced the laptop drive with another one
holding a W7 32 bit installation and tried with that.  Now scp and sftp
are both fast, about 36 MB/s.  Go figure!


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Steven Hartland]

----- Original Message ----- 
From: "Corinna Vinschen"
Sent: Tuesday, January 10, 2012 4:28 PM
Subject: Re: socket performance (was Re: Building cygwin1.dll)


> On Jan 10 15:24, Steven Hartland wrote:
>> If your running Windows 7 or 2k8 are you running the following hotfix, if not
>> you should try that too, just in case you machine has got a degraded tcp
>> stack.
>> 
>> http://support.microsoft.com/kb/983528
> 
> I tried that, but it doesn't install.  The installer tells me "The
> update is not applicable to your computer."  This is W7 64 bit on a
> Lenovo laptop.
> 
> What's also puzzeling is this.  I tried this on three virtual machines
> as well, W7 32, W7 64, 2008R2 64.  In all three VMs scp was almost just
> as fast as sftp, about 13 MB/s. 
> 
> So why is scp (and rsync, too, btw) half as fast on the real HW as on
> the VMs while sftp is three times faster?!?  That doesn't make sense.

One of our guys had the same, when they tried to install the fix but when
I downloaded it and tried on exactly the same machine without changing
anything it just worked.

I know it sounds silly but did you download the correct version on the
64bit machine?

If this bug is effecting your you need to ensure that you make no connections
using the machine that could possibly trigger the throughput issues prior to
testing. Otherwise you'll be using a tcp stack without scaling.

If you want me to mail you the exact download link I have from MS for this
patch to check, let me know.

    Regards
    Steve


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

On Jan 10 17:05, Steven Hartland wrote:
> ----- Original Message ----- From: "Corinna Vinschen"
> >On Jan 10 15:24, Steven Hartland wrote:
> >>http://support.microsoft.com/kb/983528
> >
> >I tried that, but it doesn't install.  The installer tells me "The
> >update is not applicable to your computer."  This is W7 64 bit on a
> >Lenovo laptop.
> >
> One of our guys had the same, when they tried to install the fix but when
> I downloaded it and tried on exactly the same machine without changing
> anything it just worked.
> 
> I know it sounds silly but did you download the correct version on the
> 64bit machine?

Well, the file I downloaded was a self-extracting zip archive and the
file it contains is called Windows6.1-KB983528-x64.msu, so I'm fairly
certain it's the right one for an AMD64 system.

> If this bug is effecting your you need to ensure that you make no connections
> using the machine that could possibly trigger the throughput issues prior to
> testing. Otherwise you'll be using a tcp stack without scaling.

Sounds tricky.  The laptop is a domain member and it's not in the same
room with me so I'm running this via rdesktop.  But that would be fixable.

However, if this issue is the problem, then why is sftp fast and only scp
and rsync slow?  I can reproduce this at will, not only once in the
right order.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Steven Hartland]

----- Original Message ----- 
From: "Corinna Vinschen" 
> Well, the file I downloaded was a self-extracting zip archive and the
> file it contains is called Windows6.1-KB983528-x64.msu, so I'm fairly
> certain it's the right one for an AMD64 system.

Windows6.1-KB983528-x64.msu is the file I have here too so unless
there's something else a play i.e. running as Administrator UAC rubish?

>> If this bug is effecting your you need to ensure that you make no connections
>> using the machine that could possibly trigger the throughput issues prior to
>> testing. Otherwise you'll be using a tcp stack without scaling.
> 
> Sounds tricky.  The laptop is a domain member and it's not in the same
> room with me so I'm running this via rdesktop.  But that would be fixable.
> 
> However, if this issue is the problem, then why is sftp fast and only scp
> and rsync slow?  I can reproduce this at will, not only once in the
> right order.

Hmm that would indeed not be explained by the KB iirc there are some
internal differences between the way sftp and scp transfer data.

Do you have a wireshark dump of the two transfers we could examine?

    Regards
    Steve


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Johan van den Berg]

On 10 Jan 2012, at 4:45 PM, Corinna Vinschen wrote:

> What Windows versions are we talking about?  Is that pre-Vista?  XP,
> for instance?  If so, setting the buffer size > 64K should have no effect.

Destination Windows: Windows Server 2008 R2 Standard 64bit (Intel Xeon) 
Destination Linux: Linux 2.6.32-71.el6 x86_64 (CentOS 6.0 Final)
Source Windows: Windows Server 2008 R2 Enterprise 64bit (Intel Xeon)
Source Linux: Linux  2.6.18-238.12.1.el5 x86_64 (CentOS 5.6 Final)

Also tried a source Windows 2000 with same effect.

> I really don't know why the performance should be so much worse than
> under Linux in your scenario, sorry.  Cygwin is not trying to do
> anything fancy.  The speed should be basically in the same range as on
> Linux.

Which means the culprit is windows (as was expected)...

> At least it is for me when using sftp.  When using scp I just found that
> I get a similar bad performance, only 6.9 MB/s instead of 35 MB/s.
> 
> Is it possible that the limiting factor is not the socket, but the pipes
> between rsync and ssh, assuming you are using rsync over ssh?

I was using rsync raw without ssh. I get the feeling I will need to read sftp's code now ;) Either sftp is setting something special on the socket options that scp / rsync is not, or sftp employs parallel connections. The main reason for using rsync is consistency of data and automation of mirroring, so it is by far the best tool for the job if you want 100% data integrity (unless someone else knows of a better tool for the job).
--

Kind regards
Johan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

On Jan 10 18:21, Steven Hartland wrote:
> ----- Original Message ----- From: "Corinna Vinschen"
> >Well, the file I downloaded was a self-extracting zip archive and the
> >file it contains is called Windows6.1-KB983528-x64.msu, so I'm fairly
> >certain it's the right one for an AMD64 system.
> 
> Windows6.1-KB983528-x64.msu is the file I have here too so unless
> there's something else a play i.e. running as Administrator UAC rubish?

Unlikely.  When trying to run .msu files, there's not evewn the choice
to run them normally or "as administrator".

> >>If this bug is effecting your you need to ensure that you make no connections
> >>using the machine that could possibly trigger the throughput issues prior to
> >>testing. Otherwise you'll be using a tcp stack without scaling.
> >
> >Sounds tricky.  The laptop is a domain member and it's not in the same
> >room with me so I'm running this via rdesktop.  But that would be fixable.
> >
> >However, if this issue is the problem, then why is sftp fast and only scp
> >and rsync slow?  I can reproduce this at will, not only once in the
> >right order.
> 
> Hmm that would indeed not be explained by the KB iirc there are some
> internal differences between the way sftp and scp transfer data.
> 
> Do you have a wireshark dump of the two transfers we could examine?

I created them.  I see the difference, but I can't explain it.  I
experimented with the TCP_NODELAY flag, but it had only a marginal
effect.  If you want to take a look into the dumps, I'll send you the
download addresses in a minute via PM.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Carson Chittom]

Corinna Vinschen writes:

> On Jan 10 18:21, Steven Hartland wrote:
>> ----- Original Message ----- From: "Corinna Vinschen"
>> >Well, the file I downloaded was a self-extracting zip archive and the
>> >file it contains is called Windows6.1-KB983528-x64.msu, so I'm fairly
>> >certain it's the right one for an AMD64 system.
>> 
>> Windows6.1-KB983528-x64.msu is the file I have here too so unless
>> there's something else a play i.e. running as Administrator UAC rubish?
>
> Unlikely.  When trying to run .msu files, there's not evewn the choice
> to run them normally or "as administrator".

That hotfix is included in Win7/2008R2's SP1, which I'm guessing you're
running.  Which of course is why the installer tells you it's not for
your OS.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: socket performance (was Re: Building cygwin1.dll)

[Corinna Vinschen]

On Jan 11 08:19, Carson Chittom wrote:
> Corinna Vinschen writes:
> 
> > On Jan 10 18:21, Steven Hartland wrote:
> >> ----- Original Message ----- From: "Corinna Vinschen"
> >> >Well, the file I downloaded was a self-extracting zip archive and the
> >> >file it contains is called Windows6.1-KB983528-x64.msu, so I'm fairly
> >> >certain it's the right one for an AMD64 system.
> >> 
> >> Windows6.1-KB983528-x64.msu is the file I have here too so unless
> >> there's something else a play i.e. running as Administrator UAC rubish?
> >
> > Unlikely.  When trying to run .msu files, there's not evewn the choice
> > to run them normally or "as administrator".
> 
> That hotfix is included in Win7/2008R2's SP1, which I'm guessing you're
> running.  Which of course is why the installer tells you it's not for
> your OS.

Uh, that explains it.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple