Solves the segfault here: http://cygwin.com/ml/cygwin/2013-09/msg00397.html
but does not address the fact that we are still screwy with regards to
rlimit.

======
Ultimately, based on my understanding of POSIX and glibc, my goal is to
have a number of changes (this patch only scratches the surface; there's
more to go):

dtable.h tracks soft and hard limits, inherited over fork and preserved
across exec

hard limit starts at OPEN_MAX_MAX and can only be reduced
soft limit starts at hard limit, and can be reduced to _POSIX_OPEN_MAX (8)
dtable.size starts at MAX(32, fork/exec size)

getdtablesize() and sysconf(_SC_OPEN_MAX) always returns the soft limit,
as in glibc and permitted by POSIX (_SC_OPEN_MAX is the only sysconf
variable that can be runtime dynamic)

dtable.size is decoupled from soft limit, and is guaranteed to be <=
hard limit.  It can grow up to current soft limit; but soft limit can
later be reduced lower than dtable.size (glibc does this); on fork and
exec, we are careful to still allow fds beyond the current soft limit.

getrlimit(RLIMIT_NOFILE, &r) => returns soft and hard limits from dtable
rather than hard limit as a constant and soft limit as current dtable.size

setrlimit(RLIMIT_NOFILE, &r) => cannot set hard limit to unlimited; soft
limit of unlimited is translated to current hard limit; hard limit
cannot be increased (EPERM) or reduced below dtable.size (EINVAL); soft
limit can be reduced arbitrarily (including below OPEN_MAX of 256)

setdtablesize() => guarantees that dtable.size is at least that large
(must be <= soft limit), but does not lower dtable.size or change limits
=====

2013-09-25  Eric Blake  <eblake@redhat.com>

	dup2: fix off-by-one crash
	* dtable.cc (dup3): Fix off-by-one.
	(find_unused_handle): Reduce time spent expanding during dup.
	* syscalls.cc (setdtablesize): Report error on invalid value.

diff --git i/winsup/cygwin/dtable.cc w/winsup/cygwin/dtable.cc
index 2501a26..c2982a8 100644
--- i/winsup/cygwin/dtable.cc
+++ w/winsup/cygwin/dtable.cc
@@ -233,7 +233,7 @@ dtable::find_unused_handle (int start)
 	if (fds[i] == NULL)
 	  return i;
     }
-  while (extend (NOFILE_INCR));
+  while (extend (MAX (NOFILE_INCR, start - size)));
   return -1;
 }

@@ -754,7 +754,7 @@ dtable::dup3 (int oldfd, int newfd, int flags)

   if (!not_open (newfd))
     close (newfd);
-  else if ((size_t) newfd > size
+  else if ((size_t) newfd >= size
 	   && find_unused_handle (newfd) < 0)
     /* couldn't extend fdtab */
     {
diff --git i/winsup/cygwin/syscalls.cc w/winsup/cygwin/syscalls.cc
index e1886e6..8c1c70a 100644
--- i/winsup/cygwin/syscalls.cc
+++ w/winsup/cygwin/syscalls.cc
@@ -2578,6 +2578,9 @@ system (const char *cmdstring)
 extern "C" int
 setdtablesize (int size)
 {
+  if (size < 0)
+    return -1;
+
   if (size <= (int)cygheap->fdtab.size || cygheap->fdtab.extend (size -
cygheap->fdtab.size))
     return 0;

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org