Problem with Cygwin's fdopen and Windows handles

Problem with Cygwin's fdopen and Windows handles

[Juanjo]

Actually there are two different problems related to my implementation of a
Common Lisp environment, ecl (http://ecls.sf.net)

The first one has to do with fork() not working, due to the fact that ECL
injects DLLs using dlopen() and they are then improperly loaded. I have seen in
the mailing list that this is a known problem with no solution so far.

To cope with that problem I had to resort to CreateProcess, a Windows routine
that allows us to redirect the input/output/error channels of a process as
needed. The problem I have is that the C streams that result from the Windows
handle can only be read with read() and not with fread(). This is a problem
because the ECL environment needs buffered I/O, with locking and so on, and we
rely on C streams for that.

I attach a small test program that segfaults when using fread() on the stream
created with fdopen(). notice that read() works.

Any help is really welcome and appreciated.

Juanjo


/* -*- mode: c; c-basic-offset: 8 -*- */
/*
    unixsys.s  -- Unix shell interface.
*/
/*
    Copyright (c) 1984, Taiichi Yuasa and Masami Hagiya.
    Copyright (c) 1990, Giuseppe Attardi.
    Copyright (c) 2001, Juan Jose Garcia Ripoll.

    ECL is free software; you can redistribute it and/or
    modify it under the terms of the GNU Library General Public
    License as published by the Free Software Foundation; either
    version 2 of the License, or (at your option) any later version.

    See file '../Copyright' for full details.
*/

#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/cygwin.h> /* For cygwin_attach_handle_to_fd() */
#include <windows.h>

int main()
{
	BOOL ok;
	STARTUPINFO st_info;
	PROCESS_INFORMATION pr_info;
	HANDLE child_stdout, child_stdin, child_stderr;
	HANDLE current = GetCurrentProcess();
	SECURITY_ATTRIBUTES attr;
	int parent_read;

	ZeroMemory(&attr, sizeof(attr));
	attr.nLength = sizeof(attr);
	attr.lpSecurityDescriptor = NULL;
	attr.bInheritHandle = TRUE;

	/* Creates a pipe that we can write to and the
	   child reads from. We duplicate one extreme of the
	   pipe so that the child does not inherit it. */
	child_stdin = NULL;
	child_stderr = NULL;
	{
		HANDLE tmp;
		ok = CreatePipe(&tmp, &child_stdout, &attr, 0);
		if (ok) {
			ok = DuplicateHandle(current, tmp, current,
					     &tmp, 0, FALSE,
					     DUPLICATE_CLOSE_SOURCE |
					     DUPLICATE_SAME_ACCESS);
			if (ok) {
				parent_read =
                                        cygwin_attach_handle_to_fd
                                        (0, -1, tmp, S_IRWXU, GENERIC_READ);
                                printf("parent_read=%d\n",parent_read);
				if (parent_read < 0)
					printf("open_osfhandle failed\n");
			}
		}
	}

	/* Launches the process
	 */
	ZeroMemory(&st_info, sizeof(st_info));
	st_info.cb = sizeof(st_info);
	st_info.lpTitle = NULL; /* No window title, just exec name */
	st_info.dwFlags = STARTF_USESTDHANDLES | STARTF_USESHOWWINDOW; /* Specify
std{in,out,err} */
	st_info.wShowWindow = SW_HIDE;
	st_info.hStdInput = child_stdin;
	st_info.hStdOutput = child_stdout;
	st_info.hStdError = child_stderr;
	ZeroMemory(&pr_info, sizeof(pr_info));
	ok = CreateProcess(NULL, "c:\\cygwin\\bin\\echo.exe \"--version\"",
			   NULL, NULL, /* lpProcess/ThreadAttributes */
			   TRUE, /* Inherit handles (for files) */
			   /*CREATE_NEW_CONSOLE |*/
			   0 /*(input == Ct || output == Ct || error == Ct ? 0 : CREATE_NO_WINDOW)*/,
			   NULL, /* Inherit environment */
			   NULL, /* Current directory */
			   &st_info, /* Startup info */
			   &pr_info); /* Process info */
	if (!ok) {
		char *message;
		printf("ABORT\n");
		FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM |
			      FORMAT_MESSAGE_ALLOCATE_BUFFER,
			      0, GetLastError(), 0, (void*)&message, 0, NULL);
		printf("%s\n", message);
		LocalFree(message);
		return 1;
	}
	/* Now reads. No problem with C read */
        {
                char c[100];
                int n = read(parent_read, c, 10);
                c[n] = 0;
                printf("c[%d] = %s\n", n, c);
        }
	/* But this segfaults */
        {
                FILE *f= fdopen(parent_read, "rb");
                char c[100];
                int n;
                printf("fp=%p\n", f);
                n = fread(c, 4, 1, f);
                c[n] = 0;
                printf("c[%d] = %s\n", n, c);
        }
	return 0;
}



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with Cygwin's fdopen and Windows handles

[Christopher Faylor]

On Sun, May 29, 2011 at 11:35:03AM +0000, Juanjo wrote:
>Actually there are two different problems related to my implementation of a
>Common Lisp environment, ecl (http://ecls.sf.net)
>
>The first one has to do with fork() not working, due to the fact that ECL
>injects DLLs using dlopen() and they are then improperly loaded. I have seen in
>the mailing list that this is a known problem with no solution so far.
>
>To cope with that problem I had to resort to CreateProcess, a Windows routine
>that allows us to redirect the input/output/error channels of a process as
>needed. The problem I have is that the C streams that result from the Windows
>handle can only be read with read() and not with fread(). This is a problem
>because the ECL environment needs buffered I/O, with locking and so on, and we
>rely on C streams for that.
>
>I attach a small test program that segfaults when using fread() on the stream
>created with fdopen(). notice that read() works.
>
>Any help is really welcome and appreciated.

Unfortunately, cygwin_attach_handle_to_fd doesn't really work.  Cygwin
needs to know the type of handle it is attaching in order to set up
the correct type of file handler.  Since it doesn't do that the handle
is of limited utility.

It is possible for the function to be more intelligent since other parts
of cygwin try to figure out the handle type by querying attributes of the
handle.  So this is a http://cygwin.com/acronyms#SHTDI scenario.

I don't know what you mean by dlopen() causing fork not to work.  That's
obviously not normally the case.  If you are seeing something like this
then maybe your dlls are not properly based to avoid collisions.  If
that is the case then you should change your link line to specify unique
load addresses for each dll.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with Cygwin's fdopen and Windows handles

[Juanjo]

Christopher Faylor <cgf-use-the-mailinglist-please <at> cygwin.com> writes:
> 
> Unfortunately, cygwin_attach_handle_to_fd doesn't really work.  Cygwin
> needs to know the type of handle it is attaching in order to set up
> the correct type of file handler.  Since it doesn't do that the handle
> is of limited utility.

If this was true, the function should have then been removed from the
manual or marked as not working. But I believe this is not right, for
read() and file handlers work perfectly and the problem only 
arises with fread() !!!

> It is possible for the function to be more intelligent since other parts
> of cygwin try to figure out the handle type by querying attributes of the
> handle.  So this is a http://cygwin.com/acronyms#SHTDI scenario.

Sorry, no time nor interest to do it myself. I am just reporting it here
because some users of our software experienced this problem. I have a
lot of SHTDI in my own project and in any case, as I said before, I believe
this is not the cause of the problem -- it lays in fread()/fdopen()
not in read().
 
> I don't know what you mean by dlopen() causing fork not to work.  That's
> obviously not normally the case.  If you are seeing something like this
> then maybe your dlls are not properly based to avoid collisions.  If
> that is the case then you should change your link line to specify unique
> load addresses for each dll.

I have seen messages in a sibling mailing list reporting that fork()
fails when a program injects libraries using various mechanisms
     http://sourceware.org/ml/cygwin/2011-04/msg00185.html

In our case we just have one core library and other libraries that
are compiled on the fly and which are loaded with dlopen(). Loading
is fine and there are no collisions, the problem is that when fork() reloads 
them they do not end up in the right positions and cygwin complains.
It is not our job to hardcode addresses for libraries to be loaded and
do what cygwin is not doing right, which is to determine the 
right order of loading.

Juanjo


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with Cygwin's fdopen and Windows handles

[Ryan Johnson]

On 30/05/2011 3:34 AM, Juanjo wrote:
> Christopher Faylor<cgf-use-the-mailinglist-please<at>  cygwin.com>  writes:
>> Unfortunately, cygwin_attach_handle_to_fd doesn't really work.  Cygwin
>> needs to know the type of handle it is attaching in order to set up
>> the correct type of file handler.  Since it doesn't do that the handle
>> is of limited utility.
> If this was true, the function should have then been removed from the
> manual or marked as not working. But I believe this is not right, for
> read() and file handlers work perfectly and the problem only
> arises with fread() !!!
>
>> It is possible for the function to be more intelligent since other parts
>> of cygwin try to figure out the handle type by querying attributes of the
>> handle.  So this is a http://cygwin.com/acronyms#SHTDI scenario.
> Sorry, no time nor interest to do it myself. I am just reporting it here
> because some users of our software experienced this problem. I have a
> lot of SHTDI in my own project and in any case, as I said before, I believe
> this is not the cause of the problem -- it lays in fread()/fdopen()
> not in read().
>
>> I don't know what you mean by dlopen() causing fork not to work.  That's
>> obviously not normally the case.  If you are seeing something like this
>> then maybe your dlls are not properly based to avoid collisions.  If
>> that is the case then you should change your link line to specify unique
>> load addresses for each dll.
> I have seen messages in a sibling mailing list reporting that fork()
> fails when a program injects libraries using various mechanisms
>       http://sourceware.org/ml/cygwin/2011-04/msg00185.html
That is a *statically linked* test program whose dependent libraries 
have intentional dll base collisions. Be glad you're using dlopen -- at 
least there cygwin can usually bully Windows into doing what it wants.

> In our case we just have one core library and other libraries that
> are compiled on the fly and which are loaded with dlopen(). Loading
> is fine and there are no collisions, the problem is that when fork() reloads
> them they do not end up in the right positions and cygwin complains.
> It is not our job to hardcode addresses for libraries to be loaded and
> do what cygwin is not doing right, which is to determine the
> right order of loading.
Unfortunately, the Windows loader has a bad habit of shuffling things 
around the address space, and it does its damage before the cygwin dll 
even loads. Cygwin does its best to hide all this from you, but Windows 
Vista/7 actually makes it impossible to guarantee every fork will 
succeed. It's all part of the cost of doing business with Windows.

That said, rebasing and setting tsaware on your dlls helps a lot, and 
there are patches in the works which should further improve the odds for 
precisely your scenario.

Ryan



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with Cygwin's fdopen and Windows handles

[Christopher Faylor]

On Mon, May 30, 2011 at 07:34:27AM +0000, Juanjo wrote:
>Christopher Faylor writes:
>>Unfortunately, cygwin_attach_handle_to_fd doesn't really work.  Cygwin
>>needs to know the type of handle it is attaching in order to set up the
>>correct type of file handler.  Since it doesn't do that the handle is
>>of limited utility.
>
>If this was true, the function should have then been removed from the
>manual or marked as not working.  But I believe this is not right, for
>read() and file handlers work perfectly and the problem only arises
>with fread() !!!

Please calm down.

I guess I shouldn't have said the "doesn't really work" and stuck with
"of limited utility".  fds attached with cygwin_attach_handle_to_fd are
not fully functional.

>> It is possible for the function to be more intelligent since other parts
>> of cygwin try to figure out the handle type by querying attributes of the
>> handle.  So this is a http://cygwin.com/acronyms#SHTDI scenario.
>
>Sorry, no time nor interest to do it myself.

I wasn't asking you to do it.  I don't think anyone cares enough about
the problem to add the low-level Windows code required to augment
cygwin_attach_handle_to_fd.

>>I don't know what you mean by dlopen() causing fork not to work.
>>That's obviously not normally the case.  If you are seeing something
>>like this then maybe your dlls are not properly based to avoid
>>collisions.  If that is the case then you should change your link line
>>to specify unique load addresses for each dll.
>
>I have seen messages in a sibling mailing list reporting that fork()
>fails when a program injects libraries using various mechanisms
>http://sourceware.org/ml/cygwin/2011-04/msg00185.html

"sibling mailing list"?  That is this mailing list.

>In our case we just have one core library and other libraries that are
>compiled on the fly and which are loaded with dlopen().  Loading is
>fine and there are no collisions, the problem is that when fork()
>reloads them they do not end up in the right positions and cygwin
>complains.  It is not our job to hardcode addresses for libraries to be
>loaded and do what cygwin is not doing right, which is to determine the
>right order of loading.

There has been lots and lots of mailing list discussion about how hard
it is to get dll load order correct in Windows when Cygwin emulates
fork.

If you have an application which uses a lot of dlls then best practice
for Windows DLLs is to build them with unique load addresses.  Barring
that you could rebase them with cygwin's rebase or rebaseall utilties.
Setting unique base addresses will actually cause your application to
load slightly faster whether you use Cygwin or not.
--
Christopher Faylor			spammer? ->	aaaspam@sourceware.org
Cygwin Co-Project Leader

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with Cygwin's fdopen and Windows handles

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

On 5/30/11 10:46 AM, Christopher Faylor wrote:
> On Mon, May 30, 2011 at 07:34:27AM +0000, Juanjo wrote:
>> Christopher Faylor writes:
>>> Unfortunately, cygwin_attach_handle_to_fd doesn't really work.  Cygwin
>>> needs to know the type of handle it is attaching in order to set up the
>>> correct type of file handler.  Since it doesn't do that the handle is
>>> of limited utility.
>>
>> If this was true, the function should have then been removed from the
>> manual or marked as not working.  But I believe this is not right, for
>> read() and file handlers work perfectly and the problem only arises
>> with fread() !!!
> 
> Please calm down.
> 
> I guess I shouldn't have said the "doesn't really work" and stuck with
> "of limited utility".  fds attached with cygwin_attach_handle_to_fd are
> not fully functional.

If cygwin_attach_handle_to_fd is ever deprecated or replaced entirely,
could we replace it with opening /proc/self/fd/handle:XXXX, where XXXX
is the decimal encoding of the handle value? It'd eliminate a Cygwin API
call and allow easier interaction with handles inherited from non-Cygwin
programs, and if the code exists to automatically detect the proper
fhandler type for a given HANDLE, the loss in API richness shouldn't matter.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Problem with Cygwin's fdopen and Windows handles

[Christopher Faylor]

On Mon, May 30, 2011 at 11:39:13AM -0700, Daniel Colascione wrote:
>On 5/30/11 10:46 AM, Christopher Faylor wrote:
>> On Mon, May 30, 2011 at 07:34:27AM +0000, Juanjo wrote:
>>> Christopher Faylor writes:
>>>> Unfortunately, cygwin_attach_handle_to_fd doesn't really work.  Cygwin
>>>> needs to know the type of handle it is attaching in order to set up the
>>>> correct type of file handler.  Since it doesn't do that the handle is
>>>> of limited utility.
>>>
>>> If this was true, the function should have then been removed from the
>>> manual or marked as not working.  But I believe this is not right, for
>>> read() and file handlers work perfectly and the problem only arises
>>> with fread() !!!
>> 
>> Please calm down.
>> 
>> I guess I shouldn't have said the "doesn't really work" and stuck with
>> "of limited utility".  fds attached with cygwin_attach_handle_to_fd are
>> not fully functional.
>
>If cygwin_attach_handle_to_fd is ever deprecated or replaced entirely,
>could we replace it with opening /proc/self/fd/handle:XXXX, where XXXX
>is the decimal encoding of the handle value?  It'd eliminate a Cygwin
>API call and allow easier interaction with handles inherited from
>non-Cygwin programs, and if the code exists to automatically detect the
>proper fhandler type for a given HANDLE, the loss in API richness
>shouldn't matter.

Getting rid of cygwin_attach_handle_to_fd would break backwards
compatibility.  It's not going anywhere.

Your scheme sounds like a whole lot of extra work that SHTDI.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with Cygwin's fdopen and Windows handles

[Juanjo]

Christopher Faylor <cgf-use-the-mailinglist-please <at> cygwin.com> writes:
> 
> Please calm down.

I am calm :-) I just happen to like exclamation signs.
 
> I guess I shouldn't have said the "doesn't really work" and stuck with
> "of limited utility".  fds attached with cygwin_attach_handle_to_fd are
> not fully functional.

You still have not answered my question. C fds are working. They
allow read() write() and close(). The question is actually why fread()
does not work. From your email I still am not convinced that
the problem is with cygwin_attach_handle_to_fd(). Do ANSI streams
need something that C file ids do not provide?

> >I have seen messages in a sibling mailing list reporting that fork()
> >fails when a program injects libraries using various mechanisms
> >http://sourceware.org/ml/cygwin/2011-04/msg00185.html
> 
> "sibling mailing list"?  That is this mailing list.

Sorry, it is very difficult for me to identify in gmane the actual name
of the mailing list.

> If you have an application which uses a lot of dlls then best practice
> for Windows DLLs is to build them with unique load addresses.  Barring
> that you could rebase them with cygwin's rebase or rebaseall utilties.
> Setting unique base addresses will actually cause your application to
> load slightly faster whether you use Cygwin or not.

It seems I did not express myself properly. Code is compiled on the fly.
DLLs do not survive beyond program execution. This is a dynamic language
(Common Lisp btw) and functions are compiled and run and consumed
quickly. Calling rebase for each invocation is overkill (it will scan the
whole system!) and I would not even know how to start assigning
fixed addresses to libraries myself.

Let me insist that we could live without fork if only
fread() on Windows handles worked -- all ECL needs is a means
to execute programs with redirected input/output & error channels
(a bit beyond what popen() does).

If the answer is we should abandon Cygwin or deprecate it for this
project, it would be nice to know.

Best regards,

Juanjo


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with Cygwin's fdopen and Windows handles

[Charles Wilson]

On 5/30/2011 5:12 PM, Juanjo wrote:
> It seems I did not express myself properly. Code is compiled on the fly.
> DLLs do not survive beyond program execution. This is a dynamic language
> (Common Lisp btw) and functions are compiled and run and consumed
> quickly. Calling rebase for each invocation is overkill (it will scan the
> whole system!) and I would not even know how to start assigning
> fixed addresses to libraries myself.

Well, one workaround is to use the --enable-auto-image-base flag when
linking your ephemeral DLL.  This generates a custom image base address
based on a hash of the pathname of the DLL.  There's no guarantee this
doesn't clash with stuff, but at least it'll be *different* for dlls
that have different names.

Dunno of any solutions or recommendations related to the rest of your
email, tho -- sorry.

--
Chuck

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Why does windows have such probs with dynamically loaded libs?

[Linda Walsh]

Christopher Faylor wrote:
> If you have an application which uses a lot of dlls then best practice
> for Windows DLLs is to build them with unique load addresses.  Barring
> that you could rebase them with cygwin's rebase or rebaseall utilties.
> Setting unique base addresses will actually cause your application to
> load slightly faster whether you use Cygwin or not.
----

Other than the stock answer of poor design, it seems loading a
dynamically linked library at run-time shouldn't be a difficult task.

1) find out # of 'segments' and size.

2) allocate space somewhere 'unused' in the address space. (malloc seems
to usually work for this)

3) load contents

4) get the symbol(s) needed and add them to the loaded address, and pass
that back to the dlopen call for patching it's call tables so future
calls can call the libs directly.

5) enjoy. ... 

So why all these problems with conflicting load addresses?

When Cygwin forks, how different is it from linux (other than stock
answer of 'alot'),  i.e.  Does it create a new process and load the same
static libs in, then have problems with dynamically loaded libs because
they aren't recorded in the static binary?

Does cygwin actually copy, or attempt to setup COW pages that are not
from static libs?   If so, wouldn't that catch dynamically loaded libs?

This may be complete insanity, but given the low level of support of MS's
own Unix subsystem, I wonder if they might be persuaded (if it was
desired) to lend more help or hooks for cygwin to do its magic reliably.  

It seems like it would be a win for MS -- since many Windows users (not
just 'end', but corporate as well) often make use of Cygwin -- you'd
think MS might think kindly toward efforts to help Cygwin work well with
current versions of windows...but then my name is given as an example in
some political dictionaries as an example under 'naïve'  ;-/...



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why does windows have such probs with dynamically loaded libs?

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2545 bytes --]

On 05/31/2011 05:18 PM, Linda Walsh wrote:
> Other than the stock answer of poor design, it seems loading a
> dynamically linked library at run-time shouldn't be a difficult task.

Poor design of Windows, not of cygwin.

> When Cygwin forks, how different is it from linux (other than stock
> answer of 'alot'),  i.e.  Does it create a new process and load the same
> static libs in, then have problems with dynamically loaded libs because
> they aren't recorded in the static binary?

Windows does _not_ have a native copy-on-write fork operation.  Linux
does.  When Linux forks, it merely needs mark all pages as
copy-on-write, then both the parent and the child share the same memory
image until one or the other execs and gives up that memory.  When
cygwin forks, it must manually hand-copy every single page of memory
from the parent into the child, including pages that can't normally be
copied by user space apps but are instead allocated by Windows (such as
dlls).  How?  By loading the same dlls in the child as in the parent,
and _hoping_ that windows loads them at the same address as the parent.

> Does cygwin actually copy, or attempt to setup COW pages that are not
> from static libs?   If so, wouldn't that catch dynamically loaded libs?

Windows doesn't support COW pages between a parent process and its
spawned child, at least not in the windows subsystem.  And the
documentation of the various subsystems lacking to the point that cygwin
has no way of reproducing the subsystem setup utilized by Interix for
its fork implementation (Interix doesn't have to care about interaction
with the windows subsystem).  So yes, cygwin really is stuck with
copying data, and for the data not directly manageable by user space,
cygwin has to go to great lengths to circumvent random behaviors
introduced by newer windows versions to try to get dlls loaded into the
same location.

> This may be complete insanity, but given the low level of support of MS's
> own Unix subsystem, I wonder if they might be persuaded (if it was
> desired) to lend more help or hooks for cygwin to do its magic reliably. 

Put yourself in Microsoft's shoes - why would you want to make it easier
for free software to replace your proprietary software?  Once you answer
that question, then you can see why it is unlikely that Cygwin will ever
have enough weight to convince MS to make our lives easier.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Why does windows have such probs with dynamically loaded libs?

[Linda Walsh]

Eric Blake wrote:
>"  By loading the same dlls in the child as in the parent,
> and _hoping_ that windows loads them at the same address as the parent."
---
	Ah-hah!...that's the main rub.   I could see that other pages would have
to be copied 'manually', baring some kernel call (probably an undocumented
NT call) that would allow for setting up COW pages, but guessed for copy given
the thorough level of MS documentation available on such matters.


> 
>> Does cygwin actually copy, or attempt to setup COW pages that are not
>> from static libs?   If so, wouldn't that catch dynamically loaded libs?
> 
> Windows doesn't support COW pages between a parent process and its
> spawned child, at least not in the windows subsystem.  And the
> documentation of the various subsystems lacking to the point that cygwin
> has no way of reproducing the subsystem setup utilized by Interix for
> its fork implementation (Interix doesn't have to care about interaction
> with the windows subsystem).
---
	Hmmm...I wonder...do you know if Interix setups COW pages on fork?
If so, why in the heck would it perform so much more slowly than cygwin
when running the same tasks (shell scripts and such that do lots of little 
forks)....  its performance was pretty bad next to cygwin, though that was
under XP, and several years back that I tested, so it may have changed).

  So yes, cygwin really is stuck with
> copying data, and for the data not directly manageable by user space,
> cygwin has to go to great lengths to circumvent random behaviors
> introduced by newer windows versions to try to get dlls loaded into the
> same location.
---
	Ahhh...the 'security enhancements!'....famous for making everyone's
life more difficult...(the major failing of security design -- if it was 
designed to be 'easier', everyone would use it!)

> 
>> This may be complete insanity, but given the low level of support of MS's
>> own Unix subsystem, I wonder if they might be persuaded (if it was
>> desired) to lend more help or hooks for cygwin to do its magic reliably. 
> 
> Put yourself in Microsoft's shoes - why would you want to make it easier
> for free software
----
	Um note, that may be 'free software', but it is running on top of
MS's paid OS.  With sufficiently well developed interoperability, I could see
MS using cygwin's compat on Windows as a protective force against people
switching to linux-desktops, since with good cygwin support, they can
have the best of both worlds -- of course there's the possibility
that 'Wine' will continue to make marked improvements which could
swing things more in the linux direction.

But even there, MS could still benefit, in that the often 
demanded apps run by Wine are MS apps.

> Once you answer
> that question, then you can see why it is unlikely that Cygwin will ever
> have enough weight to convince MS to make our lives easier.
---
Maybe MS will try to compete with Google on 'don't be evil'?   ;-)
Hey, 'it' **could** freeze over.... 

The real problem is the 'false lure' of corrupt business models like that
used by the record companies -- I'm sure SW companies would love to be able
to keep selling the same SW and getting royalties for life+50 (or whatever
it is these days)...  Who knows -- as the government gets more corrupt and
bought out by the corps, this may come to pass...









--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why does windows have such probs with dynamically loaded libs?

[Csaba Raduly]

On Wed, Jun 1, 2011 at 2:42 AM, Linda Walsh  wrote:

>        Hmmm...I wonder...do you know if Interix setups COW pages on fork?
> If so, why in the heck would it perform so much more slowly than cygwin
> when running the same tasks (shell scripts and such that do lots of little
> forks)....  its performance was pretty bad next to cygwin, though that was
> under XP, and several years back that I tested, so it may have changed).

Last year I investigated Services For Unix on Vista and found it to be
roughly on par with Cygwin in terms if compilation time of a complex
C++ project.

> Eric Blake wrote:
>> Put yourself in Microsoft's shoes - why would you want to make it easier
>> for free software

If they wanted that, they would surely decide to enhance Windows
Services For Unix; after all they paid good money for the company that
made Interix.
But, since SFU doesn't even have a proper poll(2) implementation, I'm
not holding my breath.

Csaba
-- 
GCS a+ e++ d- C++ ULS$ L+$ !E- W++ P+++$ w++$ tv+ b++ DI D++ 5++
The Tao of math: The numbers you can count are not the real numbers.
Life is complex, with real and imaginary parts.
"Ok, it boots. Which means it must be bug-free and perfect. " -- Linus Torvalds
"People disagree with me. I just ignore them." -- Linus Torvalds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Why does windows have such probs with dynamically loaded libs?

[Linda Walsh]

Csaba Raduly wrote:
> On Wed, Jun 1, 2011 at 2:42 AM, Linda Walsh  wrote:
> 
>>        Hmmm...I wonder...do you know if Interix setups COW pages on fork?
>> If so, why in the heck would it perform so much more slowly than cygwin
>> when running the same tasks (shell scripts and such that do lots of little
>> forks)....  its performance was pretty bad next to cygwin, though that was
>> under XP, and several years back that I tested, so it may have changed).
> 
> Last year I investigated Services For Unix on Vista and found it to be
> roughly on par with Cygwin in terms if compilation time of a complex
> C++ project.
----
	But that would be more compute bound where I'd expect
them to be more equal.

  I was using a shell script that made many small util tr/cat/sed/...etc)
calls, and thus would have 'banged' on process forking, proportionately
more, than cpu-usage.  I find C and C++ compilations (even though they can
take many processes), spend most of their time in compilation, which I'd
expect to be near identical ( barring differences in compiler's used to
compile the tools...etc)....


>> Eric Blake wrote:
>>> Put yourself in Microsoft's shoes - why would you want to make it easier
>>> for free software
> 
> If they wanted that, they would surely decide to enhance Windows
> Services For Unix; after all they paid good money for the company that
> made Interix.
> But, since SFU doesn't even have a proper poll(2) implementation, I'm
> not holding my breath.
----
   Depends on the costs/benefits.  They may have paid Interix to develop
it, but I would wager it's not used nearly as much as cygwin, even in the
corporate world.  It might be easer from MS's perspective to give 
cygwin more access to docs to cygwin could better integrate, than to 
dump development money into the SFU product if MS's Corp customers are
already relying more on Cygwin.   It could easily be seen as fiscal 
mismanagement to be so wasteful and companies have been sued by stock
holders for being fiscally irresponsible.

But some stockholder with some legal-backbone would have to care enough
to push buttons and in the legal arena, comment sense and truth are
easily over-ridden by legal process and bovine-excrement.
Linda



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple