Cygport and auto-manifestize compatibility manifest

Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3405 bytes --]

Hi Yaakov,
Hi everybody else interested,


I guess you saw the discussion about GetVersionEx and Windows 8.1 on the
main list.

In the meantime I found that the missing manifest files in our
executables apparently have more implications than anticipated.

Windows 8.1's Task Manager has a new column in it's detailed view.  It's
called "Operating System Context" and it tells the user how executables
are treated by the OS in terms of compatibility.  I switched this on,
and I found that all our executables without manifests are treated as if
they are only Vista-compatible and never made a step beyond that.

This has actual implications.  For instance, up to Vista the
GetOverlappedResult function has a bug in terms of a race condition
related to the usage of the event object.  Applications which are
treated as Vista-only compatible get the Vista behaviour of
GetOverlappedResult, including the race condition!

Given that everything happening is the fault of the Cygwin DLL, not the
fault of the executable, I would like to be able to tell the OS,
whatever version it is, yes, this executable is Windows X.Y compatible.

Obviously this can only work if you build a new executable and know the
OS GUID, but still...

So I was wondering if we could, somehow, tweak cygport to add a manifest
to every created executable, along the lines of the setup manifests,
without having to mention that in the cygport file.  Fully automatic.

The manifests and, as a result, the cygport package would have to be
updated every time Microsoft releases a new OS version, but that's just
adding another GUID to the manifest.

The downside (as of the time of writing) is this:

  I did not manage to add a .rsrc section to an executable using
  objcopy, without the executable being broken afterwards.

  The *only* way I was able to add a mainfest resource to an existing
  executable was a helper application which uses the Win32 function
  UpdateResource, using a crude algorithm, here with tcsh as example:

  - Create a tcsh-manifest.rc manifest (aka "steal the setup.exe manifest).
  - Create a tcsh-manifest.o file via windres.
  - Create a dummy.exe executable with just `int main(){}' and link it
    with tcsh-manifest.o.
  - Make sure tcsh.exe does not contain any section with long sectionname
    (here: .gnu_debuglink).
  - Call the helper application, which basically works like this:

      LoadLibrary("dummy.exe");
      FindResource(CREATEPROCESS_MANIFEST_RESOURCE_ID, RT_MANIFEST);
      LoadResource(...)
      LockResource(...)
      BeginUpdateResource("tcsh.exe")
      UpdateResource(RT_MANIFEST,CREATEPROCESS_MANIFEST_RESOURCE_ID,...)
      EndUpdateResource(...)

   - The resulting executable actually worked and was treated as a
     Windows 8.1 executable.

   If this hack is done after stripping, but before adding the
   .gnu_debuglink section, it could run automatically and all newly
   produced executables would be *finally* compatible with latest
   Windows versions.

Apart from the fact that it would be nice if our linker would do this
automatically and transparently, is that something we should do in
cygport for the time being?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Charles Wilson]

On 11/20/2013 8:28 AM, Corinna Vinschen wrote:
> Apart from the fact that it would be nice if our linker would do this
> automatically and transparently,

Or libtool, if you use it to link your exe?  PTC...since $new-libtool is 
pretty high on my to-do list.

It'd be better if there was an option to ld/gcc, of course -- but the 
details would be rather complicated.  You wouldn't want to invoke a 
separate executable like windres b/c then your build recipe/makefile 
would have to change.  Best if $LD_FLAGS could be used... maybe 
something hideously ugly like -w32-manifest-compat <file> [1] where 
<file> is not a full XML manifest, but rather contains a list of GUIDs 
[2], and ld/gcc autogenerates the manifest with just that stuff.

That way, if you manually create a manifest (for other purposes), you 
could just /not/ use the new flag.

I know, SHTDI...

> is that something we should do in
> cygport for the time being?

[1] or comma-separated list of GUIDs? That'd get long and ugly, very fast.

[2] not OS names, b/c then (a) ld/gcc would have to "know" the 
corresponding GUID, and (b) the GUID<->OS database would be out of date 
and require a binutils/gcc patch every time Redmond released a new 
service pack.

--
Chuck

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

On Nov 20 09:47, Charles Wilson wrote:
> On 11/20/2013 8:28 AM, Corinna Vinschen wrote:
> >Apart from the fact that it would be nice if our linker would do this
> >automatically and transparently,
> 
> Or libtool, if you use it to link your exe?  PTC...since
> $new-libtool is pretty high on my to-do list.
> 
> It'd be better if there was an option to ld/gcc, of course -- but
> the details would be rather complicated.  You wouldn't want to
> invoke a separate executable like windres b/c then your build
> recipe/makefile would have to change.  Best if $LD_FLAGS could be
> used... maybe something hideously ugly like -w32-manifest-compat
> <file> [1] where <file> is not a full XML manifest, but rather
> contains a list of GUIDs [2], and ld/gcc autogenerates the manifest
> with just that stuff.
> 
> That way, if you manually create a manifest (for other purposes),
> you could just /not/ use the new flag.

The other way around.  If your project does not link in a resource
anyway, which means that your project is WIn32 aware, then ld should
add the manifest resource by default.  Everything else means that
you have to tweak all project configury, which is only barely descibed
by a can of worms...

A simple windres call in cygport would be nice, but unfortunately windres
does not yet have this capability to add a resource to an existing exe.

> I know, SHTDI...

Yes, that's pretty much the problem.  Even my ugly workaround is bad,
because it only works on Windows.  No more cross-building on Linux :-P


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Christopher Faylor]

On Wed, Nov 20, 2013 at 04:01:01PM +0100, Corinna Vinschen wrote:
>On Nov 20 09:47, Charles Wilson wrote:
>> On 11/20/2013 8:28 AM, Corinna Vinschen wrote:
>> >Apart from the fact that it would be nice if our linker would do this
>> >automatically and transparently,
>> 
>> Or libtool, if you use it to link your exe?  PTC...since
>> $new-libtool is pretty high on my to-do list.
>> 
>> It'd be better if there was an option to ld/gcc, of course -- but
>> the details would be rather complicated.  You wouldn't want to
>> invoke a separate executable like windres b/c then your build
>> recipe/makefile would have to change.  Best if $LD_FLAGS could be
>> used... maybe something hideously ugly like -w32-manifest-compat
>> <file> [1] where <file> is not a full XML manifest, but rather
>> contains a list of GUIDs [2], and ld/gcc autogenerates the manifest
>> with just that stuff.
>> 
>> That way, if you manually create a manifest (for other purposes),
>> you could just /not/ use the new flag.
>
>The other way around.  If your project does not link in a resource
>anyway, which means that your project is WIn32 aware, then ld should
>add the manifest resource by default.  Everything else means that
>you have to tweak all project configury, which is only barely descibed
>by a can of worms...
>
>A simple windres call in cygport would be nice, but unfortunately windres
>does not yet have this capability to add a resource to an existing exe.
>
>> I know, SHTDI...
>
>Yes, that's pretty much the problem.  Even my ugly workaround is bad,
>because it only works on Windows.  No more cross-building on Linux :-P

Some more ugly hacks: Could cygwin itself create the manifests when it
runs a program iff they don't exist?  Or, alternately, setup.exe?  Or
rebase?

cgf

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1.1: Type: text/plain, Size: 3334 bytes --]

On Nov 20 16:01, Corinna Vinschen wrote:
> On Nov 20 09:47, Charles Wilson wrote:
> > On 11/20/2013 8:28 AM, Corinna Vinschen wrote:
> > >Apart from the fact that it would be nice if our linker would do this
> > >automatically and transparently,
> > 
> > Or libtool, if you use it to link your exe?  PTC...since
> > $new-libtool is pretty high on my to-do list.
> > 
> > It'd be better if there was an option to ld/gcc, of course -- but
> > the details would be rather complicated.  You wouldn't want to
> > invoke a separate executable like windres b/c then your build
> > recipe/makefile would have to change.  Best if $LD_FLAGS could be
> > used... maybe something hideously ugly like -w32-manifest-compat
> > <file> [1] where <file> is not a full XML manifest, but rather
> > contains a list of GUIDs [2], and ld/gcc autogenerates the manifest
> > with just that stuff.
> > 
> > That way, if you manually create a manifest (for other purposes),
> > you could just /not/ use the new flag.
> 
> The other way around.  If your project does not link in a resource
> anyway, which means that your project is WIn32 aware, then ld should
> add the manifest resource by default.  Everything else means that
> you have to tweak all project configury, which is only barely descibed
> by a can of worms...
> 
> A simple windres call in cygport would be nice, but unfortunately windres
> does not yet have this capability to add a resource to an existing exe.
> 
> > I know, SHTDI...
> 
> Yes, that's pretty much the problem.  Even my ugly workaround is bad,
> because it only works on Windows.  No more cross-building on Linux :-P

I added the sources for my now radically simplified test to this mail.
It consists of three files:

  add-cygwin-default-manifest.c:  The source code adding the manifest.
  cygwin-default-mainfest.rc:     The default resource containing the
                                  manifest.  It's target-agnostic.
  Makefile:                       The Makefile.

To build the executable, just run `make'.  This creates the
add-cygwin-default-manifest executable with the embedded manifest.

To add a manifest to binary "foo.exe", run `make doit EXE=foo.exe'
add-cygwin-default-manifest.exe pulls its own manifest and copies it
over to foo.exe.  This is the step which I didn't manage to reproduce
via objcopy.

Note the crude way how `make doit' works.  The UpdateResource
functionality is *extremly* sensitive against having an open handle to
the file still open in the same binary somewhere.  If you try something
like this:

  doit:
  	strip $(EXE)
	objcopy --remove-section=.gnu_debuglink $(EXE)
	./add-cygwin-default-manifest $(EXE)

then you'll get an error message "EndUpdateResource: Win32 error 5".
Apparently, `make' has an open handle to $(EXE) and this is propagated
somehow to the child processes.  As a side note, trying to call, say,
GetBinaryType before calling BeginUpdateResource, or even stuff like
CreateFile/CloseHandle(!!!) will end up with "EndUpdateResource: Win32
error 5".  As I wrote, *extremly* sensitive.  That sounds much better 
than "badly written"...


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #1.2: add-cygwin-default-manifest.c --]
[-- Type: text/plain, Size: 1990 bytes --]

#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <locale.h>
#include <sys/cygwin.h>

int
main (int argc, char **argv)
{
  PWCHAR wpath;
  HGLOBAL hResLoad;   // handle to loaded resource
  HMODULE hExe;       // handle to existing .EXE file
  HRSRC hRes;         // handle/ptr. to res. info. in hExe
  LPVOID lpResLock;   // pointer to resource data
  HANDLE h;

  setlocale (LC_ALL, "");

  if (argc != 2)
    {
      fprintf (stderr, "Usage: %s file\n", argv[0]);
      return 1;
    }

  wpath = cygwin_create_path (CCP_POSIX_TO_WIN_W, argv[1]);
  if (!wpath)
    {
      fprintf (stderr, "cygwin_create_path: %s\n", strerror (errno));
      return 1;
    }

  hExe = GetModuleHandle (NULL);
  hRes = FindResourceW(hExe,
		       MAKEINTRESOURCEW (CREATEPROCESS_MANIFEST_RESOURCE_ID),
		       MAKEINTRESOURCEW (RT_MANIFEST));
  if (!hRes)
    {
      fprintf (stderr, "FindResourceW: Win32 error %d\n", GetLastError ());
      return 1;
    }
  hResLoad = LoadResource(hExe, hRes);
  if (!hResLoad)
    {
      fprintf (stderr, "LoadResource: Win32 error %d\n", GetLastError ());
      return 1;
    }
  lpResLock = LockResource(hResLoad);
  if (!lpResLock)
    {
      fprintf (stderr, "LockResource: Win32 error %d\n", GetLastError ());
      return 1;
    }

  h = BeginUpdateResourceW (wpath, FALSE);
  if (!h)
    {
      fprintf (stderr, "BeginUpdateResourceW: Win32 error %d\n",
	       GetLastError ());
      return 1;
    }

  if (!UpdateResourceW (h, MAKEINTRESOURCEW (RT_MANIFEST),
			MAKEINTRESOURCEW (CREATEPROCESS_MANIFEST_RESOURCE_ID),
			MAKELANGID(LANG_NEUTRAL, SUBLANG_NEUTRAL),
			lpResLock, SizeofResource(hExe, hRes)))
    {
      fprintf (stderr, "UpdateResourceW: Win32 error %d\n", GetLastError ());
      return 1;
    }

  if (!EndUpdateResource (h, FALSE))
    {
      fprintf (stderr, "EndUpdateResource: Win32 error %d\n", GetLastError ());
      return 1;
    }
  return 0;
}

[-- Attachment #1.3: cygwin-default-manifest.rc --]
[-- Type: text/plain, Size: 1324 bytes --]

#include <windows.h>

LANGUAGE 0, 0

CREATEPROCESS_MANIFEST_RESOURCE_ID RT_MANIFEST MOVEABLE PURE DISCARDABLE
BEGIN
  "<?xml version=""1.0"" encoding=""UTF-8"" standalone=""yes""?>\n"
  "<assembly xmlns=""urn:schemas-microsoft-com:asm.v1"" manifestVersion=""1.0"">\n"
  "  <trustInfo xmlns=""urn:schemas-microsoft-com:asm.v3"">\n"
  "    <security>\n"
  "      <requestedPrivileges>\n"
  "        <requestedExecutionLevel level=""asInvoker"" uiAccess=""false""/>\n"
  "      </requestedPrivileges>\n"
  "    </security>\n"
  "  </trustInfo>\n"
  "  <compatibility xmlns=""urn:schemas-microsoft-com:compatibility.v1"">\n"
  "    <application>\n"
  "      <!--The ID below indicates application support for Windows Vista -->\n"
  "      <supportedOS Id=""{e2011457-1546-43c5-a5fe-008deee3d3f0}""/>\n"
  "      <!--The ID below indicates application support for Windows 7 -->\n"
  "      <supportedOS Id=""{35138b9a-5d96-4fbd-8e2d-a2440225f93a}""/>\n"
  "      <!--The ID below indicates application support for Windows 8 -->\n"
  "      <supportedOS Id=""{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}""/>\n"
  "      <!--The ID below indicates application support for Windows 8.1 -->\n"
  "      <supportedOS Id=""{1f676c76-80e1-4239-95bb-83d0f6d0da78}""/> \n"
  "    </application>\n"
  "  </compatibility>\n"
  "</assembly>\n"
END

[-- Attachment #1.4: Makefile --]
[-- Type: text/plain, Size: 408 bytes --]


OBJS=add-cygwin-default-manifest.o cygwin-default-manifest.o

all: add-cygwin-default-manifest.exe

cygwin-default-manifest.o: cygwin-default-manifest.rc
	windres -o $@ $<

add-cygwin-default-manifest.exe: $(OBJS)
	gcc $(CFLAGS) -o $@ $(OBJS)

doit:
	objcopy --remove-section=.gnu_debuglink $(EXE) $(EXE).out \
	&& ./add-cygwin-default-manifest $(EXE).out \
	&& strip $(EXE).out \
	&& mv $(EXE).out $(EXE)


[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2650 bytes --]

On Nov 20 10:43, Christopher Faylor wrote:
> On Wed, Nov 20, 2013 at 04:01:01PM +0100, Corinna Vinschen wrote:
> >On Nov 20 09:47, Charles Wilson wrote:
> >> On 11/20/2013 8:28 AM, Corinna Vinschen wrote:
> >> >Apart from the fact that it would be nice if our linker would do this
> >> >automatically and transparently,
> >> 
> >> Or libtool, if you use it to link your exe?  PTC...since
> >> $new-libtool is pretty high on my to-do list.
> >> 
> >> It'd be better if there was an option to ld/gcc, of course -- but
> >> the details would be rather complicated.  You wouldn't want to
> >> invoke a separate executable like windres b/c then your build
> >> recipe/makefile would have to change.  Best if $LD_FLAGS could be
> >> used... maybe something hideously ugly like -w32-manifest-compat
> >> <file> [1] where <file> is not a full XML manifest, but rather
> >> contains a list of GUIDs [2], and ld/gcc autogenerates the manifest
> >> with just that stuff.
> >> 
> >> That way, if you manually create a manifest (for other purposes),
> >> you could just /not/ use the new flag.
> >
> >The other way around.  If your project does not link in a resource
> >anyway, which means that your project is WIn32 aware, then ld should
> >add the manifest resource by default.  Everything else means that
> >you have to tweak all project configury, which is only barely descibed
> >by a can of worms...
> >
> >A simple windres call in cygport would be nice, but unfortunately windres
> >does not yet have this capability to add a resource to an existing exe.
> >
> >> I know, SHTDI...
> >
> >Yes, that's pretty much the problem.  Even my ugly workaround is bad,
> >because it only works on Windows.  No more cross-building on Linux :-P
> 
> Some more ugly hacks: Could cygwin itself create the manifests when it
> runs a program iff they don't exist?  Or, alternately, setup.exe?  Or
> rebase?

Cygwin itself, never I guess.  See my other mail I just sent.
As I said, *extremly* sensitive.  Rinse and repeat.

Setup could do it, but the problem is the existence of sections with
long section names, especially debug sections and the .gnu_debuglink
section.  It's quite the mess and rather complicated to perform since
you need to be sure to filter out the problematic sections, or better,
to perform the action *after* strip, and *before* adding the
.gnu_debuginfo section.

rebase is the wrong tool for this.  It handles DLLs, not executables.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2413 bytes --]

On Nov 20 17:02, Corinna Vinschen wrote:
> I added the sources for my now radically simplified test to this mail.
> It consists of three files:
> 
>   add-cygwin-default-manifest.c:  The source code adding the manifest.
>   cygwin-default-mainfest.rc:     The default resource containing the
>                                   manifest.  It's target-agnostic.
>   Makefile:                       The Makefile.
> 
> To build the executable, just run `make'.  This creates the
> add-cygwin-default-manifest executable with the embedded manifest.
> 
> To add a manifest to binary "foo.exe", run `make doit EXE=foo.exe'
> add-cygwin-default-manifest.exe pulls its own manifest and copies it
> over to foo.exe.  This is the step which I didn't manage to reproduce
> via objcopy.
> 
> Note the crude way how `make doit' works.  The UpdateResource
> functionality is *extremly* sensitive against having an open handle to
> the file still open in the same binary somewhere.  If you try something
> like this:
> 
>   doit:
>   	strip $(EXE)
> 	objcopy --remove-section=.gnu_debuglink $(EXE)
> 	./add-cygwin-default-manifest $(EXE)
> 
> then you'll get an error message "EndUpdateResource: Win32 error 5".
> Apparently, `make' has an open handle to $(EXE) and this is propagated
> somehow to the child processes.  As a side note, trying to call, say,
> GetBinaryType before calling BeginUpdateResource, or even stuff like
> CreateFile/CloseHandle(!!!) will end up with "EndUpdateResource: Win32
> error 5".  As I wrote, *extremly* sensitive.  That sounds much better 
> than "badly written"...

As an additional datapoint:  If you successfully call the UpdateResource
stuff on a file with long section names, the resulting executable will
actually run, and is cirrectly recognized as Windows 8.1 aware.

The problem is that the sections with long section names will be broken,
because the Windows functions somehow screws up the section strings.
The result is that neither objdump, nor objcopy, nor GDB recognize any
of these sections.  Calling objdump -h either prints empty section
names, or you get something like this:

  $ objdump -h ./foo
  BFD: ./foo: bad string table size 0
  objdump: ./foo: Bad value


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Achim Gratz]

Corinna Vinschen writes:
>   - Call the helper application, which basically works like this:
>
>       LoadLibrary("dummy.exe");
>       FindResource(CREATEPROCESS_MANIFEST_RESOURCE_ID, RT_MANIFEST);
>       LoadResource(...)
>       LockResource(...)
>       BeginUpdateResource("tcsh.exe")
>       UpdateResource(RT_MANIFEST,CREATEPROCESS_MANIFEST_RESOURCE_ID,...)
>       EndUpdateResource(...)
>
>    - The resulting executable actually worked and was treated as a
>      Windows 8.1 executable.
>
>    If this hack is done after stripping, but before adding the
>    .gnu_debuglink section, it could run automatically and all newly
>    produced executables would be *finally* compatible with latest
>    Windows versions.

Is that something that could be run during autorebase or bolted on to
peflags perhaps?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On Nov 20 17:26, Achim Gratz wrote:
> Corinna Vinschen writes:
> >   - Call the helper application, which basically works like this:
> >
> >       LoadLibrary("dummy.exe");
> >       FindResource(CREATEPROCESS_MANIFEST_RESOURCE_ID, RT_MANIFEST);
> >       LoadResource(...)
> >       LockResource(...)
> >       BeginUpdateResource("tcsh.exe")
> >       UpdateResource(RT_MANIFEST,CREATEPROCESS_MANIFEST_RESOURCE_ID,...)
> >       EndUpdateResource(...)
> >
> >    - The resulting executable actually worked and was treated as a
> >      Windows 8.1 executable.
> >
> >    If this hack is done after stripping, but before adding the
> >    .gnu_debuglink section, it could run automatically and all newly
> >    produced executables would be *finally* compatible with latest
> >    Windows versions.
> 
> Is that something that could be run during autorebase or bolted on to
> peflags perhaps?

Well, perhaps.  I'm just not sure it's the right thing to do it at
postinstall time.  I mean, it's not impossible, obviously, but it's
a lot of stuff per executable and running this for a few thousand .exe
files could take some time.

We would also have to make sure that the sections with long section
names are recreated after adding the .rsrc section, which is something
I don't quite see how to accomplish, right now.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Achim Gratz]

Corinna Vinschen writes:
> Well, perhaps.  I'm just not sure it's the right thing to do it at
> postinstall time.  I mean, it's not impossible, obviously, but it's
> a lot of stuff per executable and running this for a few thousand .exe
> files could take some time.

Yes, it does... but ever since I've switched to doing incremental
autorebases that time has shrunk a lot.

> We would also have to make sure that the sections with long section
> names are recreated after adding the .rsrc section, which is something
> I don't quite see how to accomplish, right now.

Hmm.  I'm out of my depths on this, but would it be possible to excise
those sections, do whatever changes are necessary to the rest of the
executable and then add them back?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

On Nov 20 18:22, Achim Gratz wrote:
> Corinna Vinschen writes:
> > Well, perhaps.  I'm just not sure it's the right thing to do it at
> > postinstall time.  I mean, it's not impossible, obviously, but it's
> > a lot of stuff per executable and running this for a few thousand .exe
> > files could take some time.
> 
> Yes, it does... but ever since I've switched to doing incremental
> autorebases that time has shrunk a lot.
> 
> > We would also have to make sure that the sections with long section
> > names are recreated after adding the .rsrc section, which is something
> > I don't quite see how to accomplish, right now.
> 
> Hmm.  I'm out of my depths on this, but would it be possible to excise
> those sections, do whatever changes are necessary to the rest of the
> executable and then add them back?

I don't know.  It's apparently more complicated than just calling
objcopy.  For instance, objcopy can export sections from a file in
whatever format you want, but it can only add back sections if they are
given as binary blobs.  If you add such a binary blob it's missing
relocation information.  Also, you have to make sure all the sections
start at the right address, thanks to the harebrained PE/COFF format.

This is apparently a big deal, otherwise it should have been no problem
to add a resource binary blob into an executable without making Windows
choke on it (ERROR_BAD_EXE_FORMAT).  Maybe I did something wrong, but I
would have no idea what option I missed.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3564 bytes --]

On Nov 20 18:32, Corinna Vinschen wrote:
> On Nov 20 18:22, Achim Gratz wrote:
> > Corinna Vinschen writes:
> > > Well, perhaps.  I'm just not sure it's the right thing to do it at
> > > postinstall time.  I mean, it's not impossible, obviously, but it's
> > > a lot of stuff per executable and running this for a few thousand .exe
> > > files could take some time.
> > 
> > Yes, it does... but ever since I've switched to doing incremental
> > autorebases that time has shrunk a lot.
> > 
> > > We would also have to make sure that the sections with long section
> > > names are recreated after adding the .rsrc section, which is something
> > > I don't quite see how to accomplish, right now.
> > 
> > Hmm.  I'm out of my depths on this, but would it be possible to excise
> > those sections, do whatever changes are necessary to the rest of the
> > executable and then add them back?
> 
> I don't know.  It's apparently more complicated than just calling
> objcopy.  For instance, objcopy can export sections from a file in
> whatever format you want, but it can only add back sections if they are
> given as binary blobs.  If you add such a binary blob it's missing
> relocation information.  Also, you have to make sure all the sections
> start at the right address, thanks to the harebrained PE/COFF format.
> 
> This is apparently a big deal, otherwise it should have been no problem
> to add a resource binary blob into an executable without making Windows
> choke on it (ERROR_BAD_EXE_FORMAT).  Maybe I did something wrong, but I
> would have no idea what option I missed.

I just made a quick test for the sake of creating a generic script to
add the manifests.

Assuming I have a file with just a single section with long section
name, the .gnu_debuglink section.  Fortunately this seems to be the
usual case.

I can extract the section content:

  $ objcopy -j .gnu_debuglink -O binary foo.exe foo.exe.gdl

Then I can drop the .gnu_debuglink from the executable.  Create a copy
of the original so as not to confuse the sensitive UpdateResource:

  $ objcopy -R .gnu_debuglink foo.exe foo.out

Then I can add the manifest:

  $ add-cygwin-default-manifest foo.out

Then it gets weird.  I'd like to create a .gnu_debuglink section
again.  The path is part of the binary foo.exe.gdl file:

  $ awk -F \\0 '{print $1;}' foo.dbgl
  foo.exe.dbg
  $ objcopy --add-gnu-debuglink=$(awk -F \\0 '{print $1;}' foo.dbgl) foo.out
  objcopy:stl7AEHg: cannot fill debug link section `foo.exe.dbg': No such file or directory

So the dbg file has to exist to be able to create a new .gnu_debuglink
section:

  $ touch $(awk -F \\0 '{print $1;}' foo.dbgl)
  $ objcopy --add-gnu-debuglink=$(awk -F \\0 '{print $1;}' foo.dbgl) foo.out

Cool.  That worked, and the resulting executable works as well.

But GDB doesn't find the symbol file anymore:

  $ gdb ./tcsh.out
  [...]
  Reading symbols from /home/corinna/manifest/tcsh.out...(no debugging symbols found)...done.
  (gdb) Quit
  $ objcopy -j .gnu_debuglink -O binary foo.out foo.out.gdl
  $ cmp foo.exe.gdl foo.out.gdl
  foo.exe.gdl foo.out.gdl differ: byte 17, line 1

Why is that?  And, if the .gnu_debuglink sections contains only a
filename and some flag value or something, why does it suddenly neglect
to search in the /usr/lib/debug directory?  How can this be fixed?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3353 bytes --]

On Nov 20 17:06, Corinna Vinschen wrote:
> On Nov 20 10:43, Christopher Faylor wrote:
> > On Wed, Nov 20, 2013 at 04:01:01PM +0100, Corinna Vinschen wrote:
> > >On Nov 20 09:47, Charles Wilson wrote:
> > >> On 11/20/2013 8:28 AM, Corinna Vinschen wrote:
> > >> >Apart from the fact that it would be nice if our linker would do this
> > >> >automatically and transparently,
> > >> 
> > >> Or libtool, if you use it to link your exe?  PTC...since
> > >> $new-libtool is pretty high on my to-do list.
> > >> 
> > >> It'd be better if there was an option to ld/gcc, of course -- but
> > >> the details would be rather complicated.  You wouldn't want to
> > >> invoke a separate executable like windres b/c then your build
> > >> recipe/makefile would have to change.  Best if $LD_FLAGS could be
> > >> used... maybe something hideously ugly like -w32-manifest-compat
> > >> <file> [1] where <file> is not a full XML manifest, but rather
> > >> contains a list of GUIDs [2], and ld/gcc autogenerates the manifest
> > >> with just that stuff.
> > >> 
> > >> That way, if you manually create a manifest (for other purposes),
> > >> you could just /not/ use the new flag.
> > >
> > >The other way around.  If your project does not link in a resource
> > >anyway, which means that your project is WIn32 aware, then ld should
> > >add the manifest resource by default.  Everything else means that
> > >you have to tweak all project configury, which is only barely descibed
> > >by a can of worms...
> > >
> > >A simple windres call in cygport would be nice, but unfortunately windres
> > >does not yet have this capability to add a resource to an existing exe.
> > >
> > >> I know, SHTDI...
> > >
> > >Yes, that's pretty much the problem.  Even my ugly workaround is bad,
> > >because it only works on Windows.  No more cross-building on Linux :-P
> > 
> > Some more ugly hacks: Could cygwin itself create the manifests when it
> > runs a program iff they don't exist?  Or, alternately, setup.exe?  Or
> > rebase?
> 
> Cygwin itself, never I guess.  See my other mail I just sent.
> As I said, *extremly* sensitive.  Rinse and repeat.
> 
> Setup could do it, but the problem is the existence of sections with
> long section names, especially debug sections and the .gnu_debuglink
> section.  It's quite the mess and rather complicated to perform since
> you need to be sure to filter out the problematic sections, or better,
> to perform the action *after* strip, and *before* adding the
> .gnu_debuginfo section.

For the records, here's the most dumb workaround:

  #!/bin/bash
  #
  # Assumes that a default manifest has been stored in /etc/defaults
  #
  cd /bin
  for i in *.exe
  do
    if [ ! -f "${i}.manifest" ]
    then
      cp /etc/defaults/default.manifest "${i}.manifest"
      # Enforce updating Activation Context Cache
      touch "${i}"
    fi
  done

This works, but it's pretty ugly.  1000 exe files, 1000 manifest files.
Hardlinks work, but are restricted to 1024 per file.  And it doesn't
actually make the output of ls /bin better.  Still, if we don't find a
better solution, this might be the way to go.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Peter Rosin]

On 2013-11-20 17:06, Corinna Vinschen wrote:
> On Nov 20 10:43, Christopher Faylor wrote:
>> On Wed, Nov 20, 2013 at 04:01:01PM +0100, Corinna Vinschen wrote:
>>> On Nov 20 09:47, Charles Wilson wrote:
>>>> I know, SHTDI...
>>>
>>> Yes, that's pretty much the problem.  Even my ugly workaround is bad,
>>> because it only works on Windows.  No more cross-building on Linux :-P
>>
>> Some more ugly hacks: Could cygwin itself create the manifests when it
>> runs a program iff they don't exist?  Or, alternately, setup.exe?  Or
>> rebase?
> 
> Cygwin itself, never I guess.  See my other mail I just sent.
> As I said, *extremly* sensitive.  Rinse and repeat.
> 
> Setup could do it, but the problem is the existence of sections with
> long section names, especially debug sections and the .gnu_debuglink
> section.  It's quite the mess and rather complicated to perform since
> you need to be sure to filter out the problematic sections, or better,
> to perform the action *after* strip, and *before* adding the
> .gnu_debuginfo section.

One snag with having setup do it, is that package maintainers running
'make check' will not test what is actually deployed (unless jumping
through hoops).

The behavior of the exes did change with the manifest after all, which
was the whole point with all this manifest crap, right?

Cheers,
Peter

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2449 bytes --]

On Nov 21 11:58, Peter Rosin wrote:
> On 2013-11-20 17:06, Corinna Vinschen wrote:
> > On Nov 20 10:43, Christopher Faylor wrote:
> >> On Wed, Nov 20, 2013 at 04:01:01PM +0100, Corinna Vinschen wrote:
> >>> On Nov 20 09:47, Charles Wilson wrote:
> >>>> I know, SHTDI...
> >>>
> >>> Yes, that's pretty much the problem.  Even my ugly workaround is bad,
> >>> because it only works on Windows.  No more cross-building on Linux :-P
> >>
> >> Some more ugly hacks: Could cygwin itself create the manifests when it
> >> runs a program iff they don't exist?  Or, alternately, setup.exe?  Or
> >> rebase?
> > 
> > Cygwin itself, never I guess.  See my other mail I just sent.
> > As I said, *extremly* sensitive.  Rinse and repeat.
> > 
> > Setup could do it, but the problem is the existence of sections with
> > long section names, especially debug sections and the .gnu_debuglink
> > section.  It's quite the mess and rather complicated to perform since
> > you need to be sure to filter out the problematic sections, or better,
> > to perform the action *after* strip, and *before* adding the
> > .gnu_debuginfo section.
> 
> One snag with having setup do it, is that package maintainers running
> 'make check' will not test what is actually deployed (unless jumping
> through hoops).
> 
> The behavior of the exes did change with the manifest after all, which
> was the whole point with all this manifest crap, right?

Absolutely.

But tweaking the existing executables is just some desperate short-term
method.  What we really need is a linker which adds a default manifest
if the project does not provide its own.

I talked to collegues, and we're going to work on a patch to ld at some
point not too far in the distance.  For the time being, we created a
workaround for ld using just a an additional cygwin-default-manifest.o
file in /usr/$(target)/lib, and a linker script tweak:

@@ -143,7 +143,8 @@ SECTIONS
   }
   .rsrc BLOCK(__section_alignment__) :
   {
-    *(.rsrc)
+    *(EXCLUDE_FILE (cygwin-default-manifest.o) .rsrc)
+    KEEP (cygwin-default-manifest.o(.rsrc))
     *(SORT(.rsrc$*))
   }
   .reloc BLOCK(__section_alignment__) :

Chris, any problem to release a new binutils with this local patch,
for now?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Christopher Faylor]

On Thu, Nov 21, 2013 at 12:48:23PM +0100, Corinna Vinschen wrote:
>On Nov 21 11:58, Peter Rosin wrote:
>> On 2013-11-20 17:06, Corinna Vinschen wrote:
>> > On Nov 20 10:43, Christopher Faylor wrote:
>> >> On Wed, Nov 20, 2013 at 04:01:01PM +0100, Corinna Vinschen wrote:
>> >>> On Nov 20 09:47, Charles Wilson wrote:
>> >>>> I know, SHTDI...
>> >>>
>> >>> Yes, that's pretty much the problem.  Even my ugly workaround is bad,
>> >>> because it only works on Windows.  No more cross-building on Linux :-P
>> >>
>> >> Some more ugly hacks: Could cygwin itself create the manifests when it
>> >> runs a program iff they don't exist?  Or, alternately, setup.exe?  Or
>> >> rebase?
>> > 
>> > Cygwin itself, never I guess.  See my other mail I just sent.
>> > As I said, *extremly* sensitive.  Rinse and repeat.
>> > 
>> > Setup could do it, but the problem is the existence of sections with
>> > long section names, especially debug sections and the .gnu_debuglink
>> > section.  It's quite the mess and rather complicated to perform since
>> > you need to be sure to filter out the problematic sections, or better,
>> > to perform the action *after* strip, and *before* adding the
>> > .gnu_debuginfo section.
>> 
>> One snag with having setup do it, is that package maintainers running
>> 'make check' will not test what is actually deployed (unless jumping
>> through hoops).
>> 
>> The behavior of the exes did change with the manifest after all, which
>> was the whole point with all this manifest crap, right?
>
>Absolutely.
>
>But tweaking the existing executables is just some desperate short-term
>method.  What we really need is a linker which adds a default manifest
>if the project does not provide its own.
>
>I talked to collegues, and we're going to work on a patch to ld at some
>point not too far in the distance.  For the time being, we created a
>workaround for ld using just a an additional cygwin-default-manifest.o
>file in /usr/$(target)/lib, and a linker script tweak:
>
>@@ -143,7 +143,8 @@ SECTIONS
>   }
>   .rsrc BLOCK(__section_alignment__) :
>   {
>-    *(.rsrc)
>+    *(EXCLUDE_FILE (cygwin-default-manifest.o) .rsrc)
>+    KEEP (cygwin-default-manifest.o(.rsrc))
>     *(SORT(.rsrc$*))
>   }
>   .reloc BLOCK(__section_alignment__) :

Shouldn't the KEEP line be after the SORT line?

If you send me a patch, I'll release new versions of binutils.

cgf

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2182 bytes --]

On Nov 21 09:27, Christopher Faylor wrote:
> On Thu, Nov 21, 2013 at 12:48:23PM +0100, Corinna Vinschen wrote:
> >I talked to collegues, and we're going to work on a patch to ld at some
> >point not too far in the distance.  For the time being, we created a
> >workaround for ld using just a an additional cygwin-default-manifest.o
> >file in /usr/$(target)/lib, and a linker script tweak:
> >
> >@@ -143,7 +143,8 @@ SECTIONS
> >   }
> >   .rsrc BLOCK(__section_alignment__) :
> >   {
> >-    *(.rsrc)
> >+    *(EXCLUDE_FILE (cygwin-default-manifest.o) .rsrc)
> >+    KEEP (cygwin-default-manifest.o(.rsrc))
> >     *(SORT(.rsrc$*))
> >   }
> >   .reloc BLOCK(__section_alignment__) :
> 
> Shouldn't the KEEP line be after the SORT line?

I don't know.  Right now this is probably entirely moot, because ld does
not correctly merge .rsrc sections anyway.  The result will be, that
Windows will only recognize the first .rsrc section added to the
executable and then ignore anything beyond that because the rsrc header
will be incorrectly refer to only this first .rsrc blob.  That's what
has to be fixed in ld.

For the time being, what happens with the above hack^Wpatch is the
following:

- If the project does not provide its own resources, the cygwin default
  manifest resource will be linked into the executable and recognized by
  Windows.

- If the project provides its own resources, they will be linked into
  the executable first, followed by the cygwin default manifest resource.
  Since the rsrc's are not correctly merged, only the project's own
  resources will be found and the cygwin default manifest resource will
  be hidden.

> If you send me a patch, I'll release new versions of binutils.

Oh, that's what I not have so far.  It requires to add the
cygwin-default-manifest.rc file to binutils, and some changes to the
Makefiles to build and install cygwin-default-manifest.o.  I don't know
how long this takes, I think either tomorrow or next Monday.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2675 bytes --]

On Nov 21 16:30, Corinna Vinschen wrote:
> On Nov 21 09:27, Christopher Faylor wrote:
> > On Thu, Nov 21, 2013 at 12:48:23PM +0100, Corinna Vinschen wrote:
> > >I talked to collegues, and we're going to work on a patch to ld at some
> > >point not too far in the distance.  For the time being, we created a
> > >workaround for ld using just a an additional cygwin-default-manifest.o
> > >file in /usr/$(target)/lib, and a linker script tweak:
> > >
> > >@@ -143,7 +143,8 @@ SECTIONS
> > >   }
> > >   .rsrc BLOCK(__section_alignment__) :
> > >   {
> > >-    *(.rsrc)
> > >+    *(EXCLUDE_FILE (cygwin-default-manifest.o) .rsrc)
> > >+    KEEP (cygwin-default-manifest.o(.rsrc))
> > >     *(SORT(.rsrc$*))
> > >   }
> > >   .reloc BLOCK(__section_alignment__) :
> > 
> > Shouldn't the KEEP line be after the SORT line?
> 
> I don't know.  Right now this is probably entirely moot, because ld does
> not correctly merge .rsrc sections anyway.  The result will be, that
> Windows will only recognize the first .rsrc section added to the
> executable and then ignore anything beyond that because the rsrc header
> will be incorrectly refer to only this first .rsrc blob.  That's what
> has to be fixed in ld.
> 
> For the time being, what happens with the above hack^Wpatch is the
> following:
> 
> - If the project does not provide its own resources, the cygwin default
>   manifest resource will be linked into the executable and recognized by
>   Windows.
> 
> - If the project provides its own resources, they will be linked into
>   the executable first, followed by the cygwin default manifest resource.
>   Since the rsrc's are not correctly merged, only the project's own
>   resources will be found and the cygwin default manifest resource will
>   be hidden.
> 
> > If you send me a patch, I'll release new versions of binutils.
> 
> Oh, that's what I not have so far.  It requires to add the
> cygwin-default-manifest.rc file to binutils, and some changes to the
> Makefiles to build and install cygwin-default-manifest.o.  I don't know
> how long this takes, I think either tomorrow or next Monday.

Oh, btw.  While we're at it, do we want the default manifest to add an
requestedExecutionLevel of "asInvoker"?  This is in theory what we want
all executables to be anyway, and it would once and for all drop the
requirement to add side-by-side manifests to the 32 bit executables
called "install", "install-info", "patch", "update-mime-database", etc.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Cygport and auto-manifestize compatibility manifest

[Corinna Vinschen]

[-- Attachment #1.1: Type: text/plain, Size: 2951 bytes --]

On Nov 20 19:14, Corinna Vinschen wrote:
> On Nov 20 18:32, Corinna Vinschen wrote:
> > On Nov 20 18:22, Achim Gratz wrote:
> > > Corinna Vinschen writes:
> > > > Well, perhaps.  I'm just not sure it's the right thing to do it at
> > > > postinstall time.  I mean, it's not impossible, obviously, but it's
> > > > a lot of stuff per executable and running this for a few thousand .exe
> > > > files could take some time.
> > > 
> > > Yes, it does... but ever since I've switched to doing incremental
> > > autorebases that time has shrunk a lot.
> > > 
> > > > We would also have to make sure that the sections with long section
> > > > names are recreated after adding the .rsrc section, which is something
> > > > I don't quite see how to accomplish, right now.
> > > 
> > > Hmm.  I'm out of my depths on this, but would it be possible to excise
> > > those sections, do whatever changes are necessary to the rest of the
> > > executable and then add them back?
> > 
> > I don't know.  It's apparently more complicated than just calling
> > objcopy.  For instance, objcopy can export sections from a file in
> > whatever format you want, but it can only add back sections if they are
> > given as binary blobs.  If you add such a binary blob it's missing
> > relocation information.  Also, you have to make sure all the sections
> > start at the right address, thanks to the harebrained PE/COFF format.
> > 
> > This is apparently a big deal, otherwise it should have been no problem
> > to add a resource binary blob into an executable without making Windows
> > choke on it (ERROR_BAD_EXE_FORMAT).  Maybe I did something wrong, but I
> > would have no idea what option I missed.
> 
> I just made a quick test for the sake of creating a generic script to
> add the manifests.
> [...]

For the records and to all interested in this.  I have a script now
which works, provided the executables are stripped and the
.gnu_debuglink section is the only section with a long section name.

The sources of the script called "add-mani.sh" and the sources of
the Windows tool necessary to update the manifest are attached.

The downside:

If you add a manifest to, say, tcsh, the tcsh.exe.dbg file from the
tcsh-debuginfo package will not have the .rsrc section, and due to the
Dwarf2 debug sections you will not be able to add the .rsrc section
successfully *and* keep the tcsh.exe.dbg file debuggable.  This in turn
results in a warning in GDB:

  warning: section .rsrc not found in /usr/lib/debug/bin/tcsh.exe.dbg

The tcsh binary is still debuggable, apparently, but still...

Therefore I think this script is only a temporary measure, until the
time the binutils package allows to accomplish this by itself.  We're
working on that.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #1.2: manifestize.tar.xz --]
[-- Type: application/x-xz, Size: 2176 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]