[RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Pierre Muller]

Following this thread
https://sourceware.org/ml/gdb-patches/2013-12/msg00073.html

The patch
https://sourceware.org/ml/gdb-cvs/2013-12/msg00054.html

introduced a failure for cygwin native build.
The problem is that __USEWIDE is not considered in the patch.

The patch below fixes this compilation error
and should allow cygwin to work as mingw.

Pierre Muller


ChangeLog entry:

2013-12-13  Pierre Muller  <muller@sourceware.org>

	Fix compilation error for cygwin native build.
	* windows-nat.c (windows_ensure_ntdll_loaded): Add
	code using wchar_t type and conversion to char string
	if __USEWIDE is defined.


diff --git a/gdb/windows-nat.c b/gdb/windows-nat.c
index f0545fc..dda9d8e 100644
--- a/gdb/windows-nat.c
+++ b/gdb/windows-nat.c
@@ -1764,17 +1764,27 @@ windows_ensure_ntdll_loaded (void)
   for (i = 0; i < (int) (cb_needed / sizeof (HMODULE)); i++)
     {
       MODULEINFO mi;
+#ifdef __USEWIDE
+      wchar_t dll_name[__PMAX];
+      char name[__PMAX];
+#else
       char dll_name[__PMAX];
-
+      char *name;
+#endif
       if (GetModuleInformation (current_process_handle, hmodules[i],
                                &mi, sizeof (mi)) == 0)
        continue;
       if (GetModuleFileNameEx (current_process_handle, hmodules[i],
                               dll_name, sizeof (dll_name)) == 0)
        continue;
-      if (FILENAME_CMP (lbasename (dll_name), "ntdll.dll") == 0)
+#ifdef __USEWIDE
+      wcstombs (name, dll_name, __PMAX);
+#else
+      name = dll_name;
+#endif
+      if (FILENAME_CMP (lbasename (name), "ntdll.dll") == 0)
        {
-         solib_end->next = windows_make_so (dll_name, mi.lpBaseOfDll);
+         solib_end->next = windows_make_so (name, mi.lpBaseOfDll);
          solib_end = solib_end->next;
          return;
        }

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Yao Qi]

On 12/14/2013 06:37 AM, Pierre Muller wrote:
> introduced a failure for cygwin native build.
> The problem is that __USEWIDE is not considered in the patch.
> 

For the record, the build failure is this:

cc1: warnings being treated as errors
../../git/gdb/windows-nat.c: In function ‘windows_ensure_ntdll_loaded’:
../../git/gdb/windows-nat.c:1773:11: error: passing argument 3 of ‘GetModuleFileNameEx’ from incompatible pointer type
../../git/gdb/windows-nat.c:1773:11: note: expected ‘LPWSTR’ but argument is of type ‘char *’
Makefile:998: recipe for target `windows-nat.o' failed

> The patch below fixes this compilation error
> and should allow cygwin to work as mingw.
> 
> Pierre Muller
> 
> 
> ChangeLog entry:
> 
> 2013-12-13  Pierre Muller  <muller@sourceware.org>
> 
> 	Fix compilation error for cygwin native build.
> 	* windows-nat.c (windows_ensure_ntdll_loaded): Add
> 	code using wchar_t type and conversion to char string
> 	if __USEWIDE is defined.

Probably, we can write

	(windows_ensure_ntdll_loaded) [__USEWIDE]: Call
	wcstombs.

> 
> 
> diff --git a/gdb/windows-nat.c b/gdb/windows-nat.c
> index f0545fc..dda9d8e 100644
> --- a/gdb/windows-nat.c
> +++ b/gdb/windows-nat.c
> @@ -1764,17 +1764,27 @@ windows_ensure_ntdll_loaded (void)
>    for (i = 0; i < (int) (cb_needed / sizeof (HMODULE)); i++)
>      {
>        MODULEINFO mi;
> +#ifdef __USEWIDE
> +      wchar_t dll_name[__PMAX];
> +      char name[__PMAX];
> +#else
>        char dll_name[__PMAX];
> -
> +      char *name;
> +#endif
>        if (GetModuleInformation (current_process_handle, hmodules[i],
>                                 &mi, sizeof (mi)) == 0)
>         continue;
>        if (GetModuleFileNameEx (current_process_handle, hmodules[i],
>                                dll_name, sizeof (dll_name)) == 0)
>         continue;
> -      if (FILENAME_CMP (lbasename (dll_name), "ntdll.dll") == 0)
> +#ifdef __USEWIDE
> +      wcstombs (name, dll_name, __PMAX);
> +#else
> +      name = dll_name;
> +#endif
> +      if (FILENAME_CMP (lbasename (name), "ntdll.dll") == 0)
>         {
> -         solib_end->next = windows_make_so (dll_name, mi.lpBaseOfDll);
> +         solib_end->next = windows_make_so (name, mi.lpBaseOfDll);
>           solib_end = solib_end->next;
>           return;
>         }

GetModuleFileNameEx is also used in get_module_name, so probably we can
copy its pattern here too

#ifdef __CYGWIN__
  len = GetModuleFileNameEx (current_process_handle, DllHandle,
			     pathbuf, __PMAX);
  if (cygwin_conv_path (CCP_WIN_W_TO_POSIX, pathbuf, dll_name_ret,
			__PMAX) < 0)
     ...
#else
  len = GetModuleFileNameEx (current_process_handle,
			     DllHandle, dll_name_ret, __PMAX);
#endif

and code is more consistent on using GetModuleFileNameEx.
-- 
Yao (齐尧)

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Pedro Alves]

On 12/16/2013 02:19 AM, Yao Qi wrote:

> GetModuleFileNameEx is also used in get_module_name, so probably we can
> copy its pattern here too
> 
> #ifdef __CYGWIN__
>   len = GetModuleFileNameEx (current_process_handle, DllHandle,
> 			     pathbuf, __PMAX);
>   if (cygwin_conv_path (CCP_WIN_W_TO_POSIX, pathbuf, dll_name_ret,
> 			__PMAX) < 0)
>      ...
> #else
>   len = GetModuleFileNameEx (current_process_handle,
> 			     DllHandle, dll_name_ret, __PMAX);
> #endif
> 
> and code is more consistent on using GetModuleFileNameEx.

At this level we're thinking in terms of Win32 debug API, and
it's irrelevant what Cygwin thinks is the Posix path of
ntdll.dll.  If this is indeed ntdll.dll, then the following
windows_make_so call will then do that conversion before
recording the dll name.

-- 
Pedro Alves

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Yao Qi]

On 12/17/2013 02:05 AM, Pedro Alves wrote:
> At this level we're thinking in terms of Win32 debug API, and
> it's irrelevant what Cygwin thinks is the Posix path of
> ntdll.dll.  If this is indeed ntdll.dll, then the following
> windows_make_so call will then do that conversion before
> recording the dll name.

My point is we use cygwin_conv_path to convert the path to type "char *"
and then compare whether it is ntdll.dll.

-- 
Yao (齐尧)

RE: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Pierre Muller]

> -----Message d'origine-----
> De : gdb-patches-owner@sourceware.org [mailto:gdb-patches-
> owner@sourceware.org] De la part de Yao Qi
> Envoyé : mardi 17 décembre 2013 01:41
> À : Pedro Alves
> Cc : Pierre Muller; gdb-patches@sourceware.org
> Objet : Re: [RFA] Fix cygwin compilation failure due to nameless
> LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing
> 
> On 12/17/2013 02:05 AM, Pedro Alves wrote:
> > At this level we're thinking in terms of Win32 debug API, and
> > it's irrelevant what Cygwin thinks is the Posix path of
> > ntdll.dll.  If this is indeed ntdll.dll, then the following
> > windows_make_so call will then do that conversion before
> > recording the dll name.
> 
> My point is we use cygwin_conv_path to convert the path to type "char
> *"
> and then compare whether it is ntdll.dll.

  As Pedro explained, I don't think this is useful here,
because we only check the filename part of the DLL,
this is not changed by the cygwin_conv_path which only converts
Windows OS paths into cygwin paths.
  Moreover, windows_make_so expect a windows OS path,
which is copied into so_original_name
while so_name gets the return value of cygwin_conv_path.

  Thus I would expect that your idea to call cygwin_conv_path 
inside windows_ensure_ntddl_loaded would lead to a
failure at the start of the windows_make_so which calls the
Win32 API function FindFirstFile.

  I hope this clarifies the point.

Pierre Muller

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Yao Qi]

On 12/14/2013 06:37 AM, Pierre Muller wrote:
> Following this thread
> https://sourceware.org/ml/gdb-patches/2013-12/msg00073.html
> 
> The patch
> https://sourceware.org/ml/gdb-cvs/2013-12/msg00054.html
> 
> introduced a failure for cygwin native build.
> The problem is that __USEWIDE is not considered in the patch.
> 
> The patch below fixes this compilation error
> and should allow cygwin to work as mingw.

Subject: [PATCH] Fix the format of one ChangeLog entry

I notice that two lines of a recent changelog entry are not prefixed
with tab.  They are prefixed with a space and a tab.  This patch
is to remove the spaces.  Pushed.

-- 
Yao (齐尧)

gdb:

2013-12-18  Yao Qi  <yao@codesourcery.com>

	* ChangeLog: Fix the format of one entry.
---
 gdb/ChangeLog |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index e9cb6ca..6e67bac 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -51,8 +51,8 @@
 
 2013-12-16  Pierre Muller  <muller@sourceware.org>
 
- 	Fix compilation error for cygwin native build.
- 	* windows-nat.c (windows_ensure_ntdll_loaded) [__USEWIDE]:
+	Fix compilation error for cygwin native build.
+	* windows-nat.c (windows_ensure_ntdll_loaded) [__USEWIDE]:
 	Call wcstombs.
 
 2013-12-16  Pedro Alves  <palves@redhat.com>
-- 
1.7.7.6

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Eli Zaretskii]

> Date: Sat, 28 Dec 2013 07:17:19 +0400
> From: Joel Brobecker <brobecker@adacore.com>
> Cc: gdb-patches@sourceware.org
> 
> I would hate to see it all lost in the mailinglist archives, though.
> How about we turn this into either a comment in the code (my prefered
> option, most of the time), or a gdbint update (on the wiki), or even
> just a wiki page somewhere. We could open, if not already done,
> a section dealing with Windows-specific issues.

I can do that, please point me to the relevant places.

Btw, there's one other quite significant nuisance with using file
names that breach the 260-character limits: you can do almost nothing
with them, once they are created, except if you write your own
programs to deal with them.  It looks like none of the programs and
commands that come with Windows out of the box are able to handle such
file names.  You cannot delete them, neither from the Explorer nor by
using DEL from the shell prompt.  The Explorer silently and
inexplicably fails to copy directories which include such files.  So
you are basically stuck with them, unless you write your own programs
to delete/copy/rename them.  (I didn't try Cygwin: perhaps its file
utilities can also be a way out.)

So: children, don't try that at home!

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Joel Brobecker]

> I took some time to test about a dozen of
> the more popular CRT functions that deal with file names and accept
> wchar_t arguments.  Most of them worked with long file names in the
> \\?\ format, but some didn't.

thanks for the thorough research, Eli; that's a pretty useful amount
of data for anyone wanting to tackle these kinds of issues.

I would hate to see it all lost in the mailinglist archives, though.
How about we turn this into either a comment in the code (my prefered
option, most of the time), or a gdbint update (on the wiki), or even
just a wiki page somewhere. We could open, if not already done,
a section dealing with Windows-specific issues.

> 
> First, _wchdir failed consistently with file names longer than 260.
> Which might not be a surprise, since SetCurrentDirectoryW is
> documented to be limited to MAX_PATH characters.  But it does mean
> that any application which wishes to lift the 260-char limitation will
> need to replace chdir and getcwd (and DTRT for each thread).
> 
> Next, _wstat fails for any file name in the \\?\ form, even if it
> doesn't exceed the 260-character limit.  By contrast, FindFirstFileW,
> on which AFAIK the MS implementation of 'stat' is based, has no
> problem with such file names, and works with filenames up to 32K
> characters.
> 
> On Windows XP (but not on Windows 7), _wfullpath also failed for long
> file names.
> 
> The upshot of all this is that it's not enough to just switch to
> wchar_t and \\?\ form of file names, there's some non-trivial support
> code that will need to be written to take advantage of that in any
> sufficiently large application (such as GDB).
> 
> Moreover, even just using Unicode APIs is of questionable value in GDB
> (as in any other console program on Windows), because:
> 
>   . supporting Unicode on the console will need very serious changes
>     in Readline and in ui-*.c modules; without such changes, one can
>     never enter or display file names outside of the current ANSI
>     codepage
> 
>   . the Windows console has very weak support for non-ASCII fonts even
>     when Unicode APIs are used to read and write to the console
>     (basically, anything outside of Europe cannot be displayed)
> 
>   . any external libraries against which GDB is linked are likely not
>     to support Unicode file names or UTF-16 text, so you cannot easily
>     communicate with them without losing Unicode capabilities
> 
>   . suppose you get from a Unicode API, such as GetModuleFileNameW, a
>     file name that cannot be expressed in the current ANSI codepage --
>     what can you do with such a file name in GDB? probably nothing
>     except displaying an error message
> 
> So I think the benefits of switching the MinGW GDB to Unicode APIs are
> questionable, although if someone steps forward and does a clean job
> for that, I'm quite sure the patches will be more than welcome.

-- 
Joel

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Eli Zaretskii]

> Date: Wed, 18 Dec 2013 21:54:16 +0100
> From: Corinna Vinschen <vinschen@redhat.com>
> 
> > > There's a lot of theory here.  Did you actually *try* it?
> > 
> > No.  Who can afford to try everything, unless they are paying you to
> > do that?
> 
> Well, it might not take much more time than musings in email...
> 
> > > I just did:
> > > 
> > > ===========================================
> > > #include <stdio.h>
> > > #include <errno.h>
> > > 
> > > int main(void)
> > > {
> > >   FILE *fp;
> > > 
> > >   fp = _wfopen (L"\\\\?\\C:\\Windows\\System32\\ntdll.dll", L"r");
> > >   if (!fp)
> > >     printf ("_wfopen w/ long pathname fails, errno = %d\n", errno);
> > >   else
> > >     {
> > >       printf ("_wfopen w/ long pathname works\n");
> > >       fclose(fp);
> > >     }
> > >    return 0;
> > > }
> > > ===========================================
> > 
> > Thanks, that's good to know.  But that's just one CRT function.  There
> > are a gazillion of them, and I'm quite sure some use MAX_PATH of 260
> > internally.  Just grep the sources of the CRT functions.
> 
> Sigh.

Since you were that unhappy, I took some time to test about a dozen of
the more popular CRT functions that deal with file names and accept
wchar_t arguments.  Most of them worked with long file names in the
\\?\ format, but some didn't.

First, _wchdir failed consistently with file names longer than 260.
Which might not be a surprise, since SetCurrentDirectoryW is
documented to be limited to MAX_PATH characters.  But it does mean
that any application which wishes to lift the 260-char limitation will
need to replace chdir and getcwd (and DTRT for each thread).

Next, _wstat fails for any file name in the \\?\ form, even if it
doesn't exceed the 260-character limit.  By contrast, FindFirstFileW,
on which AFAIK the MS implementation of 'stat' is based, has no
problem with such file names, and works with filenames up to 32K
characters.

On Windows XP (but not on Windows 7), _wfullpath also failed for long
file names.

The upshot of all this is that it's not enough to just switch to
wchar_t and \\?\ form of file names, there's some non-trivial support
code that will need to be written to take advantage of that in any
sufficiently large application (such as GDB).

Moreover, even just using Unicode APIs is of questionable value in GDB
(as in any other console program on Windows), because:

  . supporting Unicode on the console will need very serious changes
    in Readline and in ui-*.c modules; without such changes, one can
    never enter or display file names outside of the current ANSI
    codepage

  . the Windows console has very weak support for non-ASCII fonts even
    when Unicode APIs are used to read and write to the console
    (basically, anything outside of Europe cannot be displayed)

  . any external libraries against which GDB is linked are likely not
    to support Unicode file names or UTF-16 text, so you cannot easily
    communicate with them without losing Unicode capabilities

  . suppose you get from a Unicode API, such as GetModuleFileNameW, a
    file name that cannot be expressed in the current ANSI codepage --
    what can you do with such a file name in GDB? probably nothing
    except displaying an error message

So I think the benefits of switching the MinGW GDB to Unicode APIs are
questionable, although if someone steps forward and does a clean job
for that, I'm quite sure the patches will be more than welcome.

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]

On Dec 18 22:01, Eli Zaretskii wrote:
> > Date: Wed, 18 Dec 2013 20:17:57 +0100
> > From: Corinna Vinschen <vinschen@redhat.com>
> > 
> > > > I don't know, but I doubt it.  The filenames in case of calling the
> > > > _wfopen functions are probably sent directly to the underlying WIn32
> > > > functions.
> > > 
> > > I don't think they can do that, because it would mean that relative
> > > file names, "../foo", etc. are forbidden for _wfopen.  The CRT
> > > functions must go through file-name normalization, so unless they
> > > detect \\?\ file names up front and refrain from any processing, the
> > > long file names will not work, because the normalization routines are
> > > where these limitations live, AFAIK.
> > 
> > There's a lot of theory here.  Did you actually *try* it?
> 
> No.  Who can afford to try everything, unless they are paying you to
> do that?

Well, it might not take much more time than musings in email...

> > I just did:
> > 
> > ===========================================
> > #include <stdio.h>
> > #include <errno.h>
> > 
> > int main(void)
> > {
> >   FILE *fp;
> > 
> >   fp = _wfopen (L"\\\\?\\C:\\Windows\\System32\\ntdll.dll", L"r");
> >   if (!fp)
> >     printf ("_wfopen w/ long pathname fails, errno = %d\n", errno);
> >   else
> >     {
> >       printf ("_wfopen w/ long pathname works\n");
> >       fclose(fp);
> >     }
> >    return 0;
> > }
> > ===========================================
> 
> Thanks, that's good to know.  But that's just one CRT function.  There
> are a gazillion of them, and I'm quite sure some use MAX_PATH of 260
> internally.  Just grep the sources of the CRT functions.

Sigh.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Eli Zaretskii]

> Date: Wed, 18 Dec 2013 20:17:57 +0100
> From: Corinna Vinschen <vinschen@redhat.com>
> 
> > > I don't know, but I doubt it.  The filenames in case of calling the
> > > _wfopen functions are probably sent directly to the underlying WIn32
> > > functions.
> > 
> > I don't think they can do that, because it would mean that relative
> > file names, "../foo", etc. are forbidden for _wfopen.  The CRT
> > functions must go through file-name normalization, so unless they
> > detect \\?\ file names up front and refrain from any processing, the
> > long file names will not work, because the normalization routines are
> > where these limitations live, AFAIK.
> 
> There's a lot of theory here.  Did you actually *try* it?

No.  Who can afford to try everything, unless they are paying you to
do that?

> I just did:
> 
> ===========================================
> #include <stdio.h>
> #include <errno.h>
> 
> int main(void)
> {
>   FILE *fp;
> 
>   fp = _wfopen (L"\\\\?\\C:\\Windows\\System32\\ntdll.dll", L"r");
>   if (!fp)
>     printf ("_wfopen w/ long pathname fails, errno = %d\n", errno);
>   else
>     {
>       printf ("_wfopen w/ long pathname works\n");
>       fclose(fp);
>     }
>    return 0;
> }
> ===========================================

Thanks, that's good to know.  But that's just one CRT function.  There
are a gazillion of them, and I'm quite sure some use MAX_PATH of 260
internally.  Just grep the sources of the CRT functions.

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1646 bytes --]

On Dec 18 20:18, Eli Zaretskii wrote:
> > Date: Wed, 18 Dec 2013 18:31:55 +0100
> > From: Corinna Vinschen <vinschen@redhat.com>
> > 
> > > So lifting the 260-char limit means both go to Unicode _and_ use the
> > > \\?\ format of file names, is that right?
> > 
> > Right.
> > 
> > > If so, it probably means
> > > that CRT functions that accept wchar_t arrays as file names (_wfopen
> > > etc.) cannot be used with such long file names, and one needs to call
> > > the Win32 APIs directly.  Correct?
> > 
> > I don't know, but I doubt it.  The filenames in case of calling the
> > _wfopen functions are probably sent directly to the underlying WIn32
> > functions.
> 
> I don't think they can do that, because it would mean that relative
> file names, "../foo", etc. are forbidden for _wfopen.  The CRT
> functions must go through file-name normalization, so unless they
> detect \\?\ file names up front and refrain from any processing, the
> long file names will not work, because the normalization routines are
> where these limitations live, AFAIK.

There's a lot of theory here.  Did you actually *try* it?  I just
did:

===========================================
#include <stdio.h>
#include <errno.h>

int main(void)
{
  FILE *fp;

  fp = _wfopen (L"\\\\?\\C:\\Windows\\System32\\ntdll.dll", L"r");
  if (!fp)
    printf ("_wfopen w/ long pathname fails, errno = %d\n", errno);
  else
    {
      printf ("_wfopen w/ long pathname works\n");
      fclose(fp);
    }
   return 0;
}
===========================================


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Eli Zaretskii]

> Date: Wed, 18 Dec 2013 18:31:55 +0100
> From: Corinna Vinschen <vinschen@redhat.com>
> 
> > So lifting the 260-char limit means both go to Unicode _and_ use the
> > \\?\ format of file names, is that right?
> 
> Right.
> 
> > If so, it probably means
> > that CRT functions that accept wchar_t arrays as file names (_wfopen
> > etc.) cannot be used with such long file names, and one needs to call
> > the Win32 APIs directly.  Correct?
> 
> I don't know, but I doubt it.  The filenames in case of calling the
> _wfopen functions are probably sent directly to the underlying WIn32
> functions.

I don't think they can do that, because it would mean that relative
file names, "../foo", etc. are forbidden for _wfopen.  The CRT
functions must go through file-name normalization, so unless they
detect \\?\ file names up front and refrain from any processing, the
long file names will not work, because the normalization routines are
where these limitations live, AFAIK.

Anyway, thanks for your insights.

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

On Dec 18 19:28, Eli Zaretskii wrote:
> > Date: Wed, 18 Dec 2013 18:18:19 +0100
> > From: Corinna Vinschen <vinschen@redhat.com>
> > 
> > On Dec 18 19:03, Eli Zaretskii wrote:
> > > > Date: Wed, 18 Dec 2013 17:07:07 +0100
> > > > > Are you sure that 32K capability cannot be had with ANSI file names
> > > > > using the \\?\ notation?
> > > > 
> > > > Yes.  The \\?\ notation only works in the UNICODE API[*].  The reason
> > > > is that the ANSI API is just a thin layer over the actual UNICODE
> > > > functionality, and the conversion from ANSI to UNICODE is done using a
> > > > per-thread fixed-size buffer of 520 bytes.
> > > 
> > > Does this mean that using \\?\ with ANSI-encoded file names buys us
> > > 520-byte file names?
> > 
> > 260 char ANSI -> 520 bytes UTF-16
> 
> Oh, I misunderstood then.
> 
> So lifting the 260-char limit means both go to Unicode _and_ use the
> \\?\ format of file names, is that right?

Right.

> If so, it probably means
> that CRT functions that accept wchar_t arrays as file names (_wfopen
> etc.) cannot be used with such long file names, and one needs to call
> the Win32 APIs directly.  Correct?

I don't know, but I doubt it.  The filenames in case of calling the
_wfopen functions are probably sent directly to the underlying WIn32
functions.  There's no good reason to drop the long path prefixes.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Eli Zaretskii]

> Date: Wed, 18 Dec 2013 18:18:19 +0100
> From: Corinna Vinschen <vinschen@redhat.com>
> 
> On Dec 18 19:03, Eli Zaretskii wrote:
> > > Date: Wed, 18 Dec 2013 17:07:07 +0100
> > > > Are you sure that 32K capability cannot be had with ANSI file names
> > > > using the \\?\ notation?
> > > 
> > > Yes.  The \\?\ notation only works in the UNICODE API[*].  The reason
> > > is that the ANSI API is just a thin layer over the actual UNICODE
> > > functionality, and the conversion from ANSI to UNICODE is done using a
> > > per-thread fixed-size buffer of 520 bytes.
> > 
> > Does this mean that using \\?\ with ANSI-encoded file names buys us
> > 520-byte file names?
> 
> 260 char ANSI -> 520 bytes UTF-16

Oh, I misunderstood then.

So lifting the 260-char limit means both go to Unicode _and_ use the
\\?\ format of file names, is that right?  If so, it probably means
that CRT functions that accept wchar_t arrays as file names (_wfopen
etc.) cannot be used with such long file names, and one needs to call
the Win32 APIs directly.  Correct?

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

On Dec 18 19:03, Eli Zaretskii wrote:
> > Date: Wed, 18 Dec 2013 17:07:07 +0100
> > > Are you sure that 32K capability cannot be had with ANSI file names
> > > using the \\?\ notation?
> > 
> > Yes.  The \\?\ notation only works in the UNICODE API[*].  The reason
> > is that the ANSI API is just a thin layer over the actual UNICODE
> > functionality, and the conversion from ANSI to UNICODE is done using a
> > per-thread fixed-size buffer of 520 bytes.
> 
> Does this mean that using \\?\ with ANSI-encoded file names buys us
> 520-byte file names?

260 char ANSI -> 520 bytes UTF-16


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Eli Zaretskii]

> Date: Wed, 18 Dec 2013 17:07:07 +0100
> From: Corinna Vinschen <vinschen@redhat.com>
> 
> 
> [1:text/plain Hide]
> 
> On Dec 18 17:45, Eli Zaretskii wrote:
> > > Date: Wed, 18 Dec 2013 12:20:45 +0100
> > > From: Corinna Vinschen <vinschen@redhat.com>
> > > 
> > > In theory, you should never use the ANSI API on Windows, unless you're
> > > still building GDB for Windows 9x, which should be really, really dead
> > > by now, hopefully.
> > 
> > Did we decide to drop Windows 9X support?  Unicode APIs will not work
> > on Windows 9X (in fact, I think Windows will refuse to run such a
> > gdb.exe), unless we link against unicows.dll.
> 
> I didn't say that.  I said I *hoped* that 9x is dead by now.

It's still widely used in some parts of the world, I'm told.

> > > - The ANSI API only supports a single- or doublebyte codeset in almost
> > >   all language versions of Windows.  There are only a handful languages
> > >   which are using UTF-8 as ANSI codeset on Windows, most use something
> > >   like CP1252 or some other codeset which is not capable of handling all
> > >   UNICODE characters.
> > > 
> > > - The ANSI API restricts filenames to MAX_PATH (260) characters, while
> > >   the UNICODE API and the underlying OS allow paths of up to 32K.
> > 
> > But the MinGW build of GDB is still in the ANSI codepage world, and
> > will probably remain there for the observable future, because Windows
> > runtime and file APIs don't understand UTF-8 encoding,
> 
> They do understand UNICODE if you use the wide char functions.

That requires conversion back and forth, something best done in
wrapper functions, because otherwise we will need ugly ifdef's all
over the place.

> Implementing wrappers for the msvcrt functions used elsewhere in the
> code at one point shouldn't be that hard, just a bit of boring work.

Well, I just came up from such boring work (in Emacs), and I can tell
you it's nowhere near "a bit".  Maybe someone who can work on this
full-time could do this quickly, but there's no such person on board
for MinGW GDB at this time, AFAIK.

> > Are you sure that 32K capability cannot be had with ANSI file names
> > using the \\?\ notation?
> 
> Yes.  The \\?\ notation only works in the UNICODE API[*].  The reason
> is that the ANSI API is just a thin layer over the actual UNICODE
> functionality, and the conversion from ANSI to UNICODE is done using a
> per-thread fixed-size buffer of 520 bytes.

Does this mean that using \\?\ with ANSI-encoded file names buys us
520-byte file names?

>  The ANSI API is really something which should be avoided these
> days.

Unfortunately, given the way most Unix packages are written (using
'char *' for file names), this is easier said than done.  E.g., I'm
not aware of any other GNU package except Emacs that supports Unicode
APIs, and I don't think that's an accident.

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]

On Dec 18 17:45, Eli Zaretskii wrote:
> > Date: Wed, 18 Dec 2013 12:20:45 +0100
> > From: Corinna Vinschen <vinschen@redhat.com>
> > 
> > In theory, you should never use the ANSI API on Windows, unless you're
> > still building GDB for Windows 9x, which should be really, really dead
> > by now, hopefully.
> 
> Did we decide to drop Windows 9X support?  Unicode APIs will not work
> on Windows 9X (in fact, I think Windows will refuse to run such a
> gdb.exe), unless we link against unicows.dll.

I didn't say that.  I said I *hoped* that 9x is dead by now.

> > - The ANSI API only supports a single- or doublebyte codeset in almost
> >   all language versions of Windows.  There are only a handful languages
> >   which are using UTF-8 as ANSI codeset on Windows, most use something
> >   like CP1252 or some other codeset which is not capable of handling all
> >   UNICODE characters.
> > 
> > - The ANSI API restricts filenames to MAX_PATH (260) characters, while
> >   the UNICODE API and the underlying OS allow paths of up to 32K.
> 
> But the MinGW build of GDB is still in the ANSI codepage world, and
> will probably remain there for the observable future, because Windows
> runtime and file APIs don't understand UTF-8 encoding,

They do understand UNICODE if you use the wide char functions.

> and switching
> to wchar_t everywhere in GDB is unthinkable.  So unless someone
> volunteers to provide wrappers for all the library functions GDB calls
> that accept or return file names, and make those wrappers accept UTF-8
> encoded file names, we are stuck.

For a start it might be feasible to use the already existing UNICODE
code in windows-nat.c.  Implementing wrappers for the msvcrt functions
used elsewhere in the code at one point shouldn't be that hard, just
a bit of boring work.

> > Cygwin is using the UNICODE and native NT APIs exclusively
> 
> Sure, but Cygwin uses its own runtime.
> 
> > so paths in Cygwin are only restricted by the maximum OS capability
> > of 32K, and the influence of the PATH_MAX setting of 4096.
> 
> Are you sure that 32K capability cannot be had with ANSI file names
> using the \\?\ notation?

Yes.  The \\?\ notation only works in the UNICODE API[*].  The reason
is that the ANSI API is just a thin layer over the actual UNICODE
functionality, and the conversion from ANSI to UNICODE is done using a
per-thread fixed-size buffer of 520 bytes.  The ANSI API is really
something which should be avoided these days.


Corinna

[*] http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx

-- 
Corinna Vinschen
Cygwin Maintainer
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Eli Zaretskii]

> Date: Wed, 18 Dec 2013 12:20:45 +0100
> From: Corinna Vinschen <vinschen@redhat.com>
> 
> In theory, you should never use the ANSI API on Windows, unless you're
> still building GDB for Windows 9x, which should be really, really dead
> by now, hopefully.

Did we decide to drop Windows 9X support?  Unicode APIs will not work
on Windows 9X (in fact, I think Windows will refuse to run such a
gdb.exe), unless we link against unicows.dll.

> - The ANSI API only supports a single- or doublebyte codeset in almost
>   all language versions of Windows.  There are only a handful languages
>   which are using UTF-8 as ANSI codeset on Windows, most use something
>   like CP1252 or some other codeset which is not capable of handling all
>   UNICODE characters.
> 
> - The ANSI API restricts filenames to MAX_PATH (260) characters, while
>   the UNICODE API and the underlying OS allow paths of up to 32K.

But the MinGW build of GDB is still in the ANSI codepage world, and
will probably remain there for the observable future, because Windows
runtime and file APIs don't understand UTF-8 encoding, and switching
to wchar_t everywhere in GDB is unthinkable.  So unless someone
volunteers to provide wrappers for all the library functions GDB calls
that accept or return file names, and make those wrappers accept UTF-8
encoded file names, we are stuck.

> Cygwin is using the UNICODE and native NT APIs exclusively

Sure, but Cygwin uses its own runtime.

> so paths in Cygwin are only restricted by the maximum OS capability
> of 32K, and the influence of the PATH_MAX setting of 4096.

Are you sure that 32K capability cannot be had with ANSI file names
using the \\?\ notation?

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1642 bytes --]

On Dec 16 17:12, Pedro Alves wrote:
> On 12/13/2013 10:37 PM, Pierre Muller wrote:
> > Following this thread
> > https://sourceware.org/ml/gdb-patches/2013-12/msg00073.html
> > 
> > The patch
> > https://sourceware.org/ml/gdb-cvs/2013-12/msg00054.html
> > 
> > introduced a failure for cygwin native build.
> > The problem is that __USEWIDE is not considered in the patch.
> > 
> > The patch below fixes this compilation error
> > and should allow cygwin to work as mingw.
> 
> Looks fine to me.
> 
> (Though I wonder why not just use GetModuleFileNameExA
> explicitly.  In fact, it's what gdbserver does).

Maybe I'm a bit late, but I'd like to chime in here.  Don't use
GetModuleFileNameExA, use GetModuleFileNameExW.

In theory, you should never use the ANSI API on Windows, unless you're
still building GDB for Windows 9x, which should be really, really dead
by now, hopefully.

The reasons are:

- The ANSI API only supports a single- or doublebyte codeset in almost
  all language versions of Windows.  There are only a handful languages
  which are using UTF-8 as ANSI codeset on Windows, most use something
  like CP1252 or some other codeset which is not capable of handling all
  UNICODE characters.

- The ANSI API restricts filenames to MAX_PATH (260) characters, while
  the UNICODE API and the underlying OS allow paths of up to 32K.

Cygwin is using the UNICODE and native NT APIs exclusively, so paths in
Cygwin are only restricted by the maximum OS capability of 32K, and the
influence of the PATH_MAX setting of 4096.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

RE: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Pierre Muller]

> -----Message d'origine-----
> De : gdb-patches-owner@sourceware.org [mailto:gdb-patches-
> owner@sourceware.org] De la part de Pedro Alves
> Envoyé : lundi 16 décembre 2013 18:13
> À : Pierre Muller
> Cc : gdb-patches@sourceware.org
> Objet : Re: [RFA] Fix cygwin compilation failure due to nameless
> LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing
> 
> On 12/13/2013 10:37 PM, Pierre Muller wrote:
> > Following this thread
> > https://sourceware.org/ml/gdb-patches/2013-12/msg00073.html
> >
> > The patch
> > https://sourceware.org/ml/gdb-cvs/2013-12/msg00054.html
> >
> > introduced a failure for cygwin native build.
> > The problem is that __USEWIDE is not considered in the patch.
> >
> > The patch below fixes this compilation error
> > and should allow cygwin to work as mingw.
> 
> Looks fine to me.

  Thanks for the approval,
fix committed with Yao's suggestion concerning the ChangeLog entry.
Thank you, Yao!
 
> (Though I wonder why not just use GetModuleFileNameExA
> explicitly.  In fact, it's what gdbserver does).
  This is just that GetModuleFileNameEx is a function variable which is set
to 
bad_GetModuleFileNameEx if it is not found inside kernel32 DLL.

  Using GetModuleFileNameExA would still require that check,
separately from GetModuleFileNameEx, so the change would have been bigger
without any benefit, at least to my understanding...

Pierre Muller

Re: [RFA] Fix cygwin compilation failure due to nameless LOAD_DLL_DEBUG_EVENT causes ntdll.dll to be missing

[Pedro Alves]

On 12/13/2013 10:37 PM, Pierre Muller wrote:
> Following this thread
> https://sourceware.org/ml/gdb-patches/2013-12/msg00073.html
> 
> The patch
> https://sourceware.org/ml/gdb-cvs/2013-12/msg00054.html
> 
> introduced a failure for cygwin native build.
> The problem is that __USEWIDE is not considered in the patch.
> 
> The patch below fixes this compilation error
> and should allow cygwin to work as mingw.

Looks fine to me.

(Though I wonder why not just use GetModuleFileNameExA
explicitly.  In fact, it's what gdbserver does).

-- 
Pedro Alves