newlib, modern C++ and dirent.h revisited

newlib, modern C++ and dirent.h revisited

[Pekka Seppänen]

Hi.

As this problem has some historic weight to it, here's a short recap: 
Unless compiling for a specific host, which ships it's own 
<sys/dirent.h> (which is included by <dirent.h>), newlib's default 
behaviour since 2005 has been to simply #error, which causes tests to 
work as expected (to fail). Since the very beginning (early 1990s?), it 
used to be #include_next, but as <sys/dirent.h> is included by 
<dirent.h>, #include_next <dirent.h> at <sys/dirent.h> will include 
again <dirent.h>, thus producing no easily detectable error -- fair 
enough [1].

At the time the fix was implemented, in order to provide a proper 
<dirent.h> the way was to simply provide an include path where the file 
resides using e.g. -isystem, so that it appears before newlib's 
include/sys. It stills kinda works as of today, but not really, so 
here's why I'm reluctant to use it:

Currently, I'm using newlib for a tiny RTOS that runs on a certain ARM 
based processors. Under the hood, I've implemented dirent.h 
functionality and everything works as expected. To keep the toolchain 
side as clean and tidy as possible, I'm pretty much limiting all the 
customizations to simply running any configuration scripts with the 
options I need. So, e.g. my newlib doesn't really know that it will be 
running hosted on an OS that does dirent.h stuff -- and it really 
doesn't have to. While configure.host is not that involved, doing a 
custom job there would require me to maintain that file and any 
corresponding files under libc/machine/. While it would be nice, I 
simply don't have time for this -- every moment spent building the 
toolchain is taken away from maintaining the actual RTOS and everything 
that sits atop of that (it's not like newlib is the only link in the 
chain).

Now, the majority of application code which runs on the OS is written 
using modern C++ (17 and onwards). For the compiler I'm using GCC, but I 
guess any decent compiler would do it. Again, when (cross-)compiling the 
GCC I'm limiting myself to any options, that are settable via the 
autoconf's configure script because my time is limited and kludges have 
tendency to break. Here's where the things have have changed, not 
perhaps dramatically, but changed nevertheless since the 2005 patch was 
made.

Evolved from Boost.Filesystem, STL has had an (experimental) 
std::(experimental::)filesystem since 2015. I would like to use, as I 
don't have any necessity to reinvent those features. When using GCC, 
this will be compiled as libstdc++fs, as part of the libstdc++. So, when 
the cross-compiling GCC is built, the proper dirent.h must be available 
as filesystem is not a header-only library. To configure native system 
headers for the GCC, as of 2011 --with-native-system-header-dir options 
has been available. It sets up a builtin include location, from which 
the system headers are supposed to be pulled -- in reality, it is of 
course just an additional builtin include path. As an unfortunate 
effect, the native-system-header-dir appears after, not before, the 
include path set up by --with-sysroot prefix (it will appear as 
-isystem, before any CXXFLAGS, set up by autoconf).

So, the default <sys/dirent.h> gets included and the libstdc++fs does 
not get built (the presence of <dirent.h> is determined by autoconf). 
One could do a dirty copy-over kludge here, but I just don't see that as 
a proper thing to do as it'll always pile up. The same goes with simply 
removing the <sys/dirent.h> -- while either would would fix the problem 
for me, it wouldn't so much do it for the others (e.g. the official ARM 
GNU toolchain). Also, hammering the default <sys/dirent.h> with GCC's 
fixincludes isn't any better, as it has no external hooks so one would 
have to manually maintain yet another configuration file.

I presume the source of this problem, and the reason why the 2005 patch 
was written, is the way #include_next works. If you have just one 
<dirent.h>, the #include_next will function just like #include. However, 
things have evolved and nowadays at least both GCC and LLVM (clang) 
support __has_include_next. Unfortunately, only __has_include is part of 
the C++17, but I guess this makes an excellent use case for 
__has_include_next: i) should you not have any next include file, (a 
properly implemented) __has_include_next will return 0, ii) should you 
not have __has_include_next functionality, simply define 
__has_include_next as zero or check, if it's defined (as compilers 
should really define __has_include as __builtin_has_include etc., or 
whatever internal name the particular vendor chooses).

So, instead of simply doing an #error, I would very much like to see 
something like:

#if defined(__has_include_next) && __has_include_next(<dirent.h>)
#  include_next <dirent.h>
#else
#  error "<dirent.h> not supported"
#endif

and it would just work (given that #if would have to be likely split 
into #ifdef/#if etc.): If there's no additional dirent.h, or your 
compiler is of previous generation, you'll get the old error message. 
Should you have both, it'll be included next -- just like the original 
did (or wished to do) back in the day. This way one could also pull a 
standard, uncustomized prebuilt newlib installation and not have to 
worry about <dirent.h> include order so much, would you like to 
implement one yourself. Like I said, of course `the proper' way to do 
this would be customize newlib, but the way configure scripts are 
currently implemented it involves so much work that it's not simply 
worth it -- patching, copying, checking, working this all up as an yet 
another messy script, for every release etc. takes so much time while 
you simply wanted `not to fail so early'.


What do you think? Worth doing, at least looking into, or is there 
something this sort of approach would hinder? Obviously, this would need 
some testing, so that it's not suddenly 2005 all over again.


-- Pekka


[1] https://sourceware.org/ml/newlib/2005/msg00608.html

Re: newlib, modern C++ and dirent.h revisited

[Joel Sherrill]

On 5/12/2017 10:19 AM, Pekka Seppänen wrote:
> Hi.
>
> As this problem has some historic weight to it, here's a short recap:
> Unless compiling for a specific host, which ships it's own
> <sys/dirent.h> (which is included by <dirent.h>), newlib's default
> behaviour since 2005 has been to simply #error, which causes tests to
> work as expected (to fail). Since the very beginning (early 1990s?), it
> used to be #include_next, but as <sys/dirent.h> is included by
> <dirent.h>, #include_next <dirent.h> at <sys/dirent.h> will include
> again <dirent.h>, thus producing no easily detectable error -- fair
> enough [1].
>
> At the time the fix was implemented, in order to provide a proper
> <dirent.h> the way was to simply provide an include path where the file
> resides using e.g. -isystem, so that it appears before newlib's
> include/sys. It stills kinda works as of today, but not really, so
> here's why I'm reluctant to use it:
>
> Currently, I'm using newlib for a tiny RTOS that runs on a certain ARM
> based processors. Under the hood, I've implemented dirent.h
> functionality and everything works as expected. To keep the toolchain
> side as clean and tidy as possible, I'm pretty much limiting all the
> customizations to simply running any configuration scripts with the
> options I need. So, e.g. my newlib doesn't really know that it will be
> running hosted on an OS that does dirent.h stuff -- and it really
> doesn't have to. While configure.host is not that involved, doing a
> custom job there would require me to maintain that file and any
> corresponding files under libc/machine/. While it would be nice, I
> simply don't have time for this -- every moment spent building the
> toolchain is taken away from maintaining the actual RTOS and everything
> that sits atop of that (it's not like newlib is the only link in the
> chain).

libc/sys not libc/machine. machine is for target architectures.

Supporting an RTOS with code in libc/sys isn't a huge burden.
RTEMS has been doing this for years. Just integrate with
newlib properly. Then gcc and all supporting libraries will
have the proper .h files and be able to build with correct
support.

>
> Now, the majority of application code which runs on the OS is written
> using modern C++ (17 and onwards). For the compiler I'm using GCC, but I
> guess any decent compiler would do it. Again, when (cross-)compiling the
> GCC I'm limiting myself to any options, that are settable via the
> autoconf's configure script because my time is limited and kludges have
> tendency to break. Here's where the things have have changed, not
> perhaps dramatically, but changed nevertheless since the 2005 patch was
> made.
>
> Evolved from Boost.Filesystem, STL has had an (experimental)
> std::(experimental::)filesystem since 2015. I would like to use, as I
> don't have any necessity to reinvent those features. When using GCC,
> this will be compiled as libstdc++fs, as part of the libstdc++. So, when
> the cross-compiling GCC is built, the proper dirent.h must be available
> as filesystem is not a header-only library. To configure native system
> headers for the GCC, as of 2011 --with-native-system-header-dir options
> has been available. It sets up a builtin include location, from which
> the system headers are supposed to be pulled -- in reality, it is of
> course just an additional builtin include path. As an unfortunate
> effect, the native-system-header-dir appears after, not before, the
> include path set up by --with-sysroot prefix (it will appear as
> -isystem, before any CXXFLAGS, set up by autoconf).
>
> So, the default <sys/dirent.h> gets included and the libstdc++fs does
> not get built (the presence of <dirent.h> is determined by autoconf).
> One could do a dirty copy-over kludge here, but I just don't see that as
> a proper thing to do as it'll always pile up. The same goes with simply
> removing the <sys/dirent.h> -- while either would would fix the problem
> for me, it wouldn't so much do it for the others (e.g. the official ARM
> GNU toolchain). Also, hammering the default <sys/dirent.h> with GCC's
> fixincludes isn't any better, as it has no external hooks so one would
> have to manually maintain yet another configuration file.
>
> I presume the source of this problem, and the reason why the 2005 patch
> was written, is the way #include_next works. If you have just one
> <dirent.h>, the #include_next will function just like #include. However,
> things have evolved and nowadays at least both GCC and LLVM (clang)
> support __has_include_next. Unfortunately, only __has_include is part of
> the C++17, but I guess this makes an excellent use case for
> __has_include_next: i) should you not have any next include file, (a
> properly implemented) __has_include_next will return 0, ii) should you
> not have __has_include_next functionality, simply define
> __has_include_next as zero or check, if it's defined (as compilers
> should really define __has_include as __builtin_has_include etc., or
> whatever internal name the particular vendor chooses).
>
> So, instead of simply doing an #error, I would very much like to see
> something like:
>
> #if defined(__has_include_next) && __has_include_next(<dirent.h>)
> #  include_next <dirent.h>
> #else
> #  error "<dirent.h> not supported"
> #endif
>
> and it would just work (given that #if would have to be likely split
> into #ifdef/#if etc.): If there's no additional dirent.h, or your
> compiler is of previous generation, you'll get the old error message.
> Should you have both, it'll be included next -- just like the original
> did (or wished to do) back in the day. This way one could also pull a
> standard, uncustomized prebuilt newlib installation and not have to
> worry about <dirent.h> include order so much, would you like to
> implement one yourself. Like I said, of course `the proper' way to do
> this would be customize newlib, but the way configure scripts are
> currently implemented it involves so much work that it's not simply
> worth it -- patching, copying, checking, working this all up as an yet
> another messy script, for every release etc. takes so much time while
> you simply wanted `not to fail so early'.
>
>
> What do you think? Worth doing, at least looking into, or is there
> something this sort of approach would hinder? Obviously, this would need
> some testing, so that it's not suddenly 2005 all over again.
>
>
> -- Pekka
>
>
> [1] https://sourceware.org/ml/newlib/2005/msg00608.html
>

--joel