Re: newlib, modern C++ and dirent.h revisited

[Joel Sherrill <joel.sherrill@oarcorp.com>]

On 5/12/2017 10:19 AM, Pekka Seppänen wrote:
> Hi.
>
> As this problem has some historic weight to it, here's a short recap:
> Unless compiling for a specific host, which ships it's own
> <sys/dirent.h> (which is included by <dirent.h>), newlib's default
> behaviour since 2005 has been to simply #error, which causes tests to
> work as expected (to fail). Since the very beginning (early 1990s?), it
> used to be #include_next, but as <sys/dirent.h> is included by
> <dirent.h>, #include_next <dirent.h> at <sys/dirent.h> will include
> again <dirent.h>, thus producing no easily detectable error -- fair
> enough [1].
>
> At the time the fix was implemented, in order to provide a proper
> <dirent.h> the way was to simply provide an include path where the file
> resides using e.g. -isystem, so that it appears before newlib's
> include/sys. It stills kinda works as of today, but not really, so
> here's why I'm reluctant to use it:
>
> Currently, I'm using newlib for a tiny RTOS that runs on a certain ARM
> based processors. Under the hood, I've implemented dirent.h
> functionality and everything works as expected. To keep the toolchain
> side as clean and tidy as possible, I'm pretty much limiting all the
> customizations to simply running any configuration scripts with the
> options I need. So, e.g. my newlib doesn't really know that it will be
> running hosted on an OS that does dirent.h stuff -- and it really
> doesn't have to. While configure.host is not that involved, doing a
> custom job there would require me to maintain that file and any
> corresponding files under libc/machine/. While it would be nice, I
> simply don't have time for this -- every moment spent building the
> toolchain is taken away from maintaining the actual RTOS and everything
> that sits atop of that (it's not like newlib is the only link in the
> chain).

libc/sys not libc/machine. machine is for target architectures.

Supporting an RTOS with code in libc/sys isn't a huge burden.
RTEMS has been doing this for years. Just integrate with
newlib properly. Then gcc and all supporting libraries will
have the proper .h files and be able to build with correct
support.

>
> Now, the majority of application code which runs on the OS is written
> using modern C++ (17 and onwards). For the compiler I'm using GCC, but I
> guess any decent compiler would do it. Again, when (cross-)compiling the
> GCC I'm limiting myself to any options, that are settable via the
> autoconf's configure script because my time is limited and kludges have
> tendency to break. Here's where the things have have changed, not
> perhaps dramatically, but changed nevertheless since the 2005 patch was
> made.
>
> Evolved from Boost.Filesystem, STL has had an (experimental)
> std::(experimental::)filesystem since 2015. I would like to use, as I
> don't have any necessity to reinvent those features. When using GCC,
> this will be compiled as libstdc++fs, as part of the libstdc++. So, when
> the cross-compiling GCC is built, the proper dirent.h must be available
> as filesystem is not a header-only library. To configure native system
> headers for the GCC, as of 2011 --with-native-system-header-dir options
> has been available. It sets up a builtin include location, from which
> the system headers are supposed to be pulled -- in reality, it is of
> course just an additional builtin include path. As an unfortunate
> effect, the native-system-header-dir appears after, not before, the
> include path set up by --with-sysroot prefix (it will appear as
> -isystem, before any CXXFLAGS, set up by autoconf).
>
> So, the default <sys/dirent.h> gets included and the libstdc++fs does
> not get built (the presence of <dirent.h> is determined by autoconf).
> One could do a dirty copy-over kludge here, but I just don't see that as
> a proper thing to do as it'll always pile up. The same goes with simply
> removing the <sys/dirent.h> -- while either would would fix the problem
> for me, it wouldn't so much do it for the others (e.g. the official ARM
> GNU toolchain). Also, hammering the default <sys/dirent.h> with GCC's
> fixincludes isn't any better, as it has no external hooks so one would
> have to manually maintain yet another configuration file.
>
> I presume the source of this problem, and the reason why the 2005 patch
> was written, is the way #include_next works. If you have just one
> <dirent.h>, the #include_next will function just like #include. However,
> things have evolved and nowadays at least both GCC and LLVM (clang)
> support __has_include_next. Unfortunately, only __has_include is part of
> the C++17, but I guess this makes an excellent use case for
> __has_include_next: i) should you not have any next include file, (a
> properly implemented) __has_include_next will return 0, ii) should you
> not have __has_include_next functionality, simply define
> __has_include_next as zero or check, if it's defined (as compilers
> should really define __has_include as __builtin_has_include etc., or
> whatever internal name the particular vendor chooses).
>
> So, instead of simply doing an #error, I would very much like to see
> something like:
>
> #if defined(__has_include_next) && __has_include_next(<dirent.h>)
> #  include_next <dirent.h>
> #else
> #  error "<dirent.h> not supported"
> #endif
>
> and it would just work (given that #if would have to be likely split
> into #ifdef/#if etc.): If there's no additional dirent.h, or your
> compiler is of previous generation, you'll get the old error message.
> Should you have both, it'll be included next -- just like the original
> did (or wished to do) back in the day. This way one could also pull a
> standard, uncustomized prebuilt newlib installation and not have to
> worry about <dirent.h> include order so much, would you like to
> implement one yourself. Like I said, of course `the proper' way to do
> this would be customize newlib, but the way configure scripts are
> currently implemented it involves so much work that it's not simply
> worth it -- patching, copying, checking, working this all up as an yet
> another messy script, for every release etc. takes so much time while
> you simply wanted `not to fail so early'.
>
>
> What do you think? Worth doing, at least looking into, or is there
> something this sort of approach would hinder? Obviously, this would need
> some testing, so that it's not suddenly 2005 all over again.
>
>
> -- Pekka
>
>
> [1] https://sourceware.org/ml/newlib/2005/msg00608.html
>

--joel