AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Zack Weinberg]

glibc 2.25 is going to deprecate the definition of 'major', 'minor',
and 'makedev' by  sys/types.h; see
https://sourceware.org/bugzilla/show_bug.cgi?id=19239 for rationale.
(It was found to be impractical to remove sys/types.h from stdlib.h.)
Unfortunately, AC_HEADER_MAJOR does not detect that these macros are
now throwing warning messages if you don't include sys/sysmacros.h,
and this is reported to break software that uses -Werror (see the very
end of the above bug report).

The ideal solution to this problem would be if there _already existed_
a way for sys/types.h to detect that it's being included by an
autoconf test program.  When it saw that, it would not define
major/minor/makedev at all, which would poke AC_HEADER_MAJOR to go
looking for sys/sysmacros.h.  Inspecting a trivial generated configure
script, though, I don't see any obvious method.  A less ideal method
would be for autoconf and glibc to agree on a macro that
AC_HEADER_MAJOR would define before including sys/types.h, with the
same effect -- I consider this less than ideal mainly because there's
no schedule for autoconf 2.70, whereas glibc 2.25 is due out in about
five months.

Have I missed either a way to carry out the ideal solution, or a third
alternative?  Failing that, can we get the ball rolling on option B,
please?

zw

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Paul Eggert]

Zack Weinberg wrote:
> Have I missed either a way to carry out the ideal solution, or a third
> alternative?

How about changing AC_HEADER_MAJOR to detect those warning messages?

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Zack Weinberg]

On Wed, Aug 31, 2016 at 11:24 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> Zack Weinberg wrote:
>> Have I missed either a way to carry out the ideal solution, or a third
>> alternative?
>
> How about changing AC_HEADER_MAJOR to detect those warning messages?

That's difficult, but more importantly it doesn't do any good till we
get an autoconf release out -- and then everyone using AC_HEADER_MAJOR
has to regenerate their configure script and release *their* software.
I'm hoping we can find a solution that can be implemented entirely
within glibc.

zw

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 2487 bytes --]

On 09/01/2016 08:08 PM, Zack Weinberg wrote:
> On Wed, Aug 31, 2016 at 11:24 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
>> Zack Weinberg wrote:
>>> Have I missed either a way to carry out the ideal solution, or a third
>>> alternative?
>>
>> How about changing AC_HEADER_MAJOR to detect those warning messages?
> 
> That's difficult, but more importantly it doesn't do any good till we
> get an autoconf release out -- and then everyone using AC_HEADER_MAJOR
> has to regenerate their configure script and release *their* software.
> I'm hoping we can find a solution that can be implemented entirely
> within glibc.

I may be able to get some time at my $dayjob to get an autoconf release
out the door before the glibc release; how much time do I have left, to
know what priority I need to give this?

And you are correct that a new autoconf won't help projects that don't
re-run autoconf with the fix, but at a bare minimum, we can at least get
gnulib to support the same macro definition as what autoconf 2.70 will
have, so that projects using gnulib will at least not have to wait for
an autoconf release, as well as having updated docs in the released
autoconf manual that tell users what to do when they run into the problem.

One other possibility that distros can do is to prime a config.site
file, with $ac_cv_header_sys_types_h_makedev=no, to forcefully bypass
the configure check that is normally done where <sys/types.h> would
warn.  It has to be in config.site, because non-glibc systems do not
want to do that.

Meanwhile, even without an autoconf release scheduled, I am currently
working on patching the existing autoconf macro and documentation to
match the current situation.  It sounds to me like we want the following
logic:

If <sys/sysmacros.h> exists and defines major(), use that,
else if <sys/mkdev.h> exists, use that,
else if <sys/types.h> defines major(), use that,
else not available

instead of the current broken logic:

If <sys/types.h> defines makedev(), use that,
else if <sys/mkdev.h> exists, use that,
else if <sys/sysmacros.h> exists, use that,
else not available

Does anyone see any flaws in my reasoning? I'll post a patch as a
followup, if it passes testing on my end (I'm using Fedora rawhide as my
testbed, which is definitely hitting the compilation failure with
current glibc).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 2521 bytes --]

[adding libvirt list, as this is what sparked my investigations so far
today]

On 09/13/2016 04:36 PM, Eric Blake wrote:

> One other possibility that distros can do is to prime a config.site
> file, with $ac_cv_header_sys_types_h_makedev=no, to forcefully bypass
> the configure check that is normally done where <sys/types.h> would
> warn.  It has to be in config.site, because non-glibc systems do not
> want to do that.

Okay, I have confirmed that this prime-the-cache idea DOES work, using
libvirt.git commit 419bc8cf (one commit prior to d53fa838 which
installed a hack into libvirt to force the use of -Werror for the
duration of AC_HEADER_MAJOR [1]).  By itself, './configure' succeeds but
leaves MAJOR_IN_SYSMACROS undefined, which results in a build failure
later in libvirt; but when run as './configure
ac_cv_header_sys_types_h_makedev=no', MAJOR_IN_SYSMACROS gets defined
and the build succeeds.

I want to remove the libvirt hack (not all compilers understand -Werror,
even if libvirt is only ever built by gcc or clang), and my experiment
was enough to prove that:

1. distros that provide a config.site file can use this file to avoid
the problem, even for packages that have not yet rebuilt with a
new-enough autoconf to fix the issue

2. autoconf and gnulib should indeed be fixed to probe for
<sys/sysmacros.h> _prior_ to probing for whether makedev() exists in
<sys/type.h>, rather than the 2.69 logic of only checking for
<sys/sysmacros.h> if makedev() was not found earlier.

Of course, the libvirt hack should not be reverted until autoconf and
gnulib have the final solution in place.

> 
> Meanwhile, even without an autoconf release scheduled, I am currently
> working on patching the existing autoconf macro and documentation to
> match the current situation.  It sounds to me like we want the following
> logic:
> 
> If <sys/sysmacros.h> exists and defines major(), use that,
> else if <sys/mkdev.h> exists, use that,
> else if <sys/types.h> defines major(), use that,
> else not available

The current autoconf code assumes that if <sys/sysmacros.h> exists, then
it defines major()/minor()/makedev(); I think that assumption is still
safe.  Where autoconf was wrong was that it was not even probing for the
existence of <sys/sysmacros.h> if it found that <sys/types.h> was enough
to pollute the namespace with the three macros.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 786 bytes --]

On 09/13/2016 05:31 PM, Eric Blake wrote:

[hit send too soon]

> Okay, I have confirmed that this prime-the-cache idea DOES work, using
> libvirt.git commit 419bc8cf (one commit prior to d53fa838 which
> installed a hack into libvirt to force the use of -Werror for the
> duration of AC_HEADER_MAJOR [1]).  By itself, './configure' succeeds but
> leaves MAJOR_IN_SYSMACROS undefined, which results in a build failure
> later in libvirt; but when run as './configure
> ac_cv_header_sys_types_h_makedev=no', MAJOR_IN_SYSMACROS gets defined
> and the build succeeds.

I meant to add this link:

[1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=d53fa838

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Zack Weinberg]

On Tue, Sep 13, 2016 at 5:36 PM, Eric Blake <eblake@redhat.com> wrote:
> I may be able to get some time at my $dayjob to get an autoconf release
> out the door before the glibc release; how much time do I have left, to
> know what priority I need to give this?

Per https://sourceware.org/glibc/wiki/Release/2.25, the current plan
is to release glibc 2.25 on 2017-02-01.  I feel responsible for making
sure whatever needs to happen happens in time, but unfortunately my
$dayjob is being especially all-consuming right now and it's not going
to let up until next June at the earliest, so I can't make any
promises. :-(

I can at least make sure that glibc 2.25's release notes mention the
need to use autoconf 2.70, _provided_ that the 2.70 release happens
first.

> Meanwhile, even without an autoconf release scheduled, I am currently
> working on patching the existing autoconf macro and documentation to
> match the current situation.  It sounds to me like we want the following
> logic:
>
> If <sys/sysmacros.h> exists and defines major(), use that,
> else if <sys/mkdev.h> exists, use that,
> else if <sys/types.h> defines major(), use that,
> else not available

This seems sound to me, except that I would recommend we consistently
do check whether the selected header defines major().

zw

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Nick Bowler]

On 8/31/16, Zack Weinberg <zackw@panix.com> wrote:
> glibc 2.25 is going to deprecate the definition of 'major', 'minor',
> and 'makedev' by  sys/types.h; see
> https://sourceware.org/bugzilla/show_bug.cgi?id=19239 for rationale.
> (It was found to be impractical to remove sys/types.h from stdlib.h.)
> Unfortunately, AC_HEADER_MAJOR does not detect that these macros are
> now throwing warning messages if you don't include sys/sysmacros.h,
> and this is reported to break software that uses -Werror (see the very
> end of the above bug report).

It seems that the simplest short term solution is to just not use
-Werror when building packages.  Other than the warning, the header
detection worked, and the test is behaving as documented, right?

Perhaps we could also change Autoconf to do something different, so
new packages will work better with new systems.  This does nothing
for old packages but the impact seems minor.

For better or for worse, AC_HEADER_MAJOR is quite explicitly documented
as probing the headers in a particular way.  If packages are relying on
it behaving exactly as documented then we may have a problem changing
this macro to do something else.  But I expect it's not a big concern.

> The ideal solution to this problem would be if there _already existed_
> a way for sys/types.h to detect that it's being included by an
> autoconf test program.

This is not really ideal as it goes directly against the Autoconf
philosophy that the test cases are representative of the actual
compilation environment.

I have a question though, if these macros are deprecated, will they
be removed from <sys/types.h> eventually?  If they're going to be
removed, why not just punt them right now?  If they're not going to
be removed, why bother with the warnings?

Cheers,
  Nick

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Florian Weimer]

On 09/01/2016 06:37 AM, Nick Bowler wrote:

> It seems that the simplest short term solution is to just not use
> -Werror when building packages.  Other than the warning, the header
> detection worked, and the test is behaving as documented, right?

-Werror does not work because warnings from autoconf test snippets are 
fairly common.  Many of them are not valid C code at all, and GCC 
increasingly detects that (for example, “warning: conflicting types for 
built-in function”).

Florian

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Zack Weinberg]

On Thu, Sep 1, 2016 at 12:37 AM, Nick Bowler <nbowler@draconx.ca> wrote:
> On 8/31/16, Zack Weinberg <zackw@panix.com> wrote:
>> glibc 2.25 is going to deprecate the definition of 'major', 'minor',
>> and 'makedev' by  sys/types.h; see
>> https://sourceware.org/bugzilla/show_bug.cgi?id=19239 for rationale.
>> (It was found to be impractical to remove sys/types.h from stdlib.h.)
>> Unfortunately, AC_HEADER_MAJOR does not detect that these macros are
>> now throwing warning messages if you don't include sys/sysmacros.h,
>> and this is reported to break software that uses -Werror (see the very
>> end of the above bug report).
>
> It seems that the simplest short term solution is to just not use
> -Werror when building packages.  Other than the warning, the header
> detection worked, and the test is behaving as documented, right?

AC_HEADER_MAJOR is obeying the letter of its documentation but not the
spirit.  People using it reasonably expect that it should handle this
transition seamlessly for them.  They've already gone to the trouble
of writing

    #include <sys/types.h>
    #ifdef MAJOR_IN_MKDEV
    #include <sys/mkdev.h>
    #elif MAJOR_IN_SYSMACROS
    #include <sys/sysmacros.h>
    #endif

and shouldn't that be enough, not only to keep their program working
when glibc actually pulls the plug on the definitions in
<sys/types.h>, but to avoid the deprecation warnings?

>> The ideal solution to this problem would be if there _already existed_
>> a way for sys/types.h to detect that it's being included by an
>> autoconf test program.
>
> This is not really ideal as it goes directly against the Autoconf
> philosophy that the test cases are representative of the actual
> compilation environment.

Normally I'm on board with that, but it leaves me pretty well up a creek here...

> I have a question though, if these macros are deprecated, will they
> be removed from <sys/types.h> eventually?  If they're going to be
> removed, why not just punt them right now?  If they're not going to
> be removed, why bother with the warnings?

We (glibc) don't generally like to delete things from the headers with
no advance notice.  The plan is to have deprecation warnings for at
least one release, and then flip the switch.  Exactly which future
release flips the switch will depend on how many complaints we get
about the change.

zw

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Paul Eggert]

>> It seems that the simplest short term solution is to just not use
>> -Werror when building packages.  Other than the warning, the header
>> detection worked, and the test is behaving as documented, right?
> AC_HEADER_MAJOR is obeying the letter of its documentation but not the
> spirit.  People using it reasonably expect that it should handle this
> transition seamlessly for them.

Sure, but Nick's correct that Autoconf-generated code typically cannot be 
configured with -Werror, as there are too many false positives. The problem with 
AC_HEADER_MAJOR is just one of many. So I wouldn't worry about a change causing 
an error for that case. People should not configure with -Werror, that's all.

This has long been common practice among Autoconf users. For example:

> There's your problem right there.  We really do not recommend the use of
> CFLAGS=-Werror during configuration, because it simply does not work.
> ...
> Seriously - the recommendation on this list is that you never use
> -Werror during configure (except maybe to temporarily probe whether
> -Werror even works);

https://lists.gnu.org/archive/html/bug-autoconf/2012-12/msg00001.html

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Zack Weinberg]

On Thu, Sep 1, 2016 at 9:32 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
>>> It seems that the simplest short term solution is to just not use
>>> -Werror when building packages.  Other than the warning, the header
>>> detection worked, and the test is behaving as documented, right?
>>
>> AC_HEADER_MAJOR is obeying the letter of its documentation but not the
>> spirit.  People using it reasonably expect that it should handle this
>> transition seamlessly for them.
>
> Sure, but Nick's correct that Autoconf-generated code typically cannot be
> configured with -Werror, as there are too many false positives. The problem
> with AC_HEADER_MAJOR is just one of many. So I wouldn't worry about a change
> causing an error for that case. People should not configure with -Werror,
> that's all.

The original problem was that AC_HEADER_MAJOR doesn't detect that
glibc 2.25 is throwing warnings, so a *build* with -Werror fails.  I
agree that telling people to *configure* with -Werror is a
non-solution.

zw

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 1981 bytes --]

On 09/01/2016 08:16 PM, Zack Weinberg wrote:
> AC_HEADER_MAJOR is obeying the letter of its documentation but not the
> spirit.  People using it reasonably expect that it should handle this
> transition seamlessly for them.  They've already gone to the trouble
> of writing
> 
>     #include <sys/types.h>
>     #ifdef MAJOR_IN_MKDEV
>     #include <sys/mkdev.h>
>     #elif MAJOR_IN_SYSMACROS
>     #include <sys/sysmacros.h>
>     #endif
> 
> and shouldn't that be enough, not only to keep their program working
> when glibc actually pulls the plug on the definitions in
> <sys/types.h>, but to avoid the deprecation warnings?

The autoconf manual currently doesn't say how to USE the results of
AC_HEADER_MAJOR - it only states that MAJOR_IN_MKDEV or
MAJOR_IN_SYSMACROS might be defined.  But your formulation of the
correct usage _looks_ correct to me, and probably SHOULD be in the
manual.  At which point, the remaining trick to solve is that autoconf
SHOULD be setting MAJOR_IN_SYSMACROS for glibc, but isn't, because it
short-circuits at the point where it sees the pollution in
<sys/types.h>.  I think that rewriting the macro to check for
<sys/sysmacros.h> _first_, not last, will get MAJOR_IN_SYSMACROS set;
and then if programs ARE using the formulation you spell out above, then
we have the desired situation that configure works and the code compiles
without warnings.

Of course, if there were a way for glibc to detect that it was being run
under autoconf, and in that situation intentionally refuse to declare
major/minor/makedev, then existing autoconf clients will already fall
back to defining MAJOR_IN_SYSMACROS.  I just don't know if I feel
comfortable with glibc mucking with the build to compile one way under
autoconf and another way in real life, as autoconf's goal is to see what
will happen in real life.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: AC_HEADER_MAJOR vs. glibc 2.25(-to-be)

[Florian Weimer]

On 09/01/2016 04:27 AM, Zack Weinberg wrote:
> glibc 2.25 is going to deprecate the definition of 'major', 'minor',
> and 'makedev' by  sys/types.h; see
> https://sourceware.org/bugzilla/show_bug.cgi?id=19239 for rationale.
> (It was found to be impractical to remove sys/types.h from stdlib.h.)
> Unfortunately, AC_HEADER_MAJOR does not detect that these macros are
> now throwing warning messages if you don't include sys/sysmacros.h,
> and this is reported to break software that uses -Werror (see the very
> end of the above bug report).

The larger question here is whether we can make toolchain changes which 
alter the outcome of checks in existing (compiled) autoconf tests.

More and more, the invalid C used in autoconf tests blocks useful 
enhancements to the toolchain.

Is it reasonable to expect that autoconf releases happen regularly 
(twice a year), to match glibc/GCC/binutils changes, and that people run 
autoconf before compiling software packages?

Florian