Best practices in regard to -D_TIME_BITS=64

Best practices in regard to -D_TIME_BITS=64

[Kaz Kylheku]

Hi all,

What should be the best practice with regard to selecting the width of 
time_t,
which is possible with newer glibc's -D_TIME_BITS=64?

My main concern is specifically with application developers. Should
individual applications be detecting that this is available and using 
it?

Or is this best left to the toolchain/distro people? If some 32 bit
platform (like an embedded system) decides to go to 64 bit time, that
can be rolled into the default compiler options.

If the application is some free software that is packaged into a distro,
then likewise the package maintainers can decide. They can put it into
the distro-wide CFLAGS, or whatever.

I have some apprehensions about just pulling the trigger and using it
individually; like what if the application is included in some system
where it is the only thing requesting that, and it causes some
interoperability issue with other pieces.

This feels different from _FILE_OFFSET_BITS, because _FILE_OFFSET_BITS
does not solve some impeding problem that becomes worse as we approach
a certain date. A program not manipulating large files never needs to
care, even if it manipulates files; whereas there will hardly be such
a thing as a program which manipulates time and which only needs to
handle "small time" such that it's perfectly fine with a 32 bit time_t
and related structures and functions thereof.

One can force downstream people to deal with this. If the program
detects that it's on a 32 bit platform where time_t is still 32 bits
if no special compiler options are used, and where -D_TIME_BITS=64
is available and working, the configure script can fail and make
the user explicitly decide what they want to do: stick with the
32 bit time, or go to 64.

/That/ could be a best practice.

Hmm, any thoughts?

Re: Best practices in regard to -D_TIME_BITS=64

[Paul Eggert]

On 1/4/22 15:33, Kaz Kylheku via Libc-alpha wrote:

> My main concern is specifically with application developers. Should
> individual applications be detecting that this is available and using it?

Yes, that's what we're doing with applications that use Gnulib. The 
Gnulib 'largefile' module morphed from its original goal of having apps 
be able to access files larger than 2 GiB, to the more-modern goal of 
being able to access files whose sizes, device numbers, inode numbers, 
and/or timestamps exceed traditional limits. Applications using the 
'largefile' module (and pretty much every Gnulib-using application uses 
the 'largefile' module) now get 64-bit time_t by default, on platforms 
that have 64-bit time_t.


> I have some apprehensions about just pulling the trigger and using it
> individually; like what if the application is included in some system
> where it is the only thing requesting that, and it causes some
> interoperability issue with other pieces.

Gnulib has behaved this way since July, and this behavior appears in the 
latest versions of coreutils, diffutils, grep, gzip and perhaps others. 
As far as I know, nobody's reported problems yet. That's of course not 
to say there won't be problems.


> One can force downstream people to deal with this. If the program
> detects that it's on a 32 bit platform where time_t is still 32 bits
> if no special compiler options are used, and where -D_TIME_BITS=64
> is available and working, the configure script can fail and make
> the user explicitly decide what they want to do: stick with the
> 32 bit time, or go to 64.

We didn't bother doing that with Gnulib; we just went with 64-bit time_t 
if available, 32-bit time_t if not.

Last year we did add a Gnulib module 'year2038' that causes 'configure' 
to fail by default if time_t is 32-bit; configurers can override that 
with './configure --disable-year2038' so that 'configure' will succeed 
even if time_t is 32-bit. Coreutils uses this module, since we didn't 
want people to build 32-bit time_t core utilities unless they explicitly 
wanted them.

People who build or use libraries that expose time_t to the API will 
likely have more issues than the abovementioned packages (time_t is like 
off_t and ino_t in that regard).


> A program not manipulating large files never needs to
> care, even if it manipulates files

Such a program will need to care, if it's compiled with 32-bit off_t and 
is linked to a 64-bit off_t library that exposes off_t to the API. 
time_t is no different from off_t here.

Re: Best practices in regard to -D_TIME_BITS=64

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]



> On 5 Jan 2022, at 00:49, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
> On 1/4/22 15:33, Kaz Kylheku via Libc-alpha wrote:
> 
>> My main concern is specifically with application developers. Should
>> individual applications be detecting that this is available and using it?
> 
> Yes, that's what we're doing with applications that use Gnulib. The Gnulib 'largefile' module morphed from its original goal of having apps be able to access files larger than 2 GiB, to the more-modern goal of being able to access files whose sizes, device numbers, inode numbers, and/or timestamps exceed traditional limits. Applications using the 'largefile' module (and pretty much every Gnulib-using application uses the 'largefile' module) now get 64-bit time_t by default, on platforms that have 64-bit time_t.
> 

The mixing can be problematic, as noted below, when things start suddenly using 64-bit unexpectedly.

>> I have some apprehensions about just pulling the trigger and using it
>> individually; like what if the application is included in some system
>> where it is the only thing requesting that, and it causes some
>> interoperability issue with other pieces.
> 
> Gnulib has behaved this way since July, and this behavior appears in the latest versions of coreutils, diffutils, grep, gzip and perhaps others. As far as I know, nobody's reported problems yet. That's of course not to say there won't be problems.
> 

FWIW, we've hit issues in Gentoo with wget/gnutls: https://bugs.gentoo.org/828001 <https://bugs.gentoo.org/828001>.

It appears Fedora hit a similar problem too. This has led us to move forward our planning for how to
handle the migration.

We have some rough notes at https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration <https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration>
(although it's also a bit of brainstorming too).

best,
sam



[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: Best practices in regard to -D_TIME_BITS=64

[Paul Eggert]

On 1/4/22 16:53, Sam James wrote:

> FWIW, we've hit issues in Gentoo with wget/gnutls: 
> https://bugs.gentoo.org/828001 <https://bugs.gentoo.org/828001>.

Thanks for the heads-up.

Yes, not surprised about that, as mixing 64-bit time_t apps with 32-bit 
time_t libraries is a recipe for trouble if the APIs depend on time_t. 
For apps that use such libraries, it really has to be a big bang: 
configure all that stuff with --disable-year2038 -D_TIME_BITS=32 before, 
and with --enable-year2038 CFLAGS=-D_TIME_BITS=64 after.

> https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration 

Thanks. The musl notes look helpful there.

Re: Best practices in regard to -D_TIME_BITS=64

[Kaz Kylheku]

On 2022-01-04 20:43, Paul Eggert wrote:
> On 1/4/22 16:53, Sam James wrote:
> 
>> FWIW, we've hit issues in Gentoo with wget/gnutls: 
>> https://bugs.gentoo.org/828001 <https://bugs.gentoo.org/828001>.
> 
> Thanks for the heads-up.
> 
> Yes, not surprised about that, as mixing 64-bit time_t apps with
> 32-bit time_t libraries is a recipe for trouble if the APIs depend on
> time_t. For apps that use such libraries, it really has to be a big
> bang: configure all that stuff with --disable-year2038 -D_TIME_BITS=32
> before, and with --enable-year2038 CFLAGS=-D_TIME_BITS=64 after.
> 
>> https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration

I looked at this Wiki page and one thing stands out.

I would recommend to the Gentoo people to forget about CPPFLAGS, and
just use CFLAGS, even for -D material. If you put -D_TIME_BITS=64
into CFLAGS, you then don't have to deal with packages that perfectly
well honor CFLAGS, but not CPPFLAGS.

People don't know about CPPFLAGS. It seems to be just something invented
by GNU Make, referenced in some of its built-in recipes. Getting all
upstream packages to recognize CPPFLAGS (or else patching them locally)
is just a lot of make-work (pun intended).

I'm also aware of a historic *ad hoc* practice of using CPPFLAGS for C++ 
CFLAGS;
i.e. that for which CXXFLAGS is a better practice.

For what it's worth, POSIX's description of make mentions no CPPFLAGS,
but does mention CFLAGS, LDFLAGS. (It also doesn't mention LDLIBS, 
another
separation variable, but that I suspect is much more entrenched than 
CPPFLAGS,
and the separation is quite essential: you need to do that any time you 
specify
additional -l arguments for linking.)

GNU's built-in rules do some nonsensical things like:

   LINK.c := $(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)

(If CPPFLAGS is needed all the way through linking C, what purpose does 
it serve
separate from CFLAGS? And if I'm writing my own linking recipe, do I 
have to
reproduce this?)

CPPFLAGS is something that would be useful for calling a standalone C 
preprocessor
that doesn't understand compiler options.

But:

You don't need that in a GCC environment, because you can call gcc with 
options
that perform preprocessing only, and still pass other options specific 
to
compiling.

Not only that, but GNU cpp is tolerant to gcc options not pertaining to
preprocessing:

No problem:

$ echo | cpp -fstrict-aliasing > /dev/null

$ echo | cpp -fNoNExistent > /dev/null
cpp: error: unrecognized command line option ‘-fNoNExistent’; did you 
mean ‘-fno-ident’?

You don't need CPPFLAGS in a non-GCC environment with GNU make, because 
you
can easily do some GNU Make text processing to separate the preprocessor
options, if you need them.

Proof-of-concept demo:

$ make CFLAGS="-Wall -I/path/to/include -UMAC -DFOO=bar -o foo.o -lm"
cpp_only_flags == -I/path/to/include -UMAC -DFOO=bar

Source:

cpp_only_flags := $(foreach arg,                        \
                      $(CFLAGS),				\
                      $(or $(filter -D%,$(arg)),         \
		          $(filter -U%,$(arg)),         \
		          $(filter -I%,$(arg)),         \
		          $(filter -iquote%,$(arg)),    \
			  $(filter -W%,$(arg)),         \
			  $(filter -M%,$(arg))))        \
                   $(CPPFLAGS) # also pull this in

all:
	@echo cpp_only_flags == $(cpp_only_flags)

In an application, if you want separated C preprocessing flags, that
is what you have to do; you can't assume that you're built in
a distro environment which will hand them to you in a separate
CPPFLAGS variable.