* Seeking advise on most portable way to detect 64-bit off_t
@ 2024-03-01 21:06 Aliaksey Kandratsenka
2024-03-04 12:04 ` Adhemerval Zanella Netto
0 siblings, 1 reply; 4+ messages in thread
From: Aliaksey Kandratsenka @ 2024-03-01 21:06 UTC (permalink / raw)
To: libc-help; +Cc: Aliaksiej Kandracienka
[-- Attachment #1: Type: text/plain, Size: 2288 bytes --]
Hi.
In gperftools we have the feature to hook mmap calls which works by
interposing over glibc's mmap. So it has to deal with off_t complexities.
We don't use that offset argument, other than just passing it to real mmap
implementation. I am also trying to support different Linux libc-s just for
completeness. And musl being new enough and wise enough has always had
64-bit off_t (bionic hasn't and there is really no excuse...)
So with that I need some means to detect 32-bit off_t that works across
multiple libcs and has the highest hope of working in the future.
By looking around, I choose to detect 32-bit-ness of off_t by looking at
semi-obscure define _POSIX_V7_ILP32_OFF32:
https://github.com/gperftools/gperftools/blob/37b59a206e36b405dcb2ac09d502875dd629a80b/src/mmap_hook.cc#L275
It does seem to do the right thing across implementations I checked. Well
mostly.
These days, Debian folk are doing a massive 64-bit time_t transition (which
also includes mass-enabling 64-bit off_t, at last) and they're rebuilding
everything with _FILE_OFFSET_BITS=64. And my "approach" fails.
https://buildd.debian.org/status/fetch.php?pkg=google-perftools&arch=armel&ver=2.15-1.1&stamp=1709166723&file=log
Glibc doesn't adjust _POSIX_V7_ILP32_{BIGOFF,OFF32} defines based on
_FILE_OFFSET_BITS. Perhaps rightly so. But with that situation I need some
other, and hopefully more robust means to detect off_t width.
I am thinking of 3 options:
* Keep _POSIX_V7 thingy but also add a check for _FILE_OFFSET_BITS == 64.
And then behave as if off_t is 32-bit.
* #undef _FILE_OFFSET_BITS at the first line in mmap_hooks.cc. It will
cause glibc on those legacy 32-bit off_t systems to define 32-bit off_t and
give me right defines and my code will define mmap symbol with 32-bit
offset arg and mmap64 symbol with 64-bit offset. And thus matching glibc.
* just manually hardcode knowledge that glibc +
{i386,arm{el,hf},mips32,ppc32} has 32-bit off_t. But that leaves the
question of how to deal with e.g. bionic (well, I could in principle say,
bionic already being broken enough is only supported for 64-bit systems,
which is where android systems are moving anyways)
I am looking for any comments or suggestions. Particularly I'd like my fix
to be robust w.r.t. any further glibc evolution.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Seeking advise on most portable way to detect 64-bit off_t
2024-03-01 21:06 Seeking advise on most portable way to detect 64-bit off_t Aliaksey Kandratsenka
@ 2024-03-04 12:04 ` Adhemerval Zanella Netto
2024-03-07 18:31 ` Aliaksey Kandratsenka
0 siblings, 1 reply; 4+ messages in thread
From: Adhemerval Zanella Netto @ 2024-03-04 12:04 UTC (permalink / raw)
To: Aliaksey Kandratsenka, libc-help
On 01/03/24 18:06, Aliaksey Kandratsenka via Libc-help wrote:
> Hi.
>
> In gperftools we have the feature to hook mmap calls which works by
> interposing over glibc's mmap. So it has to deal with off_t complexities.
> We don't use that offset argument, other than just passing it to real mmap
> implementation. I am also trying to support different Linux libc-s just for
> completeness. And musl being new enough and wise enough has always had
> 64-bit off_t (bionic hasn't and there is really no excuse...)
>
> So with that I need some means to detect 32-bit off_t that works across
> multiple libcs and has the highest hope of working in the future.
>
> By looking around, I choose to detect 32-bit-ness of off_t by looking at
> semi-obscure define _POSIX_V7_ILP32_OFF32:
> https://github.com/gperftools/gperftools/blob/37b59a206e36b405dcb2ac09d502875dd629a80b/src/mmap_hook.cc#L275
>
> It does seem to do the right thing across implementations I checked. Well
> mostly.
>
> These days, Debian folk are doing a massive 64-bit time_t transition (which
> also includes mass-enabling 64-bit off_t, at last) and they're rebuilding
> everything with _FILE_OFFSET_BITS=64. And my "approach" fails.
> https://buildd.debian.org/status/fetch.php?pkg=google-perftools&arch=armel&ver=2.15-1.1&stamp=1709166723&file=log
>
> Glibc doesn't adjust _POSIX_V7_ILP32_{BIGOFF,OFF32} defines based on
> _FILE_OFFSET_BITS. Perhaps rightly so. But with that situation I need some
> other, and hopefully more robust means to detect off_t width.
>
> I am thinking of 3 options:
>
> * Keep _POSIX_V7 thingy but also add a check for _FILE_OFFSET_BITS == 64.
> And then behave as if off_t is 32-bit.
> * #undef _FILE_OFFSET_BITS at the first line in mmap_hooks.cc. It will
> cause glibc on those legacy 32-bit off_t systems to define 32-bit off_t and
> give me right defines and my code will define mmap symbol with 32-bit
> offset arg and mmap64 symbol with 64-bit offset. And thus matching glibc.
> * just manually hardcode knowledge that glibc +
You need to undefine _FILE_OFFSET_BITS to get the expected types for
off_t and off64_t on the mmap_hook.cc TU, since the whole idea is to
override the symbol. You can also try to poke on glibc internals and
use __off_t and __off64_t, which would not be redefined depending the
the preprocessor (but tying to internal implementation is always a bad
idea).
So I would recommend this option, which seems to what you did and it is
what sanitizers do also [1].
> {i386,arm{el,hf},mips32,ppc32} has 32-bit off_t. But that leaves the
> question of how to deal with e.g. bionic (well, I could in principle say,
> bionic already being broken enough is only supported for 64-bit systems,
> which is where android systems are moving anyways)
>
> I am looking for any comments or suggestions. Particularly I'd like my fix
> to be robust w.r.t. any further glibc evolution.
[1] https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp#L15
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Seeking advise on most portable way to detect 64-bit off_t
2024-03-04 12:04 ` Adhemerval Zanella Netto
@ 2024-03-07 18:31 ` Aliaksey Kandratsenka
2024-03-12 12:06 ` Adhemerval Zanella Netto
0 siblings, 1 reply; 4+ messages in thread
From: Aliaksey Kandratsenka @ 2024-03-07 18:31 UTC (permalink / raw)
To: Adhemerval Zanella Netto; +Cc: libc-help
[-- Attachment #1: Type: text/plain, Size: 3901 bytes --]
Thanks.
So it turns out I also needed to undef _TIME_BITS. Since glibc errors
whenever _TIME_BITS are set and _FILE_OFFSET_BITS aren't. No complaints.
IMHO it is a very reasonable thing that glibc does there. But it makes
things slightly riskier on our end.
I have a question about a potentially longer term future. Do you guys plan
to, say, default to _TIME_BITS=64 and _FILE_OFFSET_BITS=64 at any point?
Asking because it would break my code (thankfully, it won't be silent
breaking, but still).
Should I consider something else to make my code more robust ?
On Mon, Mar 4, 2024 at 7:04 AM Adhemerval Zanella Netto <
adhemerval.zanella@linaro.org> wrote:
>
>
> On 01/03/24 18:06, Aliaksey Kandratsenka via Libc-help wrote:
> > Hi.
> >
> > In gperftools we have the feature to hook mmap calls which works by
> > interposing over glibc's mmap. So it has to deal with off_t complexities.
> > We don't use that offset argument, other than just passing it to real
> mmap
> > implementation. I am also trying to support different Linux libc-s just
> for
> > completeness. And musl being new enough and wise enough has always had
> > 64-bit off_t (bionic hasn't and there is really no excuse...)
> >
> > So with that I need some means to detect 32-bit off_t that works across
> > multiple libcs and has the highest hope of working in the future.
> >
> > By looking around, I choose to detect 32-bit-ness of off_t by looking at
> > semi-obscure define _POSIX_V7_ILP32_OFF32:
> >
> https://github.com/gperftools/gperftools/blob/37b59a206e36b405dcb2ac09d502875dd629a80b/src/mmap_hook.cc#L275
> >
> > It does seem to do the right thing across implementations I checked. Well
> > mostly.
> >
> > These days, Debian folk are doing a massive 64-bit time_t transition
> (which
> > also includes mass-enabling 64-bit off_t, at last) and they're rebuilding
> > everything with _FILE_OFFSET_BITS=64. And my "approach" fails.
> >
> https://buildd.debian.org/status/fetch.php?pkg=google-perftools&arch=armel&ver=2.15-1.1&stamp=1709166723&file=log
> >
> > Glibc doesn't adjust _POSIX_V7_ILP32_{BIGOFF,OFF32} defines based on
> > _FILE_OFFSET_BITS. Perhaps rightly so. But with that situation I need
> some
> > other, and hopefully more robust means to detect off_t width.
> >
> > I am thinking of 3 options:
> >
> > * Keep _POSIX_V7 thingy but also add a check for _FILE_OFFSET_BITS == 64.
> > And then behave as if off_t is 32-bit.
> > * #undef _FILE_OFFSET_BITS at the first line in mmap_hooks.cc. It will
> > cause glibc on those legacy 32-bit off_t systems to define 32-bit off_t
> and
> > give me right defines and my code will define mmap symbol with 32-bit
> > offset arg and mmap64 symbol with 64-bit offset. And thus matching glibc.
> > * just manually hardcode knowledge that glibc +
>
> You need to undefine _FILE_OFFSET_BITS to get the expected types for
> off_t and off64_t on the mmap_hook.cc TU, since the whole idea is to
> override the symbol. You can also try to poke on glibc internals and
> use __off_t and __off64_t, which would not be redefined depending the
> the preprocessor (but tying to internal implementation is always a bad
> idea).
>
> So I would recommend this option, which seems to what you did and it is
> what sanitizers do also [1].
>
> > {i386,arm{el,hf},mips32,ppc32} has 32-bit off_t. But that leaves the
> > question of how to deal with e.g. bionic (well, I could in principle say,
> > bionic already being broken enough is only supported for 64-bit systems,
> > which is where android systems are moving anyways)
> >
> > I am looking for any comments or suggestions. Particularly I'd like my
> fix
> > to be robust w.r.t. any further glibc evolution.
>
> [1]
> https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp#L15
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Seeking advise on most portable way to detect 64-bit off_t
2024-03-07 18:31 ` Aliaksey Kandratsenka
@ 2024-03-12 12:06 ` Adhemerval Zanella Netto
0 siblings, 0 replies; 4+ messages in thread
From: Adhemerval Zanella Netto @ 2024-03-12 12:06 UTC (permalink / raw)
To: Aliaksey Kandratsenka; +Cc: libc-help
On 07/03/24 15:31, Aliaksey Kandratsenka wrote:
> Thanks.
>
> So it turns out I also needed to undef _TIME_BITS. Since glibc errors whenever _TIME_BITS are set and _FILE_OFFSET_BITS aren't. No complaints. IMHO it is a very reasonable thing that glibc does there. But it makes things slightly riskier on our end.
>
> I have a question about a potentially longer term future. Do you guys plan to, say, default to _TIME_BITS=64 and _FILE_OFFSET_BITS=64 at any point? Asking because it would break my code (thankfully, it won't be silent breaking, but still).
It was on my plan when I we started to effectively work on 64 bit
time_t, but we decided this would case too much potential breakage
(as debian is finding out recently). So I think it is unlikely that
we will even change it.
>
> Should I consider something else to make my code more robust ?
>
> On Mon, Mar 4, 2024 at 7:04 AM Adhemerval Zanella Netto <adhemerval.zanella@linaro.org <mailto:adhemerval.zanella@linaro.org>> wrote:
>
>
>
> On 01/03/24 18:06, Aliaksey Kandratsenka via Libc-help wrote:
> > Hi.
> >
> > In gperftools we have the feature to hook mmap calls which works by
> > interposing over glibc's mmap. So it has to deal with off_t complexities.
> > We don't use that offset argument, other than just passing it to real mmap
> > implementation. I am also trying to support different Linux libc-s just for
> > completeness. And musl being new enough and wise enough has always had
> > 64-bit off_t (bionic hasn't and there is really no excuse...)
> >
> > So with that I need some means to detect 32-bit off_t that works across
> > multiple libcs and has the highest hope of working in the future.
> >
> > By looking around, I choose to detect 32-bit-ness of off_t by looking at
> > semi-obscure define _POSIX_V7_ILP32_OFF32:
> > https://github.com/gperftools/gperftools/blob/37b59a206e36b405dcb2ac09d502875dd629a80b/src/mmap_hook.cc#L275 <https://github.com/gperftools/gperftools/blob/37b59a206e36b405dcb2ac09d502875dd629a80b/src/mmap_hook.cc#L275>
> >
> > It does seem to do the right thing across implementations I checked. Well
> > mostly.
> >
> > These days, Debian folk are doing a massive 64-bit time_t transition (which
> > also includes mass-enabling 64-bit off_t, at last) and they're rebuilding
> > everything with _FILE_OFFSET_BITS=64. And my "approach" fails.
> > https://buildd.debian.org/status/fetch.php?pkg=google-perftools&arch=armel&ver=2.15-1.1&stamp=1709166723&file=log <https://buildd.debian.org/status/fetch.php?pkg=google-perftools&arch=armel&ver=2.15-1.1&stamp=1709166723&file=log>
> >
> > Glibc doesn't adjust _POSIX_V7_ILP32_{BIGOFF,OFF32} defines based on
> > _FILE_OFFSET_BITS. Perhaps rightly so. But with that situation I need some
> > other, and hopefully more robust means to detect off_t width.
> >
> > I am thinking of 3 options:
> >
> > * Keep _POSIX_V7 thingy but also add a check for _FILE_OFFSET_BITS == 64.
> > And then behave as if off_t is 32-bit.
> > * #undef _FILE_OFFSET_BITS at the first line in mmap_hooks.cc. It will
> > cause glibc on those legacy 32-bit off_t systems to define 32-bit off_t and
> > give me right defines and my code will define mmap symbol with 32-bit
> > offset arg and mmap64 symbol with 64-bit offset. And thus matching glibc.
> > * just manually hardcode knowledge that glibc +
>
> You need to undefine _FILE_OFFSET_BITS to get the expected types for
> off_t and off64_t on the mmap_hook.cc TU, since the whole idea is to
> override the symbol. You can also try to poke on glibc internals and
> use __off_t and __off64_t, which would not be redefined depending the
> the preprocessor (but tying to internal implementation is always a bad
> idea).
>
> So I would recommend this option, which seems to what you did and it is
> what sanitizers do also [1].
>
> > {i386,arm{el,hf},mips32,ppc32} has 32-bit off_t. But that leaves the
> > question of how to deal with e.g. bionic (well, I could in principle say,
> > bionic already being broken enough is only supported for 64-bit systems,
> > which is where android systems are moving anyways)
> >
> > I am looking for any comments or suggestions. Particularly I'd like my fix
> > to be robust w.r.t. any further glibc evolution.
>
> [1] https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp#L15 <https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp#L15>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-03-12 12:06 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-01 21:06 Seeking advise on most portable way to detect 64-bit off_t Aliaksey Kandratsenka
2024-03-04 12:04 ` Adhemerval Zanella Netto
2024-03-07 18:31 ` Aliaksey Kandratsenka
2024-03-12 12:06 ` Adhemerval Zanella Netto
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).