[RFC] Should we declare errno with __thread on x86?

[RFC] Should we declare errno with __thread on x86?

[H.J. Lu]

errno is defined in glibc as an TLS variable on x86,  Shouldn't we
declare it as

extern __thread int errno;

instead of

#   define errno (*__errno_location ())

to avoid function call overhead?


-- 
H.J.

Re: [RFC] Should we declare errno with __thread on x86?

[Roland McGrath]

It's not machine-specific (or even really OS-specific).
All non-Hurd configurations use the TLS errno.
I don't know why you mentioned x86 specifically.

References from the libc source tree already do use TLS directly
(see include/errno.h).

So I take it you are talking about the public <errno.h> declaration.
Is that what you meant?

The simple reason we haven't done this before is because we've never done
this before.  There are no TLS symbols in our public ABIs.  When we first
changed the internal implementation to use TLS, compilers' and other tools'
support for TLS was still very new and unreliable.  Conservatism has kept
us where we are since then.

Since what you are suggesting is a change motivated solely by performance,
we should see some real-world or benchmark measurements of some kind to
justify it.  A micro-benchmark of reading errno is not that interesting,
especially since naive intuition would expect that errno is examined only
(or almost entirely) in error paths that are not hot paths and so that
same intuition would expect that real-world impact would be negligible.

If real-world performance measurement does in fact justify it, we'll still
need to be very circumspect about potential pitfalls (applications built
with older compilers, etc).  We'll need to put a lot of thought into it
collectively to achieve confidence that it's a safe and sensible change for
all our users.

I expect that whatever we conclude about errno will apply just as well to
h_errno (and maybe _res).


Thanks,
Roland

Re: [RFC] Should we declare errno with __thread on x86?

[Torvald Riegel]

On Fri, 2016-03-18 at 16:14 -0700, Roland McGrath wrote:
> If real-world performance measurement does in fact justify it, we'll still
> need to be very circumspect about potential pitfalls (applications built
> with older compilers, etc).  We'll need to put a lot of thought into it
> collectively to achieve confidence that it's a safe and sensible change for
> all our users.

I agree that this would need a thorough investigation and assessment of
risks of doing that.  One risk I see is that it would likely make it
harder for us to support errno on threads of execution that are not OS
threads (eg, coroutines, or under work-stealing implementations).  It's
just an integer, but generally I'd prefer not to expose more TLS than
necessary.

Re: [RFC] Should we declare errno with __thread on x86?

[Zack Weinberg]

On Fri, Mar 18, 2016 at 7:14 PM, Roland McGrath <roland@hack.frob.com> wrote:
> Since what you are suggesting is a change motivated solely by performance,
> we should see some real-world or benchmark measurements of some kind to
> justify it.  A micro-benchmark of reading errno is not that interesting,
> especially since naive intuition would expect that errno is examined only
> (or almost entirely) in error paths that are not hot paths and so that
> same intuition would expect that real-world impact would be negligible.

Also, even when errno is accessed in a tight loop in real code (e.g.
parsing a bunch of numbers with strtol-family functions), gcc (tested
as far back as 4.8) is able to hoist the call to __errno_location out
of the loop, probably due to the __attribute__((const)) annotation.

zw

Re: [RFC] Should we declare errno with __thread on x86?

[Florian Weimer]

On 04/12/2016 08:30 PM, Torvald Riegel wrote:
> On Fri, 2016-03-18 at 16:14 -0700, Roland McGrath wrote:
>> If real-world performance measurement does in fact justify it, we'll still
>> need to be very circumspect about potential pitfalls (applications built
>> with older compilers, etc).  We'll need to put a lot of thought into it
>> collectively to achieve confidence that it's a safe and sensible change for
>> all our users.
>
> I agree that this would need a thorough investigation and assessment of
> risks of doing that.  One risk I see is that it would likely make it
> harder for us to support errno on threads of execution that are not OS
> threads (eg, coroutines, or under work-stealing implementations).

It will not be worse than what we currently have because 
__errno_location is declared const.  Already today, resuming a call 
stack on a thread other the one on which it was suspended results in 
undefined behavior.  This also applies to most forms of TLS access (if 
not all of them), again due to compiler caching of TLS variable addresses.

Another consideration is sharing of errno across dlmopen/static dlopen 
boundaries, which is exactly the opposite (more sharing instead of 
less).  Putting errno into the TCB would likely achieve that, which is 
easier with the __errno_location scheme, I think.  But it can be done 
with TLS variable access as well because we control the dynamic linker.

(Your job is to make sure that we can implement C++ coroutines with 
something substantially more lightweight than threads, while still 
keeping all our existing binaries, particularly libraries. :)

Florian

Re: [RFC] Should we declare errno with __thread on x86?

[Florian Weimer]

On 03/18/2016 10:42 PM, H.J. Lu wrote:
> errno is defined in glibc as an TLS variable on x86,  Shouldn't we
> declare it as
>
> extern __thread int errno;
>
> instead of
>
> #   define errno (*__errno_location ())
>
> to avoid function call overhead?

Considering that errno is mostly accessed on error paths, I think more 
compact code is better.  It's not obvious that TLS variable access is a 
win in all cases according to that metric.

Florian

Re: [RFC] Should we declare errno with __thread on x86?

[Torvald Riegel]

On Wed, 2016-04-13 at 11:17 +0200, Florian Weimer wrote:
> On 04/12/2016 08:30 PM, Torvald Riegel wrote:
> > On Fri, 2016-03-18 at 16:14 -0700, Roland McGrath wrote:
> >> If real-world performance measurement does in fact justify it, we'll still
> >> need to be very circumspect about potential pitfalls (applications built
> >> with older compilers, etc).  We'll need to put a lot of thought into it
> >> collectively to achieve confidence that it's a safe and sensible change for
> >> all our users.
> >
> > I agree that this would need a thorough investigation and assessment of
> > risks of doing that.  One risk I see is that it would likely make it
> > harder for us to support errno on threads of execution that are not OS
> > threads (eg, coroutines, or under work-stealing implementations).
> 
> It will not be worse than what we currently have because 
> __errno_location is declared const.  Already today, resuming a call 
> stack on a thread other the one on which it was suspended results in 
> undefined behavior.  This also applies to most forms of TLS access (if 
> not all of them), again due to compiler caching of TLS variable addresses.

That is a an interesting point, good catch.  But I disagree with your
conclusion.

The const attribute is specified as asserting that the function does not
examine any data except the arguments.  __errno_location has no
arguments, so it would have to return the same values *every time*.
This works in a single-threaded program, but not in a multi-threaded
one.  Thus, I think that strictly speaking, it should not be const.

We could argue that this magically is meant to always be in the context
of a specific thread.  Ignoring that GCC doesn't define threads itself
(especially in something like NPTL which is about creating a notion of
threads), we could still assume that this works because in practice, the
compiler and its passes can't leak knowledge across a function used in
one thread and other one used in another thread.  But then this would
still hold for coroutines, I guess, if we can manage to have
__errno_location return a coroutine-specific location if this is
executed in a coroutine.  It would limit the amount of inlining we can
do though.

> (Your job is to make sure that we can implement C++ coroutines with 
> something substantially more lightweight than threads, while still 
> keeping all our existing binaries, particularly libraries. :)

:)

TLS, signals, and all these things make this challenging.  Maybe some
execution agents will only work with C++ code; it could happen that we
just can't preserve some of the old semantics...

Re: [RFC] Should we declare errno with __thread on x86?

[Szabolcs Nagy]

On 13/04/16 13:37, Torvald Riegel wrote:
> On Wed, 2016-04-13 at 11:17 +0200, Florian Weimer wrote:
>> On 04/12/2016 08:30 PM, Torvald Riegel wrote:
>>> On Fri, 2016-03-18 at 16:14 -0700, Roland McGrath wrote:
>>>> If real-world performance measurement does in fact justify it, we'll still
>>>> need to be very circumspect about potential pitfalls (applications built
>>>> with older compilers, etc).  We'll need to put a lot of thought into it
>>>> collectively to achieve confidence that it's a safe and sensible change for
>>>> all our users.
>>>
>>> I agree that this would need a thorough investigation and assessment of
>>> risks of doing that.  One risk I see is that it would likely make it
>>> harder for us to support errno on threads of execution that are not OS
>>> threads (eg, coroutines, or under work-stealing implementations).
>>
>> It will not be worse than what we currently have because 
>> __errno_location is declared const.  Already today, resuming a call 
>> stack on a thread other the one on which it was suspended results in 
>> undefined behavior.  This also applies to most forms of TLS access (if 
>> not all of them), again due to compiler caching of TLS variable addresses.
> 
> That is a an interesting point, good catch.  But I disagree with your
> conclusion.
> 
> The const attribute is specified as asserting that the function does not
> examine any data except the arguments.  __errno_location has no
> arguments, so it would have to return the same values *every time*.
> This works in a single-threaded program, but not in a multi-threaded
> one.  Thus, I think that strictly speaking, it should not be const.
> 

that was my conclusion too, reading the second point in
https://gcc.gnu.org/ml/gcc/2015-09/msg00354.html
but there was no further discussion then..

> We could argue that this magically is meant to always be in the context
> of a specific thread.  Ignoring that GCC doesn't define threads itself
> (especially in something like NPTL which is about creating a notion of
> threads), we could still assume that this works because in practice, the
> compiler and its passes can't leak knowledge across a function used in
> one thread and other one used in another thread.  But then this would
> still hold for coroutines, I guess, if we can manage to have
> __errno_location return a coroutine-specific location if this is
> executed in a coroutine.  It would limit the amount of inlining we can
> do though.
> 
>> (Your job is to make sure that we can implement C++ coroutines with 
>> something substantially more lightweight than threads, while still 
>> keeping all our existing binaries, particularly libraries. :)
> 
> :)
> 
> TLS, signals, and all these things make this challenging.  Maybe some
> execution agents will only work with C++ code; it could happen that we
> just can't preserve some of the old semantics...
> 
>