Aligning tcmalloc with glibc 2.35 rseq ABI

Aligning tcmalloc with glibc 2.35 rseq ABI

[Mathieu Desnoyers]

Hi Chris,

You will probably want to have a look at the userspace rseq ABI exposed by glibc 2.35 to ensure
tcmalloc becomes compatible with it.

If it helps, you can have a look at how I modified librseq to play nicely with glibc 2.35:

https://git.kernel.org/pub/scm/libs/librseq/librseq.git/

Most relevant bits:

https://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree/src/rseq.c#n108

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Chris Kennelly]

Thanks for the heads up.

I did have a question about whether the new protocol would introduce
an extra memory reference while initializing a critical section.

* With initial-exec TLS, I can directly reference __rseq_abi.
* With the new ABI, I might need to ask glibc for the address of the
registered rseq structure in its thread data.

I saw https://lkml.org/lkml/2022/1/24/859 mention using %gs, but it
also mentions consulting "rseq_offset" in the address calculation.  Is
rseq_offset something I can resolve to a constant or would I need to
access a global for it, or does glibc take care of initializing %gs to
the address of the structure registered with the kernel (for the
current thread)?

Thanks,
Chris


On Tue, Feb 1, 2022 at 9:58 AM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> Hi Chris,
>
> You will probably want to have a look at the userspace rseq ABI exposed by glibc 2.35 to ensure
> tcmalloc becomes compatible with it.
>
> If it helps, you can have a look at how I modified librseq to play nicely with glibc 2.35:
>
> https://git.kernel.org/pub/scm/libs/librseq/librseq.git/
>
> Most relevant bits:
>
> https://git.kernel.org/pub/scm/libs/librseq/librseq.git/tree/src/rseq.c#n108
>
> Thanks,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Florian Weimer]

* Chris Kennelly:

> Thanks for the heads up.
>
> I did have a question about whether the new protocol would introduce
> an extra memory reference while initializing a critical section.
>
> * With initial-exec TLS, I can directly reference __rseq_abi.
> * With the new ABI, I might need to ask glibc for the address of the
> registered rseq structure in its thread data.

You can write __rseq_offset to a static/hidden variable in an ELF
constructor, and then use pretty much the same assembler sequences as
for initial-exec TLS on most architectures.

Thanks,
Florian

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Florian Weimer]

* Florian Weimer:

> * Chris Kennelly:
>
>> Thanks for the heads up.
>>
>> I did have a question about whether the new protocol would introduce
>> an extra memory reference while initializing a critical section.
>>
>> * With initial-exec TLS, I can directly reference __rseq_abi.
>> * With the new ABI, I might need to ask glibc for the address of the
>> registered rseq structure in its thread data.
>
> You can write __rseq_offset to a static/hidden variable in an ELF
> constructor, and then use pretty much the same assembler sequences as
> for initial-exec TLS on most architectures.

And now I'm kind of worried that we should be using ptrdiff_t for
__rseq_offset because that's what the initial-exec relocations use. 8-/

Thanks,
Florian

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Mathieu Desnoyers]

----- On Feb 2, 2022, at 3:41 AM, Florian Weimer fweimer@redhat.com wrote:

> * Florian Weimer:
> 
>> * Chris Kennelly:
>>
>>> Thanks for the heads up.
>>>
>>> I did have a question about whether the new protocol would introduce
>>> an extra memory reference while initializing a critical section.
>>>
>>> * With initial-exec TLS, I can directly reference __rseq_abi.
>>> * With the new ABI, I might need to ask glibc for the address of the
>>> registered rseq structure in its thread data.
>>
>> You can write __rseq_offset to a static/hidden variable in an ELF
>> constructor, and then use pretty much the same assembler sequences as
>> for initial-exec TLS on most architectures.
> 
> And now I'm kind of worried that we should be using ptrdiff_t for
> __rseq_offset because that's what the initial-exec relocations use. 8-/

I suspect the underlying question here is: how likely is it that a libc
requires an offset of more than 2GB either way from the thread pointer
to allocate its rseq thread area on a 64-bit architecture ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Mathieu Desnoyers]

----- On Feb 2, 2022, at 6:36 AM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote:

> ----- On Feb 2, 2022, at 3:41 AM, Florian Weimer fweimer@redhat.com wrote:
> 
>> * Florian Weimer:
>> 
>>> * Chris Kennelly:
>>>
>>>> Thanks for the heads up.
>>>>
>>>> I did have a question about whether the new protocol would introduce
>>>> an extra memory reference while initializing a critical section.
>>>>
>>>> * With initial-exec TLS, I can directly reference __rseq_abi.
>>>> * With the new ABI, I might need to ask glibc for the address of the
>>>> registered rseq structure in its thread data.
>>>
>>> You can write __rseq_offset to a static/hidden variable in an ELF
>>> constructor, and then use pretty much the same assembler sequences as
>>> for initial-exec TLS on most architectures.
>> 
>> And now I'm kind of worried that we should be using ptrdiff_t for
>> __rseq_offset because that's what the initial-exec relocations use. 8-/
> 
> I suspect the underlying question here is: how likely is it that a libc
> requires an offset of more than 2GB either way from the thread pointer
> to allocate its rseq thread area on a 64-bit architecture ?

More to the point: is ptrdiff_t the correct type here ? I think so.
Do we want to revert the ABI and wait another 6 months before we
bring back rseq into glibc just for this ? I'm not sure this limitation
justifies it.

So if there is a quick way to fix that before the official 2.35 release,
I'm all for it, otherwise I cannot say that __rseq_offset being an "int"
rather than a "ptrdiff_t" will make much real-life difference (unless
I'm proven wrong). But we will be stuck with this quirk forever.

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Florian Weimer]

* Mathieu Desnoyers:

> More to the point: is ptrdiff_t the correct type here ? I think so.
> Do we want to revert the ABI and wait another 6 months before we
> bring back rseq into glibc just for this ? I'm not sure this limitation
> justifies it.
>
> So if there is a quick way to fix that before the official 2.35 release,
> I'm all for it, otherwise I cannot say that __rseq_offset being an "int"
> rather than a "ptrdiff_t" will make much real-life difference (unless
> I'm proven wrong). But we will be stuck with this quirk forever.

I'm going to post a patch.  It's fairly small.

Thanks,
Florian

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Carlos O'Donell]

On 2/2/22 10:01, Florian Weimer via Libc-alpha wrote:
> * Mathieu Desnoyers:
> 
>> More to the point: is ptrdiff_t the correct type here ? I think so.
>> Do we want to revert the ABI and wait another 6 months before we
>> bring back rseq into glibc just for this ? I'm not sure this limitation
>> justifies it.
>>
>> So if there is a quick way to fix that before the official 2.35 release,
>> I'm all for it, otherwise I cannot say that __rseq_offset being an "int"
>> rather than a "ptrdiff_t" will make much real-life difference (unless
>> I'm proven wrong). But we will be stuck with this quirk forever.
> 
> I'm going to post a patch.  It's fairly small.

I'll review this as glibc release manager for glibc 2.35.

We'll get this right before I cut the release.

-- 
Cheers,
Carlos.

Re: Aligning tcmalloc with glibc 2.35 rseq ABI

[Carlos O'Donell]

On 2/2/22 12:31, Carlos O'Donell wrote:
> On 2/2/22 10:01, Florian Weimer via Libc-alpha wrote:
>> * Mathieu Desnoyers:
>>
>>> More to the point: is ptrdiff_t the correct type here ? I think so.
>>> Do we want to revert the ABI and wait another 6 months before we
>>> bring back rseq into glibc just for this ? I'm not sure this limitation
>>> justifies it.
>>>
>>> So if there is a quick way to fix that before the official 2.35 release,
>>> I'm all for it, otherwise I cannot say that __rseq_offset being an "int"
>>> rather than a "ptrdiff_t" will make much real-life difference (unless
>>> I'm proven wrong). But we will be stuck with this quirk forever.
>>
>> I'm going to post a patch.  It's fairly small.
> 
> I'll review this as glibc release manager for glibc 2.35.
> 
> We'll get this right before I cut the release.
 
I've reviewed the static linker relocation designs, and aside from the odd-duck
for ia64, we've used ptrdiff_t sized relocations, nominally 64-bits for the
64-bit targets (though on AArch64 -mtls-size admits at most 48-bits).

The ptrdiff_t change will be a part of the glibc 2.35 release:

commit 6c33b018438ee799c29486f21d43d8100bdbd597
Author: Florian Weimer <fweimer@redhat.com>
Date:   Wed Feb 2 22:37:20 2022 +0100

    Linux: Use ptrdiff_t for __rseq_offset
    
    This matches the data size initial-exec relocations use on most
    targets.
    
    Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
    Reviewed-by: Carlos O'Donell <carlos@redhat.com>

-- 
Cheers,
Carlos.