Re: Yocto prelink status

Re: Yocto prelink status

[Richard Purdie]

Hi,

I was asked to take a look at this thread. I can probably represent the Yocto
Project and where we are with this.

> Recent discussions on glibc maillist about prelink supported is leading
> us to deprecate and remove its support on glibc.  Last year it was hinted
> that Yocto seems to be maintained the project, although without much 
> development [1].  

That is a fair summary. It is something some of our users do use but it hasn't
had much active development and we're aware there are bugs. We don't really have
people with the right expertise to address them right now unfortunately.

> A recent issue [2] with a change to support llvm lld to build glibc lead
> to a issue when prelink is used on the loader itself (which does not
> make sense since kernel will be responsible and it does not really
> care about prelink).
> 
> Finally the discussion on a recent patch proposal [3] to fix an issue
> for sym dependency makes me believe that prelink is de facto deprecated,
> since it does not work on most binaries in most architectures (20-30%
> of the system is capable of prelinking according to Mark) and it tends
> to get worse with new ABI extensions (for instance we plan to support
> DT_RELR on 2.36 [4]).  For instance, prelink on a Ubuntu 20.10 on
> x86_64 prelink -p shows that no binary could prelinked.

We did some investigation into this last year. You can see the discussion about
disabling it here:

https://lists.yoctoproject.org/g/poky/search?q=prelink

The bottom line was that with PIE enabled, it wasn't able to work. Since we
enable PIE by default, we did drop enabling prelink by default since it was not
doing anything useful in that case.

One of our community members did dive into it further and wrote up what he
found:

https://embed.endfa.net/yocto-cross-prelink-1/
https://embed.endfa.net/cross-prelink-time-memory/


Bottom line was that there was some very slight speed ups and the memory usage
was possibly worse with prelink but it was marginal. With other more specific
test cases and libraries with complex/large symbol tables, the results may be
different.

> Mark also stated that for embedded prelinked still yields some gain
> for loading time (specially for system boot) and memory consumption
> (although I am not sure if it really yield any gain for recent
> ABIs that avoid design such TEXTREL and writable PLT).
> 
> My plan is to remove prelink support for 2.35, at least to stir some
> discussion.  Unfortunately I don't have a straightforward replacement,
> I see prelink as very narrow solution that aims to fix some glibc design
> drawnbacks (such as not being focused for static linking, neither to 
> lower memory consumption). 
> 
> So I would to like if Yocto plan is to continue support prelink somewhat,
> or if you have any other ideas.

If glibc drop it, I doubt it would be viable for us and we'd just have to
follow.

I liked the fact that prelink did give some speed wins and that given Yocto
Project aims to support constrained use cases, having it available as a tool for
users was a good thing for us. We are struggling in a number of areas for
feature development/maintenance and since prelink is already struggling on our
side, we wouldn't go it alone with that.

I will post up something to the Yocto Project community highlighting we will
likely have to drop it. I did see a patch today from someone fixing prelink
support in a corner case so it is being used by people but I'm not sure anyone
would step up to help resolve the bigger issues on the glibc side or ours. We
would likely attempt to drop it sooner than later as we have an LTS coming up
and it may make sense for us to do that now if we know that is the direction
glibc upstream are going.

Cheers,

Richard


>   At glibc side, we are trying to fix
> some long-standing issues related to static linking (we deprecated
> static dlopen, and we working on the NSS support), but it still required
> some work to be on par with other static oriented libc.
> 
> [1] https://sourceware.org/pipermail/libc-alpha/2021-August/130404.html
> [2] https://sourceware.org/pipermail/libc-alpha/2021-October/131989.html
> [3] https://sourceware.org/pipermail/libc-alpha/2022-January/135189.html
> [4] https://sourceware.org/bugzilla/show_bug.cgi?id=27924



 

Re: Yocto prelink status

[Phil Blundell]

On Thu, Jan 20, 2022 at 10:51:31PM +0000, Richard Purdie via Libc-alpha wrote:
> Bottom line was that there was some very slight speed ups and the memory usage
> was possibly worse with prelink but it was marginal. With other more specific
> test cases and libraries with complex/large symbol tables, the results may be
> different.

The benefits of prelink tend to be more noticeable with large C++ DSOs because
they have a lot of RELATIVE relocs which can be eliminated entirely during
prelinking.  JUMP_SLOTs are resolved lazily in any case so the benefit of
doing that in advance is smaller, and most modern programs don't have a lot
of GLOB_DATs at all.

I'm slightly puzzled by the increase in memory footprint because prelink is
supposed to eliminate the dirty GOT pages caused by doing reloc fixups at
runtime.  But that does indeed seem to be what the numbers say.

Obviously there is a fundamental tension between ASLR and prelink.  I don't
think PIE is fundamentally incompatible with prelink (we can prelink PIC DSOs
for example) but if you're going to be loading at a different address each
time then clearly prelinking is not useful.  Given that the direction of
travel does seem to be towards ASLR everywhere it does seem that prelink is
going to be a bit of a losing proposition.

I would be slightly sad to see prelink go away entirely and if anybody on 
the Yocto side wanted to pick it up and make it work again then I would be 
happy to help them with that but I don't think I realistically have either 
the time or the enthusiasm to champion that effort myself.

p.

Re: Yocto prelink status

[Szabolcs Nagy]

The 01/21/2022 11:58, Phil Blundell via Libc-alpha wrote:
> On Thu, Jan 20, 2022 at 10:51:31PM +0000, Richard Purdie via Libc-alpha wrote:
> > Bottom line was that there was some very slight speed ups and the memory usage
> > was possibly worse with prelink but it was marginal. With other more specific
> > test cases and libraries with complex/large symbol tables, the results may be
> > different.
> 
> The benefits of prelink tend to be more noticeable with large C++ DSOs because
> they have a lot of RELATIVE relocs which can be eliminated entirely during
> prelinking.  JUMP_SLOTs are resolved lazily in any case so the benefit of
> doing that in advance is smaller, and most modern programs don't have a lot
> of GLOB_DATs at all.

glibc plans to support the DT_RELR format for more compact representation
of relative relocs. see bug 27924.

(i believe this is used on android/chromeos for the reason you cited.)

Re: Yocto prelink status

[Phil Blundell]

On Fri, Jan 21, 2022 at 04:23:20PM +0000, Szabolcs Nagy wrote:
> glibc plans to support the DT_RELR format for more compact representation
> of relative relocs. see bug 27924.
> 
> (i believe this is used on android/chromeos for the reason you cited.)

Yes.  There are actually three problems with large number of RELATIVE relocs:

1. They take up a lot of space in .rel.dyn or equivalent which causes binary
bloat.

2. All of them need to be processed at startup (you can't do them lazily) and
this adds delay before you get to main()

3. Data pages that have had relocations applied to them are now dirty and
unshareable so they occupy more space in memory.

As I understand DT_RELR it solves the first of these but not the other two.

Phil

Re: Yocto prelink status

[Richard Purdie]

On Fri, 2022-01-21 at 11:58 +0100, Phil Blundell wrote:
> On Thu, Jan 20, 2022 at 10:51:31PM +0000, Richard Purdie via Libc-alpha wrote:
> > Bottom line was that there was some very slight speed ups and the memory usage
> > was possibly worse with prelink but it was marginal. With other more specific
> > test cases and libraries with complex/large symbol tables, the results may be
> > different.
> 
> The benefits of prelink tend to be more noticeable with large C++ DSOs because
> they have a lot of RELATIVE relocs which can be eliminated entirely during
> prelinking.  JUMP_SLOTs are resolved lazily in any case so the benefit of
> doing that in advance is smaller, and most modern programs don't have a lot
> of GLOB_DATs at all.
> 
> I'm slightly puzzled by the increase in memory footprint because prelink is
> supposed to eliminate the dirty GOT pages caused by doing reloc fixups at
> runtime.  But that does indeed seem to be what the numbers say.
> 
> Obviously there is a fundamental tension between ASLR and prelink.  I don't
> think PIE is fundamentally incompatible with prelink (we can prelink PIC DSOs
> for example) but if you're going to be loading at a different address each
> time then clearly prelinking is not useful.  Given that the direction of
> travel does seem to be towards ASLR everywhere it does seem that prelink is
> going to be a bit of a losing proposition.
> 
> I would be slightly sad to see prelink go away entirely and if anybody on 
> the Yocto side wanted to pick it up and make it work again then I would be 
> happy to help them with that but I don't think I realistically have either 
> the time or the enthusiasm to champion that effort myself.

I did just want to loop back and agree with this last paragraph in particular.
I'm sad to see it being removed but I have too many other issues and not the
right amount of time to spend on this one. I suspect there could be ways to
configure things for better numbers but I doubt those configurations would be
acceptable so it doesn't really seem worth looking into them.

Whilst you mention making it work again, I'd imagine without the glibc support
that would be hard? I doubt this is something we could carry without upstream
help. 

In looking at removing it from Yocto Project, I did notice we filed a bug in
2016:

https://sourceware.org/bugzilla/show_bug.cgi?id=20488

which never made any progress, which is kind of a sign of the problem too :(. We
still carry that patch.

Cheers,

Richard

Re: Yocto prelink status

[Mark Hatle]

On 2/3/22 9:26 AM, Richard Purdie wrote:
> On Fri, 2022-01-21 at 11:58 +0100, Phil Blundell wrote:
>> On Thu, Jan 20, 2022 at 10:51:31PM +0000, Richard Purdie via Libc-alpha wrote:
>>> Bottom line was that there was some very slight speed ups and the memory usage
>>> was possibly worse with prelink but it was marginal. With other more specific
>>> test cases and libraries with complex/large symbol tables, the results may be
>>> different.
>>
>> The benefits of prelink tend to be more noticeable with large C++ DSOs because
>> they have a lot of RELATIVE relocs which can be eliminated entirely during
>> prelinking.  JUMP_SLOTs are resolved lazily in any case so the benefit of
>> doing that in advance is smaller, and most modern programs don't have a lot
>> of GLOB_DATs at all.
>>
>> I'm slightly puzzled by the increase in memory footprint because prelink is
>> supposed to eliminate the dirty GOT pages caused by doing reloc fixups at
>> runtime.  But that does indeed seem to be what the numbers say.
>>
>> Obviously there is a fundamental tension between ASLR and prelink.  I don't
>> think PIE is fundamentally incompatible with prelink (we can prelink PIC DSOs
>> for example) but if you're going to be loading at a different address each
>> time then clearly prelinking is not useful.  Given that the direction of
>> travel does seem to be towards ASLR everywhere it does seem that prelink is
>> going to be a bit of a losing proposition.
>>
>> I would be slightly sad to see prelink go away entirely and if anybody on
>> the Yocto side wanted to pick it up and make it work again then I would be
>> happy to help them with that but I don't think I realistically have either
>> the time or the enthusiasm to champion that effort myself.
> 
> I did just want to loop back and agree with this last paragraph in particular.
> I'm sad to see it being removed but I have too many other issues and not the
> right amount of time to spend on this one. I suspect there could be ways to
> configure things for better numbers but I doubt those configurations would be
> acceptable so it doesn't really seem worth looking into them.
> 
> Whilst you mention making it work again, I'd imagine without the glibc support
> that would be hard? I doubt this is something we could carry without upstream
> help.
> 
> In looking at removing it from Yocto Project, I did notice we filed a bug in
> 2016:
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=20488
> 
> which never made any progress, which is kind of a sign of the problem too :(. We
> still carry that patch.

This bug is partially what started the discussion on removing it.  Someone from 
libc-alpha contacted us trying to understand the bug report.

--Mark

> Cheers,
> 
> Richard

Re: Yocto prelink status

[Joseph Myers]

On Wed, 19 Jan 2022, Adhemerval Zanella via Libc-alpha wrote:

> Recent discussions on glibc maillist about prelink supported is leading
> us to deprecate and remove its support on glibc.  Last year it was hinted
> that Yocto seems to be maintained the project, although without much 
> development [1].  

Also it's not clear there are any active patch reviewers any more, at 
least based on my experience with AArch64 support fixes (sent to Mark in 
May 2020; the yocto@yoctoproject.org mailing list rejects email from 
non-subscribers, or did at that time; see https://github.com/jsm28/prelink 
branch cross_prelink_aarch64_fixes for a git tree with those fixes based 
on a merge of the cross_prelink branch into cross_prelink_aarch64).

-- 
Joseph S. Myers
joseph@codesourcery.com

Yocto prelink status

[Adhemerval Zanella]

Hi Nicolas,

Siddhesh told me to contact you since are involved on Yocto project.

Recent discussions on glibc maillist about prelink supported is leading
us to deprecate and remove its support on glibc.  Last year it was hinted
that Yocto seems to be maintained the project, although without much 
development [1].  

A recent issue [2] with a change to support llvm lld to build glibc lead
to a issue when prelink is used on the loader itself (which does not
make sense since kernel will be responsible and it does not really
care about prelink).

Finally the discussion on a recent patch proposal [3] to fix an issue
for sym dependency makes me believe that prelink is de facto deprecated,
since it does not work on most binaries in most architectures (20-30%
of the system is capable of prelinking according to Mark) and it tends
to get worse with new ABI extensions (for instance we plan to support
DT_RELR on 2.36 [4]).  For instance, prelink on a Ubuntu 20.10 on
x86_64 prelink -p shows that no binary could prelinked.

Mark also stated that for embedded prelinked still yields some gain
for loading time (specially for system boot) and memory consumption
(although I am not sure if it really yield any gain for recent
ABIs that avoid design such TEXTREL and writable PLT).

My plan is to remove prelink support for 2.35, at least to stir some
discussion.  Unfortunately I don't have a straightforward replacement,
I see prelink as very narrow solution that aims to fix some glibc design
drawnbacks (such as not being focused for static linking, neither to 
lower memory consumption). 

So I would to like if Yocto plan is to continue support prelink somewhat,
or if you have any other ideas.  At glibc side, we are trying to fix
some long-standing issues related to static linking (we deprecated
static dlopen, and we working on the NSS support), but it still required
some work to be on par with other static oriented libc.

[1] https://sourceware.org/pipermail/libc-alpha/2021-August/130404.html
[2] https://sourceware.org/pipermail/libc-alpha/2021-October/131989.html
[3] https://sourceware.org/pipermail/libc-alpha/2022-January/135189.html
[4] https://sourceware.org/bugzilla/show_bug.cgi?id=27924