Re: How to set a system-wide library path policy?

Re: How to set a system-wide library path policy?

[Carlos O'Donell]

On 6/7/19 10:53 AM, Jeffrey Walton wrote:
> I'm having trouble figuring out how to setup library path policy on Fedora 29.

The system loader is in glibc. Adding libc-help to the CC.

> I need to articulate to the system loader:
> 
> 1. programs in /bin must only link to libs in /lib64
> 2. programs in /usr/bin must only link to libs in /usr/lib64
> 3. programs in /usr/local/bin may use libraries in /usr/local/lib64 or
> /usr/lib64

OK.

> As far as I know, the distro supplies (1) and (2). They know nothing
> about my libraries in /usr/local. When the distro is ready, it will
> push updated programs and libraries as needed. The distro binaries
> should not use my libraries.

Why would your distro binaries use your libraries? Such libraries are
not present at static link time when they are built in the Feodra builders.

The only reason your distro libraries would be used is if you used
LD_LIBRARY_PATH that pointed to them, in which case you would be altering
the global search order.

> The reason I need it is, the libs in /usr/local have various
> compatibility problems. I've had bash, grep and sed from /usr and
> /usr/bin die because I built a library in /usr/local with a sanitizer,
> and the program or another library had undefined behavior or a memory
> error that surfaced. It basically makes the system unusable.

How did your system binaries end up using libraries from /usr/local/lib64?

Did you use LD_LIBRARY_PATH?

> Additional problems I have managed to side-step so far include
> removing unaligned access in /usr/local programs and libraries.
> Sometimes that involves modifying a struct. Different stuct
> definitions are surely going to cause problems.

It will cause problems.

> How do I tell the system loader to only use distro libraries for
> distro binaries?

Don't run them with options that cause the dynamic lodaer to search
for libraries in alternate locations?

Alternative, and this is a heavy hammer option, you use the LD_AUDIT
mechanism and write an audit module that rewrites the lookup paths
for system binaries to *never* look in /usr/local-anything.

See `man rtld-audit`, and la_objsearch().

-- 
Cheers,
Carlos.

Re: How to set a system-wide library path policy?

[Jeffrey Walton]

On Fri, Jun 7, 2019 at 4:46 PM Carlos O'Donell <carlos@redhat.com> wrote:
>
> On 6/7/19 10:53 AM, Jeffrey Walton wrote:
> > I'm having trouble figuring out how to setup library path policy on Fedora 29.
>
> The system loader is in glibc. Adding libc-help to the CC.
>
> > I need to articulate to the system loader:
> >
> > 1. programs in /bin must only link to libs in /lib64
> > 2. programs in /usr/bin must only link to libs in /usr/lib64
> > 3. programs in /usr/local/bin may use libraries in /usr/local/lib64 or
> > /usr/lib64
>
> OK.
>
> > As far as I know, the distro supplies (1) and (2). They know nothing
> > about my libraries in /usr/local. When the distro is ready, it will
> > push updated programs and libraries as needed. The distro binaries
> > should not use my libraries.
>
> Why would your distro binaries use your libraries? Such libraries are
> not present at static link time when they are built in the Feodra builders.
>
> The only reason your distro libraries would be used is if you used
> LD_LIBRARY_PATH that pointed to them, in which case you would be altering
> the global search order.

Thanks Carlos,

Here is a Solaris example because I'm updating Git for the 2.20.0
release. Linux behaves the same way as Solaris, so there' no material
difference here.

I'm also cutting over to 64-bit default per Sun's recommendation at
https://docs.oracle.com/cd/E37838_01/html/E66175/features-1.html.

$ command -v git
/usr/bin/git

$ git checkout master -f && git pull
ld.so.1: git: fatal: /usr/local/lib/libpcre2-8.so.0: wrong ELF class: ELFCLASS64
Killed

Sun does not follow their own recommendation and provides a 32-bit
Git. Then, the distros binaries are using the libraries I am building.

As for your question about why this happens... who knows. Some genius
thought it was a good idea to link to the wrong library...

Personally, I don't get the use case: compile and link against one
version of a library at compile time, then runtime link against the
wrong library. Apparently someone asserted that was the number one use
case, and represented it as such out of the box (q.v.).

Jeff

Re: How to set a system-wide library path policy?

[Jeffrey Walton]

On Sun, Jun 9, 2019 at 5:34 PM Jeffrey Walton <noloader@gmail.com> wrote:
>
> On Fri, Jun 7, 2019 at 4:46 PM Carlos O'Donell <carlos@redhat.com> wrote:
> >
> > On 6/7/19 10:53 AM, Jeffrey Walton wrote:
> > > I'm having trouble figuring out how to setup library path policy on Fedora 29.
> >
> > The system loader is in glibc. Adding libc-help to the CC.
> >
> > > I need to articulate to the system loader:
> > >
> > > 1. programs in /bin must only link to libs in /lib64
> > > 2. programs in /usr/bin must only link to libs in /usr/lib64
> > > 3. programs in /usr/local/bin may use libraries in /usr/local/lib64 or
> > > /usr/lib64
> >
> > OK.
> >
> > > As far as I know, the distro supplies (1) and (2). They know nothing
> > > about my libraries in /usr/local. When the distro is ready, it will
> > > push updated programs and libraries as needed. The distro binaries
> > > should not use my libraries.
> >
> > Why would your distro binaries use your libraries? Such libraries are
> > not present at static link time when they are built in the Feodra builders.
> >
> > The only reason your distro libraries would be used is if you used
> > LD_LIBRARY_PATH that pointed to them, in which case you would be altering
> > the global search order.
>
> Thanks Carlos,
>
> Here is a Solaris example because I'm updating Git for the 2.20.0
> release. Linux behaves the same way as Solaris, so there' no material
> difference here.
> ...

Here's a Ubuntu Bionic Aarch64 example when running apt-get. The stuff
in /usr/local was built for GnuTLS testing. Guile 2.2 was built from
sources:

Calculating upgrade... Done
The following packages will be upgraded:
  dbus dbus-x11 libdbus-1-3 libglib2.0-0 libglib2.0-data libpython2.7
  libpython2.7-minimal libpython2.7-stdlib libpython3.6 libpython3.6-minimal
  libpython3.6-stdlib libssl-dev libssl1.1 openssl python2.7 python2.7-minimal
  python3.6 python3.6-minimal vim vim-common vim-runtime vim-tiny xxd
...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
/sbin/ldconfig.real: /usr/local/lib/libguile-2.2.so.1.3.1-gdb.scm is
not an ELF file - it has the wrong magic bytes at the start.
...

Jeff

Re: How to set a system-wide library path policy?

[Carlos O'Donell]

On 6/11/19 3:56 PM, Jeffrey Walton wrote:
> On Sun, Jun 9, 2019 at 5:34 PM Jeffrey Walton <noloader@gmail.com> wrote:
>>
>> On Fri, Jun 7, 2019 at 4:46 PM Carlos O'Donell <carlos@redhat.com> wrote:
>>>
>>> On 6/7/19 10:53 AM, Jeffrey Walton wrote:
>>>> I'm having trouble figuring out how to setup library path policy on Fedora 29.
>>>
>>> The system loader is in glibc. Adding libc-help to the CC.
>>>
>>>> I need to articulate to the system loader:
>>>>
>>>> 1. programs in /bin must only link to libs in /lib64
>>>> 2. programs in /usr/bin must only link to libs in /usr/lib64
>>>> 3. programs in /usr/local/bin may use libraries in /usr/local/lib64 or
>>>> /usr/lib64
>>>
>>> OK.
>>>
>>>> As far as I know, the distro supplies (1) and (2). They know nothing
>>>> about my libraries in /usr/local. When the distro is ready, it will
>>>> push updated programs and libraries as needed. The distro binaries
>>>> should not use my libraries.
>>>
>>> Why would your distro binaries use your libraries? Such libraries are
>>> not present at static link time when they are built in the Feodra builders.
>>>
>>> The only reason your distro libraries would be used is if you used
>>> LD_LIBRARY_PATH that pointed to them, in which case you would be altering
>>> the global search order.
>>
>> Thanks Carlos,
>>
>> Here is a Solaris example because I'm updating Git for the 2.20.0
>> release. Linux behaves the same way as Solaris, so there' no material
>> difference here.
>> ...
> 
> Here's a Ubuntu Bionic Aarch64 example when running apt-get. The stuff
> in /usr/local was built for GnuTLS testing. Guile 2.2 was built from
> sources:
> 
> Calculating upgrade... Done
> The following packages will be upgraded:
>   dbus dbus-x11 libdbus-1-3 libglib2.0-0 libglib2.0-data libpython2.7
>   libpython2.7-minimal libpython2.7-stdlib libpython3.6 libpython3.6-minimal
>   libpython3.6-stdlib libssl-dev libssl1.1 openssl python2.7 python2.7-minimal
>   python3.6 python3.6-minimal vim vim-common vim-runtime vim-tiny xxd
> ...
> Processing triggers for libc-bin (2.27-3ubuntu1) ...
> /sbin/ldconfig.real: /usr/local/lib/libguile-2.2.so.1.3.1-gdb.scm is
> not an ELF file - it has the wrong magic bytes at the start.
> ...

The dynamic loader does not normally search /usr/local/lib.

The use of ldconfig does not normally scan non-DSOs.

Why is it searching /usr/local/lib?

Do you have entries in /etc/ld.so.conf to force it to look
for libraries in /usr/local/lib?

Have extra patches been applied to this glibc to change the behaviour?

You should be taking this up with Ubuntu.

-- 
Cheers,
Carlos.

Re: How to set a system-wide library path policy?

[Florian Weimer]

* Carlos O'Donell:

> Why is it searching /usr/local/lib?

Debian adds /usr/local/lib to the search path, to match binutils
behavior.

Thanks,
Florian