Using non-system glibc

Using non-system glibc

[Mahmood Naderan]

Hi
How can I use a custom glibc instead of the default system version?
I tried nginx with:


    $ GLIBCDIR=/opt/glibc-2.23-install 
    $ ./configure --with-ld-opt="-Wl,--emit-relocs"
    $ make && make install


But the final binary has no sign of my glibc installation


    $ ldd install/sbin/nginx
            linux-vdso.so.1 =>  (0x00007ffc07d43000)
            libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0a8d2f3000)
            libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0a8d0d6000)
            libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f0a8ce9e000)
            libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f0a8cc2e000)
            libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f0a8ca14000)
            libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0a8c64a000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f0a8d4f7000)


Any guess?


Regards,
Mahmood

Re: Using non-system glibc

[Florian Weimer]

* Mahmood Naderan via Libc-help:

> How can I use a custom glibc instead of the default system version?

Please see the parallel thread “how to link with old versions of glibc?”
for some suggestions.

If it's a just a matter of running one application with a different
glibc version, you can use the testrun.sh script in the build tree.

Thanks,
Florian

Re: Using non-system glibc

[Mahmood Naderan]

>Please see the parallel thread “how to link with old versions of glibc?”
>for some suggestions.

Didn't find that via google
 https://www.google.com/search?safe=off&client=firefox-b-1-d&sxsrf=ALeKk00i0RHucEYMaEUPoJ515GN2M8hfaQ%3A1590601354199&ei=iqbOXrfPC8-PlwSQuZP4CA&q=%E2%80%9Chow+to+link+with+old+versions+of+glibc%3F%E2%80%9D&oq=%E2%80%9Chow+to+link+with+old+versions+of+glibc%3F%E2%80%9D&gs_lcp=CgZwc3ktYWIQDDoECAAQRzoHCCMQ6gIQJ1DthgJYqZECYKeaAmgBcAN4AIAB-QGIAfkBkgEDMi0xmAEAoAEBoAECqgEHZ3dzLXdperABCg&sclient=psy-ab&ved=0ahUKEwi3vMjly9TpAhXPx4UKHZDcBI8Q4dUDCAs



>If it's a just a matter of running one application with a different
>glibc version, you can use the testrun.sh script in the build tree.



Excuse me I didn't understand the testrun.sh.Is this a normal output then? How can I use that?
$ ./testrun.sh
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
This program usually lives in the file `/lib/ld.so', and special directives
in executable files using ELF shared libraries tell the system's program
loader to load the helper program from this file.  This helper program loads
the shared libraries needed by the program executable, prepares the program
to run, and runs it.  You may invoke this helper program directly from the
command line to load and run an ELF executable file; this is like executing
that file itself, but always uses this helper program from the file you
specified, instead of the helper program file specified in the executable
file you run.  This is mostly of use for maintainers to test new versions
of this helper program; chances are you did not intend to run this program.

  --list                list all dependencies and how they are resolved
  --verify              verify that given object really is a dynamically linked
                        object we can handle
  --inhibit-cache       Do not use /opt/glibc-2.23-install/etc/ld.so.cache
  --library-path PATH   use given PATH instead of content of the environment
                        variable LD_LIBRARY_PATH
  --inhibit-rpath LIST  ignore RUNPATH and RPATH information in object names
                        in LIST
  --audit LIST          use objects named in LIST as auditors




Regards,
Mahmood 

Re: Using non-system glibc

[Mahmood Naderan]

Excuse me, Should I do something like this:



$ /opt/glibc-2.23-install/testrun.sh --list nginx
nginx: error while loading shared libraries: nginx: cannot open shared object file


but in the same directory as I said, ldd works

$ ldd nginx
        linux-vdso.so.1 =>  (0x00007ffdad776000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb9a3ef4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb9a3cd7000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fb9a3a9f000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fb9a382f000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb9a3615000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb9a324b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb9a40f8000)




Regards,
Mahmood 

    On Wednesday, May 27, 2020, 10:17:26 PM GMT+4:30, Mahmood Naderan via Libc-help <libc-help@sourceware.org> wrote:  
 
 >Please see the parallel thread “how to link with old versions of glibc?”
>for some suggestions.

Didn't find that via google
 https://www.google.com/search?safe=off&client=firefox-b-1-d&sxsrf=ALeKk00i0RHucEYMaEUPoJ515GN2M8hfaQ%3A1590601354199&ei=iqbOXrfPC8-PlwSQuZP4CA&q=%E2%80%9Chow+to+link+with+old+versions+of+glibc%3F%E2%80%9D&oq=%E2%80%9Chow+to+link+with+old+versions+of+glibc%3F%E2%80%9D&gs_lcp=CgZwc3ktYWIQDDoECAAQRzoHCCMQ6gIQJ1DthgJYqZECYKeaAmgBcAN4AIAB-QGIAfkBkgEDMi0xmAEAoAEBoAECqgEHZ3dzLXdperABCg&sclient=psy-ab&ved=0ahUKEwi3vMjly9TpAhXPx4UKHZDcBI8Q4dUDCAs



>If it's a just a matter of running one application with a different
>glibc version, you can use the testrun.sh script in the build tree.



Excuse me I didn't understand the testrun.sh.Is this a normal output then? How can I use that?
$ ./testrun.sh
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
This program usually lives in the file `/lib/ld.so', and special directives
in executable files using ELF shared libraries tell the system's program
loader to load the helper program from this file.  This helper program loads
the shared libraries needed by the program executable, prepares the program
to run, and runs it.  You may invoke this helper program directly from the
command line to load and run an ELF executable file; this is like executing
that file itself, but always uses this helper program from the file you
specified, instead of the helper program file specified in the executable
file you run.  This is mostly of use for maintainers to test new versions
of this helper program; chances are you did not intend to run this program.

  --list                list all dependencies and how they are resolved
  --verify              verify that given object really is a dynamically linked
                        object we can handle
  --inhibit-cache       Do not use /opt/glibc-2.23-install/etc/ld.so.cache
  --library-path PATH   use given PATH instead of content of the environment
                        variable LD_LIBRARY_PATH
  --inhibit-rpath LIST  ignore RUNPATH and RPATH information in object names
                        in LIST
  --audit LIST          use objects named in LIST as auditors




Regards,
Mahmood 
  

Re: Using non-system glibc

[Mahmood Naderan]

On the way to use a custom glibc build, I tried to modify the Makefile of the program (nginx) with 


-Wl,--rpath=/opt/glibc-2.23-install -Wl,--dynamic-linker=/opt/glibc-2.23-install/lib/ld-2.23.so

After build and install, I see the following ldd output

$ ldd install/sbin/nginx
        linux-vdso.so.1 =>  (0x00007ffd6c335000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fad02350000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fad02133000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fad01efb000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fad01c8b000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fad01a71000)
        libc.so.6 => /opt/glibc-2.23-install/libc.so.6 (0x00007fad016d0000)
        /opt/glibc-2.23-install/lib/ld-2.23.so => /lib64/ld-linux-x86-64.so.2 (0x00007fad02554000)



The last line shows that my own glibc build is using system libc. But the lic.so.6 file my program is using, comes from my own glibc build.

So, is that all? Or something is missing? Any advice?


Regards,
Mahmood 

Re: Using non-system glibc

[Florian Weimer]

* Mahmood Naderan:

> On the way to use a custom glibc build, I tried to modify the Makefile of the program (nginx)
> with 
>
> -Wl,--rpath=/opt/glibc-2.23-install -Wl,--dynamic-linker=/opt/glibc-2.23-install/lib/ld-2.23.so
>
> After build and install, I see the following ldd output
>
> $ ldd install/sbin/nginx
>         linux-vdso.so.1 =>  (0x00007ffd6c335000)
>         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fad02350000)
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fad02133000)
>         libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fad01efb000)
>         libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fad01c8b000)
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fad01a71000)
>         libc.so.6 => /opt/glibc-2.23-install/libc.so.6 (0x00007fad016d0000)
>         /opt/glibc-2.23-install/lib/ld-2.23.so => /lib64/ld-linux-x86-64.so.2
> (0x00007fad02554000)
>
> The last line shows that my own glibc build is using system libc. But the lic.so.6 file my
> program is using, comes from my own glibc build.
>
> So, is that all? Or something is missing? Any advice?

I would try to reference the dynamic linker under its soname, like this:

  -Wl,--dynamic-linker=/opt/glibc-2.23-install/ld-linux-x86-64.so.2

Not sure if it makes a difference.

It's also concerning that libdl.so.2 and libcrypt.so.1 are loaded from
system paths.  libcrypt.so.1 could be fine, depending on whether it uses
internal glibc symbols or not.

Thanks,
Florian

Re: Using non-system glibc

[Mahmood Naderan]

>I would try to reference the dynamic linker under its soname, like this:
>
>  -Wl,--dynamic-linker=/opt/glibc-2.23-install/ld-linux-x86-64.so.2

I don't see such file. Is that normal?

mahmood@hp2:/opt/glibc-2.23-install$ find . -name ld*
./ld.map
mahmood@hp2:/opt/glibc-2.23-install$


>It's also concerning that libdl.so.2 and libcrypt.so.1 are loaded from
>system paths.  libcrypt.so.1 could be fine, depending on whether it uses
>internal glibc symbols or not.

That is also my concern. For my work I need that --emit-reloc be available. Does that mean I have to manually recompile all staffs? from kernel to libraries and applications?




Regards,
Mahmood 

Re: Using non-system glibc

[Florian Weimer]

* Mahmood Naderan:

>>I would try to reference the dynamic linker under its soname, like this:
>>
>>  -Wl,--dynamic-linker=/opt/glibc-2.23-install/ld-linux-x86-64.so.2
>
> I don't see such file. Is that normal?
>
> mahmood@hp2:/opt/glibc-2.23-install$ find . -name ld*
> ./ld.map

Please use:

$ find . -name 'ld*'

> That is also my concern. For my work I need that --emit-reloc be
> available. Does that mean I have to manually recompile all staffs?
> from kernel to libraries and applications?

If your new glibc is only compatible with --emit-relocs libraries, and
your existing libraries have not been built this way, then yes, you have
to rebuild them.  I don't know what you are trying to accomplish, so
there might be another way around that.

Thanks,
Florian

Re: Using non-system glibc

[Mahmood Naderan]

>Please use:
>
>$ find . -name 'ld*'

Right. I now see

mahmood@hp2:/opt/glibc-2.23-install$ ls -l lib/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 10 May   27 15:01 lib/ld-linux-x86-64.so.2 -> ld-2.23.so
mahmood@hp2:/opt/glibc-2.23-install$mahmood@hp2:/opt/glibc-2.23-install$ ls -l lib/ld-2.23.so
-rwxr-xr-x 1 root root 1680552 May   27 15:01 lib/ld-2.23.so



I just used 

-Wl,--dynamic-linker=/opt/glibc-2.23-install/lib/ld-2.23.so
Don't know if that symlink and actual file name can cause this problem. I will try that.

>I don't know what you are trying to accomplish, so
>there might be another way around that.
Thanks for the feedback. I am trying to analyze my program with BOLT [1] which is a compiler optimization tool and it needs --emit-relocs in the binary file. 

With all default system files, I have built nginx with --emit-relocs. Then I create a perf data file and give it to BOLT. Profiling with :u (user instructions), there is no problem. However, profiling with :k I get some errors about stale profiles. At first I thought the stale profiles are related to system libraries that are not compiled with --emit-relocs.
When I check the name of the stale profiles, these are actually nginx functions, e.g. ngx_sprintf, and I guess since it is profiling kernel instructions, system calls from ngx_sprintf hit to the files that are not using --emit-relocs and BOLT gives me the root function name (nginx).
I know my problem is off topic, but I am trying to convince myself that if I can build system files with --emit-relocs, my problems will be solved... Sorry for talking too much.



Regards,
Mahmood