How to modify LD_LIBRARY_PATH

How to modify LD_LIBRARY_PATH

[Arkady]

Hi,

My goal is to create a build environment for the kernel modules which
target different kernels in the different distros - Ubuntu, CentOS,
etc.

I am trying to build the kernel modules in a chroot environment based
on CentOS6. The approach works fine for CentOS kernels.
When I am trying to compile a module for the Ubuntu kernel
3.8.0-31-generic all build commands similar to this one fail:

env -uARCH -uKBUILD_EXTMOD -uCROSS_COMPILE -uKBUILD_IMAGE
-uKCONFIG_CONFIG -uINSTALL_PATH -uLD_LIBRARY_PATH
PATH=/usr/bin:/bin:/opt/rh/python27/root/usr/bin:/opt/rh/devtoolset-4/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
make -C //lib/modules/3.8.0-31-generic/build
M=/tmp/stap5IyPLq/typequery_kmod_23 modules CONFIG_DE>

/usr/src/linux-headers-3.8.0-31-generic/arch/x86/Makefile:103:
CONFIG_X86_X32 enabled but no binutils support
  Building modules, stage 2.
  MODPOST 1 modules
scripts/mod/modpost: /lib64/libc.so.6: version `GLIBC_2.14' not found
(required by scripts/mod/modpost)
make[1]: *** [__modpost] Error 1
make: *** [modules] Error 2

I have GLIBC=2.14 installed in /opt/glibc-2.14/lib

In the function
make_any_make_cmd() I see "Sanitize environment variables for kbuild
invocation" and explicit undef for LD_LIBRARY_PATH
How can I modify LD_LIBRARY_PATH anyway?

Thank you, Arkady.

Re: How to modify LD_LIBRARY_PATH

[Arkady]

I have patched make_any_make_cmd() and moved one small step forward.
The build of the script still fails for Ubuntu kernels with the error
below.
Is the idea is feasible? Does it make sense to use the same and rather
old CentOS6 chroot for building SystemTap modules?

Thank you, Arkady.

semantic error: while processing probe
kernel.function("__sock_create@/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1259").return
from: socket.create.return from: socket.create.return
   thrown from: elaborate.cxx:5649
semantic error: not accessible at this address (pc:
0xffffffff815caf20) [man error::dwarf]: identifier '$protocol' at
/usr/local/share/systemtap/tapset/linux/socket.stp:688:20
        dieoffset: 0x5324703 from
/root/rpms/usr/lib/debug/boot/vmlinux-3.8.0-31-generic
        function: __sock_create at
/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1261
        alternative locations:
[0xffffffff815caf25,0xffffffff815caf81],
[0xffffffff815caf81,0xffffffff815cafa1],
[0xffffffff815cafab,0xffffffff815cb11f]
   thrown from: dwflpp.cxx:3246
        source: protocol = @entry($protocol)

Re: How to modify LD_LIBRARY_PATH

[Arkady]

And another build problem


semantic error: while processing probe
kernel.function("__sock_create@/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1259").return
from: socket.create.return from: socket.create.return
   thrown from: elaborate.cxx:5649
semantic error: not accessible at this address (pc:
0xffffffff815caf20) [man error::dwarf]: identifier '$protocol' at
/usr/local/share/systemtap/tapset/linux/socket.stp:688:20
        dieoffset: 0x5324703 from
/root/rpms/usr/lib/debug/boot/vmlinux-3.8.0-31-generic
        function: __sock_create at
/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1261
        alternative locations:
[0xffffffff815caf25,0xffffffff815caf81],
[0xffffffff815caf81,0xffffffff815cafa1],
[0xffffffff815cafab,0xffffffff815cb11f]
   thrown from: dwflpp.cxx:3246
        source: protocol = @entry($protocol)

                                  ^

On Wed, Mar 15, 2017 at 9:35 AM, Arkady <arkady.miasnikov.ca@gmail.com> wrote:
> I have patched make_any_make_cmd() and moved one small step forward.
> The build of the script still fails for Ubuntu kernels with the error
> below.
> Is the idea is feasible? Does it make sense to use the same and rather
> old CentOS6 chroot for building SystemTap modules?
>
> Thank you, Arkady.
>
> semantic error: while processing probe
> kernel.function("__sock_create@/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1259").return
> from: socket.create.return from: socket.create.return
>    thrown from: elaborate.cxx:5649
> semantic error: not accessible at this address (pc:
> 0xffffffff815caf20) [man error::dwarf]: identifier '$protocol' at
> /usr/local/share/systemtap/tapset/linux/socket.stp:688:20
>         dieoffset: 0x5324703 from
> /root/rpms/usr/lib/debug/boot/vmlinux-3.8.0-31-generic
>         function: __sock_create at
> /build/buildd/linux-lts-raring-3.8.0/net/socket.c:1261
>         alternative locations:
> [0xffffffff815caf25,0xffffffff815caf81],
> [0xffffffff815caf81,0xffffffff815cafa1],
> [0xffffffff815cafab,0xffffffff815cb11f]
>    thrown from: dwflpp.cxx:3246
>         source: protocol = @entry($protocol)

Re: How to modify LD_LIBRARY_PATH

[Arkady]

Update. I succeeded to use CentOS6 chroot for building the kernel
module for 4.2.0-38-generic (Ubuntu) and 2.6.32-642.el6.x86_64
(CentOS6)

but it fails for 4.4.0-66-generic (Ubuntu)

Pass 2: analyzed script: 171 probes, 89 functions, 117 embeds, 107
globals using 253976virt/229432res/5620shr/224636data kb, in
5220usr/270sys/5501real ms.
Pass 3: translated to C into "/tmp/stap2N1zpy/secdo_src.c" using
253976virt/229624res/5812shr/224636data kb, in 370usr/10sys/382real
ms.
arch/x86/Makefile:148: CONFIG_X86_X32 enabled but no binutils support
gcc: gcc: unrecognized option '-no-pie'
unrecognized option '-no-pie'
gcc: unrecognized option '-no-pie'
cc1: warnings being treated as errors
/tmp/stap2N1zpy/secdo_src.c: In function 'systemtap_module_exit':
/tmp/stap2N1zpy/secdo_src.c:50609: error: the frame size of 3712 bytes
is larger than 512 bytes
make[1]: *** [/tmp/stap2N1zpy/secdo_src.o] Error 1
make: *** [_module_/tmp/stap2N1zpy] Error 2
WARNING: kbuild exited with status: 2

and 3.8.0-31-generic (Ubuntu)

semantic error: while processing probe
kernel.function("__sock_create@/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1259").return
from: socket.create.return from: socket.create.return
semantic error: not accessible at this address (pc:
0xffffffff815caf20) [man error::dwarf]: identifier '$protocol' at
/usr/local/share/systemtap/tapset/linux/socket.stp:688:20
        dieoffset: 0x5324703 from
/root/rpms/usr/lib/debug/boot/vmlinux-3.8.0-31-generic
        function: __sock_create at
/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1261
        alternative locations:
[0xffffffff815caf25,0xffffffff815caf81],
[0xffffffff815caf81,0xffffffff815cafa1],
[0xffffffff815cafab,0xffffffff815cb11f]
        source: protocol = @entry($protocol)


I suspect that I am trying to solve the problem in a wrong way. If my
goal is to generate kernel modules for a large number of different
kernels and distributions what is a reasonable approach?


Thank you, Arkady,

On Wed, Mar 15, 2017 at 10:19 AM, Arkady <arkady.miasnikov.ca@gmail.com> wrote:
> And another build problem
>
>
> semantic error: while processing probe
> kernel.function("__sock_create@/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1259").return
> from: socket.create.return from: socket.create.return
>    thrown from: elaborate.cxx:5649
> semantic error: not accessible at this address (pc:
> 0xffffffff815caf20) [man error::dwarf]: identifier '$protocol' at
> /usr/local/share/systemtap/tapset/linux/socket.stp:688:20
>         dieoffset: 0x5324703 from
> /root/rpms/usr/lib/debug/boot/vmlinux-3.8.0-31-generic
>         function: __sock_create at
> /build/buildd/linux-lts-raring-3.8.0/net/socket.c:1261
>         alternative locations:
> [0xffffffff815caf25,0xffffffff815caf81],
> [0xffffffff815caf81,0xffffffff815cafa1],
> [0xffffffff815cafab,0xffffffff815cb11f]
>    thrown from: dwflpp.cxx:3246
>         source: protocol = @entry($protocol)
>
>                                   ^
>
> On Wed, Mar 15, 2017 at 9:35 AM, Arkady <arkady.miasnikov.ca@gmail.com> wrote:
>> I have patched make_any_make_cmd() and moved one small step forward.
>> The build of the script still fails for Ubuntu kernels with the error
>> below.
>> Is the idea is feasible? Does it make sense to use the same and rather
>> old CentOS6 chroot for building SystemTap modules?
>>
>> Thank you, Arkady.
>>
>> semantic error: while processing probe
>> kernel.function("__sock_create@/build/buildd/linux-lts-raring-3.8.0/net/socket.c:1259").return
>> from: socket.create.return from: socket.create.return
>>    thrown from: elaborate.cxx:5649
>> semantic error: not accessible at this address (pc:
>> 0xffffffff815caf20) [man error::dwarf]: identifier '$protocol' at
>> /usr/local/share/systemtap/tapset/linux/socket.stp:688:20
>>         dieoffset: 0x5324703 from
>> /root/rpms/usr/lib/debug/boot/vmlinux-3.8.0-31-generic
>>         function: __sock_create at
>> /build/buildd/linux-lts-raring-3.8.0/net/socket.c:1261
>>         alternative locations:
>> [0xffffffff815caf25,0xffffffff815caf81],
>> [0xffffffff815caf81,0xffffffff815cafa1],
>> [0xffffffff815cafab,0xffffffff815cb11f]
>>    thrown from: dwflpp.cxx:3246
>>         source: protocol = @entry($protocol)

Re: How to modify LD_LIBRARY_PATH

[David Smith]

(2nd try here)

Arkady,

OK, I think I see what you are trying to do here. I think systemtap
does do what you want, just not the way you are trying to do it.

systemtap's notion of cross-compiling isn't true cross-compiling - we
don't do a chroot into some new environment and run with files and
executables in the chroot. systemtap's version of cross-compiling is
more like "hey, I want to be able to compile kernel modules for
kernels released with RHEL 7.2, 7.3, and 7.4". We're still going to
use the system's compiler for all the kernel-development environments
installed. That works great when you are staying within the same
distro. That doesn't work when you are trying to run across distros
(who have different compilers which support different options as
you've discovered). You definitely want to compile a kernel module
with the same compiler that you compiled the kernel with, since you
want structures, etc. to be laid out the same.

(Note that I've never actually worked on this systemtap feature, so
all the above is from memory and recollections and could be completely
wrong. If so, someone will come along and correct me.)

So, how should you be doing this? You'll want to set up a systemtap
compile server - one for each distro. Note that each compile server
system doesn't have to be a real system, it could be a virtual
machine. In the background, the systemtap servers use avahi to
broadcast what kernel versions they support. When the '--use-sever'
option is used with the systemtap client, the client looks for a match
to the system it is compiling for and ships the script off to the
appropriate systemtap compile server and then the module gets
transparently shipped back to the client. From the client's point of
view, nothing changes except for adding in the '--use-server' option.

I believe that using the compile server will solve your problem.

On Tue, Mar 14, 2017 at 7:12 AM, Arkady <arkady.miasnikov.ca@gmail.com> wrote:
> Hi,
>
> My goal is to create a build environment for the kernel modules which
> target different kernels in the different distros - Ubuntu, CentOS,
> etc.
>
> I am trying to build the kernel modules in a chroot environment based
> on CentOS6. The approach works fine for CentOS kernels.
> When I am trying to compile a module for the Ubuntu kernel
> 3.8.0-31-generic all build commands similar to this one fail:
>
> env -uARCH -uKBUILD_EXTMOD -uCROSS_COMPILE -uKBUILD_IMAGE
> -uKCONFIG_CONFIG -uINSTALL_PATH -uLD_LIBRARY_PATH
> PATH=/usr/bin:/bin:/opt/rh/python27/root/usr/bin:/opt/rh/devtoolset-4/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> make -C //lib/modules/3.8.0-31-generic/build
> M=/tmp/stap5IyPLq/typequery_kmod_23 modules CONFIG_DE>
>
> /usr/src/linux-headers-3.8.0-31-generic/arch/x86/Makefile:103:
> CONFIG_X86_X32 enabled but no binutils support
>   Building modules, stage 2.
>   MODPOST 1 modules
> scripts/mod/modpost: /lib64/libc.so.6: version `GLIBC_2.14' not found
> (required by scripts/mod/modpost)
> make[1]: *** [__modpost] Error 1
> make: *** [modules] Error 2
>
> I have GLIBC=2.14 installed in /opt/glibc-2.14/lib
>
> In the function
> make_any_make_cmd() I see "Sanitize environment variables for kbuild
> invocation" and explicit undef for LD_LIBRARY_PATH
> How can I modify LD_LIBRARY_PATH anyway?
>
> Thank you, Arkady.



-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to modify LD_LIBRARY_PATH

[Arkady]

Just to help Google search.

I have solved the problem of cross compilation of the kernel modules
by compiling (almost) natively in Docker containers.

See for details https://github.com/larytet/YALAS/tree/master/docker

Thanks.

On Thu, Mar 16, 2017 at 11:51 PM, David Smith <dsmith@redhat.com> wrote:
> (2nd try here)
>
> Arkady,
>
> OK, I think I see what you are trying to do here. I think systemtap
> does do what you want, just not the way you are trying to do it.
>
> systemtap's notion of cross-compiling isn't true cross-compiling - we
> don't do a chroot into some new environment and run with files and
> executables in the chroot. systemtap's version of cross-compiling is
> more like "hey, I want to be able to compile kernel modules for
> kernels released with RHEL 7.2, 7.3, and 7.4". We're still going to
> use the system's compiler for all the kernel-development environments
> installed. That works great when you are staying within the same
> distro. That doesn't work when you are trying to run across distros
> (who have different compilers which support different options as
> you've discovered). You definitely want to compile a kernel module
> with the same compiler that you compiled the kernel with, since you
> want structures, etc. to be laid out the same.
>
> (Note that I've never actually worked on this systemtap feature, so
> all the above is from memory and recollections and could be completely
> wrong. If so, someone will come along and correct me.)
>
> So, how should you be doing this? You'll want to set up a systemtap
> compile server - one for each distro. Note that each compile server
> system doesn't have to be a real system, it could be a virtual
> machine. In the background, the systemtap servers use avahi to
> broadcast what kernel versions they support. When the '--use-sever'
> option is used with the systemtap client, the client looks for a match
> to the system it is compiling for and ships the script off to the
> appropriate systemtap compile server and then the module gets
> transparently shipped back to the client. From the client's point of
> view, nothing changes except for adding in the '--use-server' option.
>
> I believe that using the compile server will solve your problem.
>
> On Tue, Mar 14, 2017 at 7:12 AM, Arkady <arkady.miasnikov.ca@gmail.com> wrote:
>> Hi,
>>
>> My goal is to create a build environment for the kernel modules which
>> target different kernels in the different distros - Ubuntu, CentOS,
>> etc.
>>
>> I am trying to build the kernel modules in a chroot environment based
>> on CentOS6. The approach works fine for CentOS kernels.
>> When I am trying to compile a module for the Ubuntu kernel
>> 3.8.0-31-generic all build commands similar to this one fail:
>>
>> env -uARCH -uKBUILD_EXTMOD -uCROSS_COMPILE -uKBUILD_IMAGE
>> -uKCONFIG_CONFIG -uINSTALL_PATH -uLD_LIBRARY_PATH
>> PATH=/usr/bin:/bin:/opt/rh/python27/root/usr/bin:/opt/rh/devtoolset-4/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>> make -C //lib/modules/3.8.0-31-generic/build
>> M=/tmp/stap5IyPLq/typequery_kmod_23 modules CONFIG_DE>
>>
>> /usr/src/linux-headers-3.8.0-31-generic/arch/x86/Makefile:103:
>> CONFIG_X86_X32 enabled but no binutils support
>>   Building modules, stage 2.
>>   MODPOST 1 modules
>> scripts/mod/modpost: /lib64/libc.so.6: version `GLIBC_2.14' not found
>> (required by scripts/mod/modpost)
>> make[1]: *** [__modpost] Error 1
>> make: *** [modules] Error 2
>>
>> I have GLIBC=2.14 installed in /opt/glibc-2.14/lib
>>
>> In the function
>> make_any_make_cmd() I see "Sanitize environment variables for kbuild
>> invocation" and explicit undef for LD_LIBRARY_PATH
>> How can I modify LD_LIBRARY_PATH anyway?
>>
>> Thank you, Arkady.
>
>
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)