Generating Kernel module for Other Computers

Generating Kernel module for Other Computers

[Daniel Doron]

Hi,

I am trying to automate the creation of the driver from one host for
multiple computers.
The purpose is:
1. compile the driver with the latest STAP version
2. automate it for multiple kernels at once
3. distribute the driver to each machine

My approach was to use stap "-r" flag. So for example I would use my
Arch Linux to download and extract all relevant source for Centos 6
and compile my module for it.

Centos 6.9:
---------------
get the following packages:
kernel-2.6.32-696.el6.x86_64.rpm
kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm
kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm
kernel-devel-2.6.32-696.el6.x86_64.rpm

extract them to some location.

then:
---
export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules

soft link vmlinux for STAP
ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux
~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux

sudo stap -vv -r
/home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64
-e 'probe vfs.read {exit()}' -m simple


the error I get:
include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h:
No such file or directory
 #include gcc_header(__GNUC__)

made me realize this is not good because STAP relies on the gcc
version I use to get the header in the kernel. Of course I am using
gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is
4.4.7)

I could:
1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4
stap -r ....
2. other suggestion?


Thanks.
Daniel.

Re: Generating Kernel module for Other Computers

[Arkady]

You shall consider one of the following

*  A dedicated virtual machine with the correct kernel version
(vagrant?) where you install the debug packages and STAP using
yum/apt-get
*  A Docker container - this is what I am using - where you install
debug packages manually. You will have to fix some symbolic links.

On Sun, Jul 2, 2017 at 6:29 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
> Hi,
>
> I am trying to automate the creation of the driver from one host for
> multiple computers.
> The purpose is:
> 1. compile the driver with the latest STAP version
> 2. automate it for multiple kernels at once
> 3. distribute the driver to each machine
>
> My approach was to use stap "-r" flag. So for example I would use my
> Arch Linux to download and extract all relevant source for Centos 6
> and compile my module for it.
>
> Centos 6.9:
> ---------------
> get the following packages:
> kernel-2.6.32-696.el6.x86_64.rpm
> kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm
> kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm
> kernel-devel-2.6.32-696.el6.x86_64.rpm
>
> extract them to some location.
>
> then:
> ---
> export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules
>
> soft link vmlinux for STAP
> ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux
> ~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux
>
> sudo stap -vv -r
> /home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64
> -e 'probe vfs.read {exit()}' -m simple
>
>
> the error I get:
> include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h:
> No such file or directory
>  #include gcc_header(__GNUC__)
>
> made me realize this is not good because STAP relies on the gcc
> version I use to get the header in the kernel. Of course I am using
> gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is
> 4.4.7)
>
> I could:
> 1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4
> stap -r ....
> 2. other suggestion?
>
>
> Thanks.
> Daniel.

Re: Generating Kernel module for Other Computers

[Arkady]

P.S. toolchain is only one problem. There are many others. For
example, where to get the correct package from, how to install the
packages, how to fix the links in the way that STAP will find the
packages.
Every build requires ~50s. It is possible to build in parallel. If you
need more than one kernel module the system shall scale reasonably
both performance wise and storage wise.

On Sun, Jul 2, 2017 at 6:29 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
> Hi,
>
> I am trying to automate the creation of the driver from one host for
> multiple computers.
> The purpose is:
> 1. compile the driver with the latest STAP version
> 2. automate it for multiple kernels at once
> 3. distribute the driver to each machine
>
> My approach was to use stap "-r" flag. So for example I would use my
> Arch Linux to download and extract all relevant source for Centos 6
> and compile my module for it.
>
> Centos 6.9:
> ---------------
> get the following packages:
> kernel-2.6.32-696.el6.x86_64.rpm
> kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm
> kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm
> kernel-devel-2.6.32-696.el6.x86_64.rpm
>
> extract them to some location.
>
> then:
> ---
> export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules
>
> soft link vmlinux for STAP
> ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux
> ~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux
>
> sudo stap -vv -r
> /home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64
> -e 'probe vfs.read {exit()}' -m simple
>
>
> the error I get:
> include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h:
> No such file or directory
>  #include gcc_header(__GNUC__)
>
> made me realize this is not good because STAP relies on the gcc
> version I use to get the header in the kernel. Of course I am using
> gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is
> 4.4.7)
>
> I could:
> 1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4
> stap -r ....
> 2. other suggestion?
>
>
> Thanks.
> Daniel.

Re: Generating Kernel module for Other Computers

[David Smith]

Another approach here would be to use the existing systemtap
client/server mechanism. You'd set up a server for each distro you
wanted to support (note that each compile server system doesn't have
to be a real system, it could be a virtual machine). In the
background, the systemtap servers use avahi to broadcast what kernel
versions they support. When the '--use-sever' option is used with the
systemtap client, the client looks for a match to the system it is
compiling for and ships the script off to the appropriate systemtap
compile server and then the module gets transparently shipped back to
the client. From the client's point of view, nothing changes except
for adding in the '--use-server' option.

(Note that we're doing some work on a more web-based system, but it is
still quite early in development.)

On Sun, Jul 2, 2017 at 12:20 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> P.S. toolchain is only one problem. There are many others. For
> example, where to get the correct package from, how to install the
> packages, how to fix the links in the way that STAP will find the
> packages.
> Every build requires ~50s. It is possible to build in parallel. If you
> need more than one kernel module the system shall scale reasonably
> both performance wise and storage wise.
>
> On Sun, Jul 2, 2017 at 6:29 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
>> Hi,
>>
>> I am trying to automate the creation of the driver from one host for
>> multiple computers.
>> The purpose is:
>> 1. compile the driver with the latest STAP version
>> 2. automate it for multiple kernels at once
>> 3. distribute the driver to each machine
>>
>> My approach was to use stap "-r" flag. So for example I would use my
>> Arch Linux to download and extract all relevant source for Centos 6
>> and compile my module for it.
>>
>> Centos 6.9:
>> ---------------
>> get the following packages:
>> kernel-2.6.32-696.el6.x86_64.rpm
>> kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm
>> kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm
>> kernel-devel-2.6.32-696.el6.x86_64.rpm
>>
>> extract them to some location.
>>
>> then:
>> ---
>> export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules
>>
>> soft link vmlinux for STAP
>> ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux
>> ~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux
>>
>> sudo stap -vv -r
>> /home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64
>> -e 'probe vfs.read {exit()}' -m simple
>>
>>
>> the error I get:
>> include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h:
>> No such file or directory
>>  #include gcc_header(__GNUC__)
>>
>> made me realize this is not good because STAP relies on the gcc
>> version I use to get the header in the kernel. Of course I am using
>> gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is
>> 4.4.7)
>>
>> I could:
>> 1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4
>> stap -r ....
>> 2. other suggestion?
>>
>>
>> Thanks.
>> Daniel.



-- 
David Smith
Principal Software Engineer
Red Hat

Re: Generating Kernel module for Other Computers

[Arkady]

On Mon, Jul 3, 2017 at 7:17 PM, David Smith <dsmith@redhat.com> wrote:
> Another approach here would be to use the existing systemtap
> client/server mechanism. You'd set up a server for each distro you
> wanted to support (note that each compile server system doesn't have
> to be a real system, it could be a virtual machine). In the
> background, the systemtap servers use avahi to broadcast what kernel
> versions they support.
>

This way to compile and deploy the drivers is great for situations
when we can control the environment. For example, we can install avahi
library, stap-run, allow UDP connections, multicast.

There are other scenarios as well. I have to deliver the driver
precompiled for all kernels existing in the network. I can not assume
that the production server can connect to anything besides a
proprietary service running above websocket. I am not allowed, for
example, to compile modules on the fly and deploy w/o testing the
drivers in the lab first. I have to support between 10 to 20 kernels
across 3-5 different distributions in a single deployment. If I choose
VM route - 20GB per VM - I start to feel the storage costs.

For a large number of different kernels I do not see many alternatives
besides container/chroot.

>
> --
> David Smith
> Principal Software Engineer
> Red Hat

Re: Generating Kernel module for Other Computers

[David Smith]

On Mon, Jul 3, 2017 at 12:17 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> On Mon, Jul 3, 2017 at 7:17 PM, David Smith <dsmith@redhat.com> wrote:
>> Another approach here would be to use the existing systemtap
>> client/server mechanism. You'd set up a server for each distro you
>> wanted to support (note that each compile server system doesn't have
>> to be a real system, it could be a virtual machine). In the
>> background, the systemtap servers use avahi to broadcast what kernel
>> versions they support.
>>
>
> This way to compile and deploy the drivers is great for situations
> when we can control the environment. For example, we can install avahi
> library, stap-run, allow UDP connections, multicast.
>
> There are other scenarios as well. I have to deliver the driver
> precompiled for all kernels existing in the network. I can not assume
> that the production server can connect to anything besides a
> proprietary service running above websocket. I am not allowed, for
> example, to compile modules on the fly and deploy w/o testing the
> drivers in the lab first. I have to support between 10 to 20 kernels
> across 3-5 different distributions in a single deployment. If I choose
> VM route - 20GB per VM - I start to feel the storage costs.
>
> For a large number of different kernels I do not see many alternatives
> besides container/chroot.

I think I see where you are coming from. But, your situation might not
be the same as Daniel's situation.

Even in your situation, you say you've got 3-5 different
distributions. With the existing client/server mechanism, you should
only need 1 vm/system per distro - not one per kernel. Each vm/system
can support multiple kernels from that distro. For example, in
Daniel's case he's trying to support Centos 6.9. One Centos 6.9
VM/system server can support multiple Centos 6.9 kernels.

(As I mentioned earlier, we're moving in the direction of using
containers, but we aren't there yet.)

-- 
David Smith
Principal Software Engineer
Red Hat

Re: Generating Kernel module for Other Computers

[Arkady]

>
> I think I see where you are coming from. But, your situation might not
> be the same as Daniel's situation.
>
> Even in your situation, you say you've got 3-5 different
> distributions. With the existing client/server mechanism, you should
> only need 1 vm/system per distro - not one per kernel. Each vm/system
> can support multiple kernels from that distro. For example, in
> Daniel's case he's trying to support Centos 6.9. One Centos 6.9
> VM/system server can support multiple Centos 6.9 kernels.
>

Indeed I do not keep a VM per kernel. I do in the lab where I run
tests, but not on the build server. And even the former can be made to
reuse the VM by replacing the kernel. The argument of 20GB x 10VM
still stands.

> (As I mentioned earlier, we're moving in the direction of using
> containers, but we aren't there yet.)

We talked about this before. I would love to help the project because
meanwhile, as you know and can see, I brought up a system with some,
arguably ugly shortcuts. For example, I hard coded package
dependencies, I install the packages by uncompressing them. I have a
rudimental support for parallel build, but not for testing and so on.


>
> --
> David Smith
> Principal Software Engineer
> Red Hat

Re: Generating Kernel module for Other Computers

[David Smith]

On Mon, Jul 3, 2017 at 9:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> (As I mentioned earlier, we're moving in the direction of using
>> containers, but we aren't there yet.)
>
> We talked about this before. I would love to help the project because
> meanwhile, as you know and can see, I brought up a system with some,
> arguably ugly shortcuts. For example, I hard coded package
> dependencies, I install the packages by uncompressing them. I have a
> rudimental support for parallel build, but not for testing and so on.

Feel free to look at the server side code in the 'httpd' subdirectory.
I'm in the process now of trying to think through some of the
container issues. Your prototype was interesting to look at. If you
have any thoughts, let me know.

-- 
David Smith
Principal Software Engineer
Red Hat