* Generating Kernel module for Other Computers @ 2017-07-02 15:29 Daniel Doron 2017-07-02 17:10 ` Arkady 2017-07-02 17:21 ` Arkady 0 siblings, 2 replies; 8+ messages in thread From: Daniel Doron @ 2017-07-02 15:29 UTC (permalink / raw) To: systemtap Hi, I am trying to automate the creation of the driver from one host for multiple computers. The purpose is: 1. compile the driver with the latest STAP version 2. automate it for multiple kernels at once 3. distribute the driver to each machine My approach was to use stap "-r" flag. So for example I would use my Arch Linux to download and extract all relevant source for Centos 6 and compile my module for it. Centos 6.9: --------------- get the following packages: kernel-2.6.32-696.el6.x86_64.rpm kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm kernel-devel-2.6.32-696.el6.x86_64.rpm extract them to some location. then: --- export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules soft link vmlinux for STAP ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux ~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux sudo stap -vv -r /home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64 -e 'probe vfs.read {exit()}' -m simple the error I get: include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h: No such file or directory #include gcc_header(__GNUC__) made me realize this is not good because STAP relies on the gcc version I use to get the header in the kernel. Of course I am using gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is 4.4.7) I could: 1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4 stap -r .... 2. other suggestion? Thanks. Daniel. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Generating Kernel module for Other Computers 2017-07-02 15:29 Generating Kernel module for Other Computers Daniel Doron @ 2017-07-02 17:10 ` Arkady 2017-07-02 17:21 ` Arkady 1 sibling, 0 replies; 8+ messages in thread From: Arkady @ 2017-07-02 17:10 UTC (permalink / raw) To: Daniel Doron; +Cc: systemtap You shall consider one of the following * A dedicated virtual machine with the correct kernel version (vagrant?) where you install the debug packages and STAP using yum/apt-get * A Docker container - this is what I am using - where you install debug packages manually. You will have to fix some symbolic links. On Sun, Jul 2, 2017 at 6:29 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote: > Hi, > > I am trying to automate the creation of the driver from one host for > multiple computers. > The purpose is: > 1. compile the driver with the latest STAP version > 2. automate it for multiple kernels at once > 3. distribute the driver to each machine > > My approach was to use stap "-r" flag. So for example I would use my > Arch Linux to download and extract all relevant source for Centos 6 > and compile my module for it. > > Centos 6.9: > --------------- > get the following packages: > kernel-2.6.32-696.el6.x86_64.rpm > kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm > kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm > kernel-devel-2.6.32-696.el6.x86_64.rpm > > extract them to some location. > > then: > --- > export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules > > soft link vmlinux for STAP > ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux > ~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux > > sudo stap -vv -r > /home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64 > -e 'probe vfs.read {exit()}' -m simple > > > the error I get: > include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h: > No such file or directory > #include gcc_header(__GNUC__) > > made me realize this is not good because STAP relies on the gcc > version I use to get the header in the kernel. Of course I am using > gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is > 4.4.7) > > I could: > 1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4 > stap -r .... > 2. other suggestion? > > > Thanks. > Daniel. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Generating Kernel module for Other Computers 2017-07-02 15:29 Generating Kernel module for Other Computers Daniel Doron 2017-07-02 17:10 ` Arkady @ 2017-07-02 17:21 ` Arkady 2017-07-03 16:17 ` David Smith 1 sibling, 1 reply; 8+ messages in thread From: Arkady @ 2017-07-02 17:21 UTC (permalink / raw) To: Daniel Doron; +Cc: systemtap P.S. toolchain is only one problem. There are many others. For example, where to get the correct package from, how to install the packages, how to fix the links in the way that STAP will find the packages. Every build requires ~50s. It is possible to build in parallel. If you need more than one kernel module the system shall scale reasonably both performance wise and storage wise. On Sun, Jul 2, 2017 at 6:29 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote: > Hi, > > I am trying to automate the creation of the driver from one host for > multiple computers. > The purpose is: > 1. compile the driver with the latest STAP version > 2. automate it for multiple kernels at once > 3. distribute the driver to each machine > > My approach was to use stap "-r" flag. So for example I would use my > Arch Linux to download and extract all relevant source for Centos 6 > and compile my module for it. > > Centos 6.9: > --------------- > get the following packages: > kernel-2.6.32-696.el6.x86_64.rpm > kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm > kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm > kernel-devel-2.6.32-696.el6.x86_64.rpm > > extract them to some location. > > then: > --- > export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules > > soft link vmlinux for STAP > ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux > ~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux > > sudo stap -vv -r > /home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64 > -e 'probe vfs.read {exit()}' -m simple > > > the error I get: > include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h: > No such file or directory > #include gcc_header(__GNUC__) > > made me realize this is not good because STAP relies on the gcc > version I use to get the header in the kernel. Of course I am using > gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is > 4.4.7) > > I could: > 1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4 > stap -r .... > 2. other suggestion? > > > Thanks. > Daniel. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Generating Kernel module for Other Computers 2017-07-02 17:21 ` Arkady @ 2017-07-03 16:17 ` David Smith 2017-07-03 17:18 ` Arkady 0 siblings, 1 reply; 8+ messages in thread From: David Smith @ 2017-07-03 16:17 UTC (permalink / raw) To: Arkady; +Cc: Daniel Doron, systemtap Another approach here would be to use the existing systemtap client/server mechanism. You'd set up a server for each distro you wanted to support (note that each compile server system doesn't have to be a real system, it could be a virtual machine). In the background, the systemtap servers use avahi to broadcast what kernel versions they support. When the '--use-sever' option is used with the systemtap client, the client looks for a match to the system it is compiling for and ships the script off to the appropriate systemtap compile server and then the module gets transparently shipped back to the client. From the client's point of view, nothing changes except for adding in the '--use-server' option. (Note that we're doing some work on a more web-based system, but it is still quite early in development.) On Sun, Jul 2, 2017 at 12:20 PM, Arkady <arkady.miasnikov@gmail.com> wrote: > P.S. toolchain is only one problem. There are many others. For > example, where to get the correct package from, how to install the > packages, how to fix the links in the way that STAP will find the > packages. > Every build requires ~50s. It is possible to build in parallel. If you > need more than one kernel module the system shall scale reasonably > both performance wise and storage wise. > > On Sun, Jul 2, 2017 at 6:29 PM, Daniel Doron <danielmeirdoron@gmail.com> wrote: >> Hi, >> >> I am trying to automate the creation of the driver from one host for >> multiple computers. >> The purpose is: >> 1. compile the driver with the latest STAP version >> 2. automate it for multiple kernels at once >> 3. distribute the driver to each machine >> >> My approach was to use stap "-r" flag. So for example I would use my >> Arch Linux to download and extract all relevant source for Centos 6 >> and compile my module for it. >> >> Centos 6.9: >> --------------- >> get the following packages: >> kernel-2.6.32-696.el6.x86_64.rpm >> kernel-debuginfo-2.6.32-696.3.2.el6.x86_64.rpm >> kernel-debuginfo-common-x86_64-2.6.32-696.3.2.el6.x86_64.rpm >> kernel-devel-2.6.32-696.el6.x86_64.rpm >> >> extract them to some location. >> >> then: >> --- >> export SYSTEMTAP_DEBUGINFO_PATH=/home/danield/Downloads/centos-2.6.32/usr/lib/debug/lib/modules >> >> soft link vmlinux for STAP >> ln -s ~/Downloads/centos-2.6.32/usr/lib/debug/lib/modules/2.6.32-696.3.2.el6.x86_64/vmlinux >> ~/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64/vmlinux >> >> sudo stap -vv -r >> /home/danield/Downloads/centos-2.6.32/usr/src/kernels/2.6.32-696.el6.x86_64 >> -e 'probe vfs.read {exit()}' -m simple >> >> >> the error I get: >> include/linux/compiler-gcc.h:94:1: fatal error: linux/compiler-gcc7.h: >> No such file or directory >> #include gcc_header(__GNUC__) >> >> made me realize this is not good because STAP relies on the gcc >> version I use to get the header in the kernel. Of course I am using >> gcc 7 which doesn't exist for Centos 6.9 kernel' (latest version is >> 4.4.7) >> >> I could: >> 1. download the relevant gcc version and maybe pass CC=/usr/bin/gcc4 >> stap -r .... >> 2. other suggestion? >> >> >> Thanks. >> Daniel. -- David Smith Principal Software Engineer Red Hat ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Generating Kernel module for Other Computers 2017-07-03 16:17 ` David Smith @ 2017-07-03 17:18 ` Arkady 2017-07-03 18:49 ` David Smith 0 siblings, 1 reply; 8+ messages in thread From: Arkady @ 2017-07-03 17:18 UTC (permalink / raw) To: David Smith; +Cc: Daniel Doron, systemtap On Mon, Jul 3, 2017 at 7:17 PM, David Smith <dsmith@redhat.com> wrote: > Another approach here would be to use the existing systemtap > client/server mechanism. You'd set up a server for each distro you > wanted to support (note that each compile server system doesn't have > to be a real system, it could be a virtual machine). In the > background, the systemtap servers use avahi to broadcast what kernel > versions they support. > This way to compile and deploy the drivers is great for situations when we can control the environment. For example, we can install avahi library, stap-run, allow UDP connections, multicast. There are other scenarios as well. I have to deliver the driver precompiled for all kernels existing in the network. I can not assume that the production server can connect to anything besides a proprietary service running above websocket. I am not allowed, for example, to compile modules on the fly and deploy w/o testing the drivers in the lab first. I have to support between 10 to 20 kernels across 3-5 different distributions in a single deployment. If I choose VM route - 20GB per VM - I start to feel the storage costs. For a large number of different kernels I do not see many alternatives besides container/chroot. > > -- > David Smith > Principal Software Engineer > Red Hat ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Generating Kernel module for Other Computers 2017-07-03 17:18 ` Arkady @ 2017-07-03 18:49 ` David Smith 2017-07-04 2:53 ` Arkady 0 siblings, 1 reply; 8+ messages in thread From: David Smith @ 2017-07-03 18:49 UTC (permalink / raw) To: Arkady; +Cc: Daniel Doron, systemtap On Mon, Jul 3, 2017 at 12:17 PM, Arkady <arkady.miasnikov@gmail.com> wrote: > On Mon, Jul 3, 2017 at 7:17 PM, David Smith <dsmith@redhat.com> wrote: >> Another approach here would be to use the existing systemtap >> client/server mechanism. You'd set up a server for each distro you >> wanted to support (note that each compile server system doesn't have >> to be a real system, it could be a virtual machine). In the >> background, the systemtap servers use avahi to broadcast what kernel >> versions they support. >> > > This way to compile and deploy the drivers is great for situations > when we can control the environment. For example, we can install avahi > library, stap-run, allow UDP connections, multicast. > > There are other scenarios as well. I have to deliver the driver > precompiled for all kernels existing in the network. I can not assume > that the production server can connect to anything besides a > proprietary service running above websocket. I am not allowed, for > example, to compile modules on the fly and deploy w/o testing the > drivers in the lab first. I have to support between 10 to 20 kernels > across 3-5 different distributions in a single deployment. If I choose > VM route - 20GB per VM - I start to feel the storage costs. > > For a large number of different kernels I do not see many alternatives > besides container/chroot. I think I see where you are coming from. But, your situation might not be the same as Daniel's situation. Even in your situation, you say you've got 3-5 different distributions. With the existing client/server mechanism, you should only need 1 vm/system per distro - not one per kernel. Each vm/system can support multiple kernels from that distro. For example, in Daniel's case he's trying to support Centos 6.9. One Centos 6.9 VM/system server can support multiple Centos 6.9 kernels. (As I mentioned earlier, we're moving in the direction of using containers, but we aren't there yet.) -- David Smith Principal Software Engineer Red Hat ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Generating Kernel module for Other Computers 2017-07-03 18:49 ` David Smith @ 2017-07-04 2:53 ` Arkady 2017-07-05 13:20 ` David Smith 0 siblings, 1 reply; 8+ messages in thread From: Arkady @ 2017-07-04 2:53 UTC (permalink / raw) To: David Smith; +Cc: Daniel Doron, systemtap > > I think I see where you are coming from. But, your situation might not > be the same as Daniel's situation. > > Even in your situation, you say you've got 3-5 different > distributions. With the existing client/server mechanism, you should > only need 1 vm/system per distro - not one per kernel. Each vm/system > can support multiple kernels from that distro. For example, in > Daniel's case he's trying to support Centos 6.9. One Centos 6.9 > VM/system server can support multiple Centos 6.9 kernels. > Indeed I do not keep a VM per kernel. I do in the lab where I run tests, but not on the build server. And even the former can be made to reuse the VM by replacing the kernel. The argument of 20GB x 10VM still stands. > (As I mentioned earlier, we're moving in the direction of using > containers, but we aren't there yet.) We talked about this before. I would love to help the project because meanwhile, as you know and can see, I brought up a system with some, arguably ugly shortcuts. For example, I hard coded package dependencies, I install the packages by uncompressing them. I have a rudimental support for parallel build, but not for testing and so on. > > -- > David Smith > Principal Software Engineer > Red Hat ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Generating Kernel module for Other Computers 2017-07-04 2:53 ` Arkady @ 2017-07-05 13:20 ` David Smith 0 siblings, 0 replies; 8+ messages in thread From: David Smith @ 2017-07-05 13:20 UTC (permalink / raw) To: Arkady; +Cc: Daniel Doron, systemtap On Mon, Jul 3, 2017 at 9:53 PM, Arkady <arkady.miasnikov@gmail.com> wrote: >> (As I mentioned earlier, we're moving in the direction of using >> containers, but we aren't there yet.) > > We talked about this before. I would love to help the project because > meanwhile, as you know and can see, I brought up a system with some, > arguably ugly shortcuts. For example, I hard coded package > dependencies, I install the packages by uncompressing them. I have a > rudimental support for parallel build, but not for testing and so on. Feel free to look at the server side code in the 'httpd' subdirectory. I'm in the process now of trying to think through some of the container issues. Your prototype was interesting to look at. If you have any thoughts, let me know. -- David Smith Principal Software Engineer Red Hat ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-07-05 13:20 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-07-02 15:29 Generating Kernel module for Other Computers Daniel Doron 2017-07-02 17:10 ` Arkady 2017-07-02 17:21 ` Arkady 2017-07-03 16:17 ` David Smith 2017-07-03 17:18 ` Arkady 2017-07-03 18:49 ` David Smith 2017-07-04 2:53 ` Arkady 2017-07-05 13:20 ` David Smith
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).