[Bug translator/24720] New: stap Can't insert module and exit with error "Invalid module format"

[Bug translator/24720] New: stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

            Bug ID: 24720
           Summary: stap Can't insert module and exit with error "Invalid
                    module format"
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: translator
          Assignee: systemtap at sourceware dot org
          Reporter: gromero at linux dot vnet.ibm.com
  Target Milestone: ---

stap can't install module in Pass 5 and fails with "Invalid module format":

gromero@2q14gto1:~/systemtap/bin$ sudo ./stap -g -k -v -r
/home/gromero/git/linux tm.d
Pass 1: parsed user script and 496 library scripts using
101632virt/89920res/7936shr/81472data kb, in 520usr/10sys/529real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using
148544virt/139200res/8960shr/128384data kb, in 880usr/10sys/881real ms.
Pass 3: translated to C into "/tmp/stap1fSq2k/stap_25649_src.c" using
148544virt/139456res/9216shr/128384data kb, in 0usr/10sys/11real ms.
Pass 4: compiled C into "stap_25649.ko" in 2690usr/310sys/3134real ms.
Pass 5: starting run.
ERROR: Couldn't insert module '/tmp/stap1fSq2k/stap_25649.ko': Invalid module
format
WARNING: /home/gromero/systemtap/bin/staprun exited with status: 1
Pass 5: run completed in 10usr/0sys/3real ms.
Pass 5: run failed.  [man error::pass5]
Keeping temporary directory "/tmp/stap1fSq2k"

If Guru mode is disabled it installs correctly:

gromero@2q14gto1:~/systemtap/bin$ sudo ./stap -k -v -r /home/gromero/git/linux
tm.d
Pass 1: parsed user script and 496 library scripts using
101568virt/89856res/7936shr/81408data kb, in 510usr/10sys/525real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using
148416virt/139136res/8960shr/128256data kb, in 860usr/10sys/873real ms.
Pass 3: translated to C into "/tmp/stap2dpqpG/stap_22990_src.c" using
148416virt/139392res/9216shr/128256data kb, in 10usr/0sys/11real ms.
Pass 4: compiled C into "stap_22990.ko" in 14460usr/900sys/15826real ms.
Pass 5: starting run.
tm_unavailable() -> apache2
tm_unavailable() -> apache2
tm_unavailable() -> apache2
^CPass 5: run completed in 0usr/20sys/10304real ms.
Keeping temporary directory "/tmp/stap2dpqpG"

Also, passing flags "-B CONFIG_MODVERSIONS=y" with Guru mode work-around the
issue:

gromero@2q14gto1:~/systemtap/bin$ sudo ./stap -B CONFIG_MODVERSIONS=y -g -k -v
-r /home/gromero/git/linux tm.d
Pass 1: parsed user script and 496 library scripts using
101568virt/89920res/7936shr/81408data kb, in 520usr/10sys/528real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using
148480virt/139136res/8960shr/128320data kb, in 860usr/10sys/882real ms.
Pass 3: translated to C into "/tmp/stapXWnZ4f/stap_24910_src.c" using
148480virt/139392res/9216shr/128320data kb, in 10usr/10sys/11real ms.
Pass 4: compiled C into "stap_24910.ko" in 14420usr/900sys/15758real ms.
Pass 5: starting run.
tm_unavailable() -> apache2
tm_unavailable() -> apache2
tm_unavailable() -> apache2
tm_unavailable() -> apache2

Probe in use is trivial but will only work on recent PPC64 kernels:

gromero@2q14gto1:~/systemtap/bin$ cat tm.d 
probe kernel.function("tm_unavailable").inline {
  printf("tm_unavailable() -> %s\n", execname());
}

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

Gustavo Romero <gromero at linux dot vnet.ibm.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gromero at linux dot vnet.ibm.com

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #1 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Additional information on kernel, gcc, and stap version used:

gromero@2q14gto1:~/systemtap/bin$ ./stap --version
Systemtap translator/driver (version 4.2/0.168, commit
release-4.1-35-g82354b99d037)
Copyright (C) 2005-2019 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 5.1-rc2
enabled features: AVAHI BPF LIBSQLITE3 NLS NSS
gromero@2q14gto1:~/systemtap/bin$ gcc -v 
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/powerpc64le-linux-gnu/6/lto-wrapper
Target: powerpc64le-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 6.3.0-18+deb9u1'
--with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs
--enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-6 --program-prefix=powerpc64le-linux-gnu- --enable-shared
--enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/
--enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes
--with-default-libstdcxx-abi=new --enable-gnu-unique-object
--disable-libquadmath --enable-plugin --enable-default-pie --with-system-zlib
--disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-ppc64el/jre --enable-java-home
--with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-ppc64el
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-ppc64el
--with-arch-directory=ppc64le --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
--enable-objc-gc=auto --enable-secureplt --with-cpu=power8
--enable-targets=powerpcle-linux --disable-multilib --enable-multiarch
--with-long-double-128 --enable-checking=release --build=powerpc64le-linux-gnu
--host=powerpc64le-linux-gnu --target=powerpc64le-linux-gnu
Thread model: posix
gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 
gromero@2q14gto1:~/systemtap/bin$ uname -a
Linux 2q14gto1 5.2.0-rc5-debuginfo-00239-g241e39004581 #4 SMP Fri Jun 21
12:06:45 CDT 2019 ppc64le GNU/Linux

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #2 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Created attachment 11856
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11856&action=edit
kernel config

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #3 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Created attachment 11857
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11857&action=edit
stap-report wo/ root

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #4 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Created attachment 11858
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11858&action=edit
stap-report w/ root

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #5 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
Git source is also in sync with booted kernel:

gromero@2q14gto1:~/git/linux$ git describe 
v5.2-rc5-239-g241e39004581
gromero@2q14gto1:~/git/linux$ uname -a
Linux 2q14gto1 5.2.0-rc5-debuginfo-00239-g241e39004581 #4 SMP Fri Jun 21
12:06:45 CDT 2019 ppc64le GNU/Linux

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

Serhei Makarov <me at serhei dot io> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2019-06-21
                 CC|                            |me at serhei dot io
     Ever confirmed|0                           |1

--- Comment #6 from Serhei Makarov <me at serhei dot io> ---
Thanks for providing the detailed info. This problem was caused by the prior
commit [1] attempting to fix a much rarer problem (a guru-mode module compiled
for one kernel can sometimes run on a different kernel version if there is no
difference in the ABIs checked by modsymversioning).

[1]:
https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=c8084763b75fe4c8c61a0ddfa69e2d13f04de18e;hp=3a29a2eef30f191afbc1bc117b6e5a2812dfe91b

For now I reverted the commit in SystemTap git master since it clearly had
undesirable results. I'll use your info to reproduce the problem for myself and
see if there's a better way to fix it that works across all the different
configs. Keeping the PR open to track that work.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[gromero at linux dot vnet.ibm.com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #7 from Gustavo Romero <gromero at linux dot vnet.ibm.com> ---
(In reply to Serhei Makarov from comment #6)
> Thanks for providing the detailed info. This problem was caused by the prior
> commit [1] attempting to fix a much rarer problem (a guru-mode module
> compiled for one kernel can sometimes run on a different kernel version if
> there is no difference in the ABIs checked by modsymversioning).
> 
> [1]:
> https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=c8084763b75fe4c8c61a0ddfa69e2d13f04de18e;
> hp=3a29a2eef30f191afbc1bc117b6e5a2812dfe91b
> 
> For now I reverted the commit in SystemTap git master since it clearly had
> undesirable results. I'll use your info to reproduce the problem for myself
> and see if there's a better way to fix it that works across all the
> different configs. Keeping the PR open to track that work.

Thanks for the analysis and explanations, Serhei!

If you need any help on reproducing/testing it please let me know.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/24720] stap Can't insert module and exit with error "Invalid module format"

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=24720

--- Comment #8 from Serhei Makarov <me at serhei dot io> ---
Checked back on this since I want to re-enable the strict vermagic checking for
the upcoming stap release. Unfortunately, what prevented diagnosing this issue
back when this PR was filed is that stap -v does not share the full error
message when loading the module, only the last line (based on the modprobe
error code, see moderror() in staprun/staprun_func.c).

One example of a situation where vermagic might be mismatched (on ppc,
interestingly enough, just as with the original problem report) has a full
message along the following lines:

  # insmod autofs4-pg.ko
  autofs4: version magic
  '4.11.0-rc3-gcc-5.4.1-00017-g5a61ef74f269 SMP mod_unload modversions '
  should be
  '4.11.0-rc3-gcc-5.4.1-00017-g5a61ef74f269-dirty SMP mod_unload modversions
mprofile-kernel'
  insmod: ERROR: could not insert module autofs4-pg.ko: Invalid module format

https://github.com/torvalds/linux/commit/43e24e82f35291d4c1ca78877ce1b20d3aeb78f1

Not sure if it's related to the original reported situation.

* * *

I went back to a powerpc machine to repro the problem, but enabling the strict
vermagic check (the original cause of the issue) worked fine for me, so there
is some additional condition present on the machine where it was reported.

-- 
You are receiving this mail because:
You are the assignee for the bug.