Build-id mismatch: "kernel" vs. "vmlinux" byte 0 (0x82 vs 0x47)

Build-id mismatch: "kernel" vs. "vmlinux" byte 0 (0x82 vs 0x47)

[zhenzhong.duan]

Hi maillist,
This issue was originally found by our partner. I reproduced it with
upstream systemtap. Any idea on this?

I run with newest systemtap and it fail with below msg.
[root@zhenzhong1 ~]# ./systemtap-2.0-5283/bin/stap -ge 'probe
kernel.function("sys_read"){printf("aa");}'
ERROR: Build-id mismatch: "kernel" vs. "vmlinux" byte 0 (0x82 vs 0x47)
address 0xffffffff8151236c rc 0
WARNING: /root/systemtap-2.0-5283/bin/staprun exited with status: 1
Pass 5: run failed. Try again with another '--vp 00001' option.

[root@zhenzhong1 ~]# eu-readelf -n
/usr/lib/debug/lib/modules/2.6.39-200.29.2.el6uek.x86_64/vmlinux |grep Build
Build ID: 825a617a9ee7c66883b0538d241ee012f4ff340f
[root@zhenzhong1 ~]# eu-readelf -n
/boot/vmlinuz-2.6.39-200.29.2.el6uek.x86_64 |grep Build
Build ID: 4769e7a19cb104e200cd082583af38795574c761

I'm sure kernel and debuginfo match eath other.
Linux version 2.6.39-200.29.2.el6uek.x86_64
(mockbuild@ca-build44.us.oracle.com) (gcc version 4.4.6 20110731 (Red
Hat 4.4.6-3) (GCC) ) #1 SMP Sat Jul 14 10:50:56 EDT 2012

regards
zduan

Re: Build-id mismatch: "kernel" vs. "vmlinux" byte 0 (0x82 vs 0x47)

[Josh Stone]

Hi zduan,

On 07/23/2012 08:07 PM, zhenzhong.duan wrote:
> Hi maillist,
> This issue was originally found by our partner. I reproduced it with
> upstream systemtap. Any idea on this?
> 
> I run with newest systemtap and it fail with below msg.
> [root@zhenzhong1 ~]# ./systemtap-2.0-5283/bin/stap -ge 'probe
> kernel.function("sys_read"){printf("aa");}'
> ERROR: Build-id mismatch: "kernel" vs. "vmlinux" byte 0 (0x82 vs 0x47)
> address 0xffffffff8151236c rc 0
> WARNING: /root/systemtap-2.0-5283/bin/staprun exited with status: 1
> Pass 5: run failed. Try again with another '--vp 00001' option.
> 
> [root@zhenzhong1 ~]# eu-readelf -n
> /usr/lib/debug/lib/modules/2.6.39-200.29.2.el6uek.x86_64/vmlinux |grep Build
> Build ID: 825a617a9ee7c66883b0538d241ee012f4ff340f
> [root@zhenzhong1 ~]# eu-readelf -n
> /boot/vmlinuz-2.6.39-200.29.2.el6uek.x86_64 |grep Build
> Build ID: 4769e7a19cb104e200cd082583af38795574c761

This is good information, thank you.  However, it shows that systemtap
is correct that the Build IDs are not the same.  The stap error message
reported "0x82 vs 0x47", and indeed the first bytes that you found in
the files show that exact difference.

> I'm sure kernel and debuginfo match eath other.
> Linux version 2.6.39-200.29.2.el6uek.x86_64
> (mockbuild@ca-build44.us.oracle.com) (gcc version 4.4.6 20110731 (Red
> Hat 4.4.6-3) (GCC) ) #1 SMP Sat Jul 14 10:50:56 EDT 2012

Unfortunately, it is still possible for these to be mismatched, even if
the version strings are identical.  In order for the Build IDs to match,
the packages must come from the exact same compilation run.  One clue to
check this is the "Build Date" information in the rpms, which you may
find with this command:

rpm -qif /boot/vmlinuz-2.6.39-200.29.2.el6uek.x86_64 \
  /usr/lib/debug/lib/modules/2.6.39-200.29.2.el6uek.x86_64/vmlinux

It probably won't match the exact "10:50:56 EDT" time given in your
kernel string, but rather some time after that when the rpms are
actually written.  The kernel and kernel-debuginfo rpms should be the
same build date and time as each other though.

Hope this helps, and let us know what that says.

Josh

Re: Build-id mismatch: "kernel" vs. "vmlinux" byte 0 (0x82 vs 0x47)

[zhenzhong.duan]

Hi Josh,
thanks for your comment.
>> I'm sure kernel and debuginfo match eath other.
>> Linux version 2.6.39-200.29.2.el6uek.x86_64
>> (mockbuild@ca-build44.us.oracle.com) (gcc version 4.4.6 20110731 (Red
>> Hat 4.4.6-3) (GCC) ) #1 SMP Sat Jul 14 10:50:56 EDT 2012
> Unfortunately, it is still possible for these to be mismatched, even if
> the version strings are identical.  In order for the Build IDs to match,
> the packages must come from the exact same compilation run.  One clue to
> check this is the "Build Date" information in the rpms, which you may
> find with this command:
>
> rpm -qif /boot/vmlinuz-2.6.39-200.29.2.el6uek.x86_64 \
>    /usr/lib/debug/lib/modules/2.6.39-200.29.2.el6uek.x86_64/vmlinux
>
> It probably won't match the exact "10:50:56 EDT" time given in your
> kernel string, but rather some time after that when the rpms are
> actually written.  The kernel and kernel-debuginfo rpms should be the
> same build date and time as each other though.
>
> Hope this helps, and let us know what that says.
>
> Josh
As you predict, kernel string/kernel-uek pkg/kernel-uek-debuginfo pkg's 
time signature are all different.
In fact, 2.6.39-200.29.2.el6uek are a variant of upstream 3.2.5 or 
later. So this maybe an upstream issue.

[root@zhenzhong1 ~]# rpm -qif /boot/vmlinuz-2.6.39-200.29.2.el6uek.x86_64 \
 >   /usr/lib/debug/lib/modules/2.6.39-200.29.2.el6uek.x86_64/vmlinux
Name        : kernel-uek                   Relocations: (not relocatable)
Version     : 2.6.39                            Vendor: Oracle America
Release     : 200.29.2.el6uek               Build Date: Sat 14 Jul 2012 
11:33:03 AM EDT
Install Date: Fri 20 Jul 2012 04:52:29 AM EDT      Build Host: 
ca-build44.us.oracle.com
Group       : System Environment/Kernel     Source RPM: 
kernel-uek-2.6.39-200.29.2.el6uek.src.rpm
Size        : 103285531                        License: GPLv2
Signature   : RSA/8, Sat 14 Jul 2012 11:50:15 AM EDT, Key ID 
72f97b74ec551f03
URL         : http://www.kernel.org/
Summary     : The Linux kernel
...
Name        : kernel-uek-debuginfo         Relocations: (not relocatable)
Version     : 2.6.39                            Vendor: Oracle America
Release     : 200.29.2.el6uek               Build Date: Sat 14 Jul 2012 
11:33:03 AM EDT
Install Date: Mon 23 Jul 2012 05:40:40 AM EDT      Build Host: 
ca-build44.us.oracle.com
Group       : Development/Debug             Source RPM: 
kernel-uek-2.6.39-200.29.2.el6uek.src.rpm
Size        : 1553026121                       License: GPLv2
Signature   : RSA/8, Sat 14 Jul 2012 11:50:02 AM EDT, Key ID 
72f97b74ec551f03
URL         : http://www.kernel.org/
Summary     : Debug information for package kernel-uek

Re: Build-id mismatch: "kernel" vs. "vmlinux" byte 0 (0x82 vs 0x47)

[Josh Stone]

On 07/23/2012 11:10 PM, zhenzhong.duan wrote:
> As you predict, kernel string/kernel-uek pkg/kernel-uek-debuginfo pkg's 
> time signature are all different.

Hmm, that's not exactly what I meant.  The kernel string has a different
time, yes - that's to be expected.  The two rpm's actually have the same
"Build Date: Sat 14 Jul 2012 11:33:03 AM EDT" though, which indicates
that they actually should match up.

It's OK that the Signature dates are different.  That's just when the
PGP/GPG signatures are added at a later stage, and they don't have to
line up.

So since they're from the same build, it's really odd that the buildids
don't match.  Perhaps try "rpm -V kernel-uek kernel-uek-debuginfo" to
make sure that the files are still consistent with the rpm.

> In fact, 2.6.39-200.29.2.el6uek are a variant of upstream 3.2.5 or 
> later. So this maybe an upstream issue.

I doubt it's upstream -- this seems like purely a packaging issue to me.
 Fedora has gone through those kernel versions, and is even newer now,
but hasn't had this problem.

I see in Fedora's kernel.spec that there's a kludged call to debugedit
to preserve a consistent buildid, so maybe that's broken in uek.

Josh