ARM EABI Linux, breakpoints cause SIGILL and target dies

ARM EABI Linux, breakpoints cause SIGILL and target dies

[Nicholas Sherlock]

Hey all,

I have Ubuntu's userspace code running inside a chroot on top of
Android's Linux kernel on my phone. Using this, I've built GDB 7.3
from Ubuntu's source package. Here's the system details:

# uname -a
Linux localhost 2.6.35.13 #4 PREEMPT Wed Jan 18 20:25:06 PST 2012
armv7l armv7l armv7l GNU/Linux

# cat /proc/cpuinfo
Processor       : ARMv7 Processor rev 2 (v7l)
BogoMIPS        : 163.57
Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3
CPU implementer : 0x51
CPU architecture: 7
CPU variant     : 0x1
CPU part        : 0x00f
CPU revision    : 2

Hardware        : runnymede
Revision        : 0002
EngineerID      : 0001
Serial          : 0000000000000000

Here's my source file test.c:

#include <stdio.h>

int main(void) {
  printf("Hello, world!\n");
  printf("Goodbye, world!\n");
  return 0;
}

I'm building it on my target with this GCC:

# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro
4.5.2-8ubuntu4'
--with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.5 --enable-shared --enable-multiarch
--with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi
--without-included-gettext --enable-threads=posix
--with-gxx-include-dir=/usr/include/c++/4.5
--libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/
--enable-clocale=gnu --enable-libstdcxx-debug
--enable-libstdcxx-time=yes --enable-plugin --enable-gold
--enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc
--disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp
--with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror
--enable-checking=release --build=arm-linux-gnueabi
--host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)

And this command line:

gcc -g3 -static -o a.out test.c

Running a.out alone or with GDB works fine, but any operation that
causes GDB to set a breakpoint results in the target being killed by
SIGILL:

# gdb a.out
GNU gdb (Ubuntu/Linaro 7.3.1-2011.12-0ubuntu2) 7.3-2011.12
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabi".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /root/a.out...done.
(gdb) run
Starting program: /root/a.out
Hello, world!
Goodbye, world!
[Inferior 1 (process 14296) exited normally]
(gdb) start
Temporary breakpoint 1 at 0x81f4: file test.c, line 4.
Starting program: /root/a.out

Program received signal SIGILL, Illegal instruction.
0x000081f6 in main () at test.c:4
4         printf("Hello, world!\n");
(gdb) next

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.

I noticed that this patch was supposed to add support for SIGILL breakpoints:

http://old.nabble.com/RFC%3A-Updates-support-for-breakpoints-that-generate-SIGILL-p27363713.html

And the target-specific code says that it tries to generate a SIGILL
(arm-linux-tdep.c):

/* However, the EABI syscall interface (new in Nov. 2005) does not look at
   the operand of the swi if old-ABI compatibility is disabled.  Therefore,
   use an undefined instruction instead.  This is supported as of kernel
   version 2.5.70 (May 2003), so should be a safe assumption for EABI
   binaries.  */

So I would expect that this would work. Another piece of the puzzle, I
have a different phone here running a different Linux kernel, but the
same Ubuntu usermode binaries, where GDB breakpoints work perfectly:

$ uname -a
Linux localhost 2.6.37.6-cyanogenmod-01509-g8913be8 #1 PREEMPT Wed Jul
27 21:31:24 EDT 2011 armv7l GNU/Linux

$ cat /proc/cpuinfo

Processor       : ARMv7 Processor rev 2 (v7l)
BogoMIPS: 662.40
Features: swp half thumb fastmult vfp edsp thumbee neon
CPU implementer : 0x51
CPU architecture: 7
CPU variant     : 0x0
CPU part: 0x00f
CPU revision    : 2

Hardware: mahimahi
Revision: 0081
Serial  : 0000000000000000

How do I begin to debug this problem? I have the source code available
for both kernels if there is something to investigate there.

Thanks,
Nicholas Sherlock

Re: ARM EABI Linux, breakpoints cause SIGILL and target dies

[Ulrich Weigand]

Nicholas Sherlock wrote:

> Running a.out alone or with GDB works fine, but any operation that
> causes GDB to set a breakpoint results in the target being killed by
> SIGILL:
[snip]
> So I would expect that this would work. Another piece of the puzzle, I
> have a different phone here running a different Linux kernel, but the
> same Ubuntu usermode binaries, where GDB breakpoints work perfectly:
[snip]
> How do I begin to debug this problem? I have the source code available
> for both kernels if there is something to investigate there.

So there's two issues here:

- The kernel is supposed to recognize the special undefined instructions
  use to implement breakpoints, and deliver SIGTRAP instead of SIGILL if
  execution hits one of those.  It may be that the older of the two kernels
  does not properly handle this, in particular for Thumb-2 breakpoints
  which were added only recently.

  If you have the kernel sources, you might want to compare the routines
  installed via register_undef_hook in arch/arm/kernel/ptrace.c.

- Even on old kernels that return SIGILL, there is apparently some code
  in GDB that tries to recognize breakpoints anyway.  It may well be
  that this code does not (any longer) work correctly; it is never
  exercised on recent kernels, so a bug might have crept in ...

  Can you do a run with "set debug infrun 1" in the case where you
  get the SIGILL?


Bye,
Ulrich


-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: ARM EABI Linux, breakpoints cause SIGILL and target dies

[Nicholas Sherlock]

Hi Ulrich, thanks for the reply,

>  If you have the kernel sources, you might want to compare the routines
>  installed via register_undef_hook in arch/arm/kernel/ptrace.c.

So, the new kernel adds a bunch of code to support hardware
breakpoints, but I couldn't see any other changes there:

http://pastebin.com/HqTtdayG

> - Even on old kernels that return SIGILL, there is apparently some code
>  in GDB that tries to recognize breakpoints anyway.  It may well be
>  that this code does not (any longer) work correctly; it is never
>  exercised on recent kernels, so a bug might have crept in ...
>
>  Can you do a run with "set debug infrun 1" in the case where you
>  get the SIGILL?

Yep, here's the result:

Reading symbols from /root/a.out...done.
(gdb) set debug infrun 1
(gdb) start
Temporary breakpoint 1 at 0x81f4: file test.c, line 4.
Starting program: /root/a.out
infrun: proceed (addr=0x8170, signal=0, step=0)
infrun: resume (step=0, signal=0), trap_expected=0
infrun: wait_for_inferior (treat_exec_as_sigtrap=0)
infrun: target_wait (-1, status) =
infrun:   19563 [process 19563],
infrun:   status->kind = stopped, signal = SIGILL
infrun: infwait_normal_state
infrun: TARGET_WAITKIND_STOPPED
infrun: stop_pc = 0x81f6
infrun: random signal 4

Program received signal SIGILL, Illegal instruction.
infrun: stop_stepping
0x000081f6 in main () at test.c:4
4         printf("Hello, world!\n");
(gdb)

Cheers,
Nicholas Sherlock

Re: ARM EABI Linux, breakpoints cause SIGILL and target dies

[Nicholas Sherlock]

So, I patched this line in infrun.c which is supposed to check that
the SIGILL was caused by a breakpoint:

      if (breakpoint_inserted_here_p (get_regcache_aspace (regcache),
				      regcache_read_pc (regcache)))

And changed it to:

      if (ecs->ws.value.sig == TARGET_SIGNAL_ILL ||
breakpoint_inserted_here_p (get_regcache_aspace (regcache),
				      regcache_read_pc (regcache)))

Now the signal is corrected reinterpreted as SIGTRAP:

Program received signal SIGTRAP, Trace/breakpoint trap.
0x000081f6 in main () at test.c:4
4         printf("Hello, world!\n");

But the break address is incorrect, it's halfway through the code for
calling printf:

    81f4:       f24b 60a4       movw    r0, #46756      ; 0xb6a4
    81f8:       f2c0 0004       movt    r0, #4
    81fc:       f000 fce8       bl      8bd0 <_IO_puts>

So now I'm trying to fix the code for adjusting the PC after the trap.

Cheers,
Nicholas Sherlock

Re: ARM EABI Linux, breakpoints cause SIGILL and target dies

[Ulrich Weigand]

Nicholas Sherlock wrote:

> So, I patched this line in infrun.c which is supposed to check that
> the SIGILL was caused by a breakpoint:
> 
>       if (breakpoint_inserted_here_p (get_regcache_aspace (regcache),
> 				      regcache_read_pc (regcache)))
> 
> And changed it to:
> 
>       if (ecs->ws.value.sig == TARGET_SIGNAL_ILL ||
> breakpoint_inserted_here_p (get_regcache_aspace (regcache),
> 				      regcache_read_pc (regcache)))
> 
> Now the signal is corrected reinterpreted as SIGTRAP:
> 
> Program received signal SIGTRAP, Trace/breakpoint trap.
> 0x000081f6 in main () at test.c:4
> 4         printf("Hello, world!\n");
> 
> But the break address is incorrect, it's halfway through the code for
> calling printf:
> 
>     81f4:       f24b 60a4       movw    r0, #46756      ; 0xb6a4
>     81f8:       f2c0 0004       movt    r0, #4
>     81fc:       f000 fce8       bl      8bd0 <_IO_puts>

This is quite odd; it reports a breakpoint in the middle of a
Thumb-2 instruction.  On ARM, the *kernel* is supposed to adjust the
PC so that it points to the start of the breakpoint instructions.

If this does not work correctly for some reason, this would explain
both why GDB doesn't recognize the situation *and* also why the
kernel doesn't recognize the breakpoint instruction to convert the
signal into a SIGTRAP.

> So now I'm trying to fix the code for adjusting the PC after the trap.

GDB is not supposed to do any adjusting on ARM, this is done by the kernel;
see e.g. arch/arm/traps.c:

asmlinkage void __exception do_undefinstr(struct pt_regs *regs)
{
        unsigned int correction = thumb_mode(regs) ? 2 : 4;
        unsigned int instr;
        siginfo_t info;
        void __user *pc;

        /*
         * According to the ARM ARM, PC is 2 or 4 bytes ahead,
         * depending whether we're in Thumb mode or not.
         * Correct this offset.
         */
        regs->ARM_pc -= correction;


Do you see any differences either here in traps.c, or in one of the
entry-*.S assembler files, between your two kernels?

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: ARM EABI Linux, breakpoints cause SIGILL and target dies

[Nicholas Sherlock]

On 24 January 2012 02:29, Ulrich Weigand <uweigand@de.ibm.com> wrote:
> If this does not work correctly for some reason, this would explain
> both why GDB doesn't recognize the situation *and* also why the
> kernel doesn't recognize the breakpoint instruction to convert the
> signal into a SIGTRAP.

Yeah, that sounds likely.

> Do you see any differences either here in traps.c, or in one of the
> entry-*.S assembler files, between your two kernels?

Thanks for the pointer! I found the commit that fixed this bug in the
ARM kernel entry routine:

https://github.com/cyanogen/cm-kernel-exp/commit/d12aee22b3e025a4b76d4078f76542ea8cbd0315#diff-0

I'll port that patch over to my kernel and see if it fixes GDB.

Cheers,
Nicholas Sherlock

Re: ARM EABI Linux, breakpoints cause SIGILL and target dies

[Nicholas Sherlock]

On 24 January 2012 15:43, Nicholas Sherlock <n.sherlock@gmail.com> wrote:
> Thanks for the pointer! I found the commit that fixed this bug in the
> ARM kernel entry routine:
>
> https://github.com/cyanogen/cm-kernel-exp/commit/d12aee22b3e025a4b76d4078f76542ea8cbd0315#diff-0
>
> I'll port that patch over to my kernel and see if it fixes GDB.

The fix does indeed fix the issue and now GDB works perfectly. Thanks
for the help!

Cheers,
Nicholas Sherlock

Re: ARM EABI Linux, breakpoints cause SIGILL and target dies

[Ulrich Weigand]

Nicholas Sherlock rwote:
> On 24 January 2012 15:43, Nicholas Sherlock <n.sherlock@gmail.com> wrote:
> > Thanks for the pointer! I found the commit that fixed this bug in the
> > ARM kernel entry routine:
> >
> > https://github.com/cyanogen/cm-kernel-exp/commit/d12aee22b3e025a4b76d4078f76542ea8cbd0315#diff-0
> >
> > I'll port that patch over to my kernel and see if it fixes GDB.
> 
> The fix does indeed fix the issue and now GDB works perfectly. Thanks
> for the help!

Great, thanks for the follow-up!

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com