Incorrect unwind when throwing exceptions - possible cause?

Incorrect unwind when throwing exceptions - possible cause?

[Juraj Oršulić]

I have a hair-tearing problem with x86_64 gcc 9.3.0 (Ubuntu 20.04),
-O2 and the following "minimal" example
(the real minimal example also links in a bunch of 3rd party libraries
and is a part of a big project, so I can't really provide a minimal example,
but I slimmed it down to basically this):

class Application() {
public:
  Application(int argc, char **argv) {}

  int process() {
     if (argc > 1)  { throw std::runtime_error("Exception"); }

    // do some processing here
    return 0;
  }

int main(int argc, char **argv) {
  Application app(argc, argv);
  int result = 0;

  try {
    result = app.process();
  } catch (const std::exception &e) {
    result = 1;
    puts(e.what());
  }

  return result;
}

I get crashes if I throw and there's some more core after the if() in process().

This is an excerpt of how .eh_frame looks like when it works (90% code
in process() commented out):

00014c84 000000000000002c 00000000 CIE
  Version:               1
  Augmentation:          "zPLR"
  Code alignment factor: 1
  Data alignment factor: -8
  Return address column: 16
  Augmentation data:     9b 49 4f 3c 00 1b 1b
  DW_CFA_def_cfa: r7 (rsp) ofs 8
  DW_CFA_offset: r16 (rip) at cfa-8
  DW_CFA_def_cfa: r6 (rbp) ofs 16
  DW_CFA_offset: r3 (rbx) at cfa-56
  DW_CFA_offset: r6 (rbp) at cfa-16
  DW_CFA_offset: r12 (r12) at cfa-48
  DW_CFA_offset: r13 (r13) at cfa-40
  DW_CFA_offset: r14 (r14) at cfa-32
  DW_CFA_offset: r15 (r15) at cfa-24
  DW_CFA_nop
  DW_CFA_nop
  DW_CFA_nop

00014cb4 0000000000000014 00000034 FDE cie=00014c84
pc=000000000025081e..00000000002508d6
  Augmentation data:     d4 7e 20 00
  DW_CFA_nop
  DW_CFA_nop
  DW_CFA_nop

However, if I put just a bit more stuff inside process() after that
throwing if(), gcc changes the process function prologue a bit
and the unwind table looks like this:

0001ce38 0000000000000034 00013428 FDE cie=00009a14
pc=0000000000252534..0000000000256376
  Augmentation data:     ab 34 20 00
  DW_CFA_def_cfa_expression (DW_OP_breg6 (rbp): -40; DW_OP_deref)
  DW_CFA_expression: r3 (rbx) (DW_OP_breg6 (rbp): -48)
  DW_CFA_expression: r6 (rbp) (DW_OP_breg6 (rbp): 0)
  DW_CFA_expression: r12 (r12) (DW_OP_breg6 (rbp): -32)
  DW_CFA_expression: r13 (r13) (DW_OP_breg6 (rbp): -24)
  DW_CFA_expression: r14 (r14) (DW_OP_breg6 (rbp): -16)
  DW_CFA_expression: r15 (r15) (DW_OP_breg6 (rbp): -8)

So it basically references everything against rbp instead of against
the cfa, great. Here are the libunwind debug prints.
Notice how after processing the expression OP_breg(r6,0x0), thus
calculating the new value for r6 (rbp) by dereferencing itself with
offset zero, the change is reflected in the calculations for subsequent
registers r12-r15 which are wrong.
`apply_reg_state` in libunwind writes the new address for the value of
r6. However, the offsets for r12-r15 are supposed to be calculated
against the inner stack frame's value of rbp (0x7fffffffca70), not against
the new one (0x7fffffffcab0)!


     >_ULx86_64_reuse_frame: reuse frame ip=0x5555557a9d46
cfa=0x7fffffff6cc0 format=0 addr=0x0 offset=+0
              >_ULx86_64_dwarf_eval_expr: len=3, pushing cfa=0x7fffffff6cc0
               >_ULx86_64_dwarf_eval_expr: OP_breg(r6,0xffffffffffffffd8)
                >access_mem: mem[00007fffffff6c90] -> 7fffffffca70
               >_ULx86_64_dwarf_eval_expr: OP_deref
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fffffffca90
              >_ULx86_64_dwarf_eval_expr: len=2, pushing cfa=0x7fffffff6cc0
               >_ULx86_64_dwarf_eval_expr: OP_breg(r6,0xffffffffffffffd0)
                >access_mem: mem[00007fffffff6c90] -> 7fffffffca70
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fffffffca40
              >_ULx86_64_dwarf_eval_expr: len=2, pushing cfa=0x7fffffff6cc0
               >_ULx86_64_dwarf_eval_expr: OP_breg(r6,0x0)
                >access_mem: mem[00007fffffff6c90] -> 7fffffffca70
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fffffffca70
              >_ULx86_64_dwarf_eval_expr: len=2, pushing cfa=0x7fffffff6cc0
               >_ULx86_64_dwarf_eval_expr: OP_breg(r6,0xffffffffffffffe0)
                >access_mem: mem[00007fffffffca70] -> 7fffffffcab0
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fffffffca90
              >_ULx86_64_dwarf_eval_expr: len=2, pushing cfa=0x7fffffff6cc0
               >_ULx86_64_dwarf_eval_expr: OP_breg(r6,0xffffffffffffffe8)
                >access_mem: mem[00007fffffffca70] -> 7fffffffcab0
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fffffffca98
              >_ULx86_64_dwarf_eval_expr: len=2, pushing cfa=0x7fffffff6cc0
               >_ULx86_64_dwarf_eval_expr: OP_breg(r6,0xfffffffffffffff0)
                >access_mem: mem[00007fffffffca70] -> 7fffffffcab0
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fffffffcaa0
              >_ULx86_64_dwarf_eval_expr: len=2, pushing cfa=0x7fffffff6cc0
               >_ULx86_64_dwarf_eval_expr: OP_breg(r6,0xfffffffffffffff8)
                >access_mem: mem[00007fffffffca70] -> 7fffffffcab0
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fffffffcaa8
                >access_mem: mem[00007fffffffca88] -> 5555558b67a9
               >_ULx86_64_dwarf_step: returning 1
  >_ULx86_64_step: returning 1
              >_ULx86_64_dwarf_find_proc_info: looking for IP=0x5555558b67a8
               >_ULx86_64_dwarf_callback: checking , base=0x555555554000)
               >_ULx86_64_dwarf_callback: found table `':
segbase=0x555557215230, len=57926, gp=0x5555576567e8,
table_data=0x55555721523c
               >lookup: e->start_ip_offset = fffffffffebc5730
...........
               >lookup: e->start_ip_offset = fffffffffe6a1410
               >_ULx86_64_dwarf_search_unwind_table:
ip=0x5555558b67a8, start_ip=0xfffffffffe6a1410
 >_ULx86_64_dwarf_search_unwind_table: e->fde_offset = 8e0b0, segbase
= 555557215230, debug_frame_base = 0, fde_addr = 5555572a32e0
            >_ULx86_64_dwarf_extract_proc_info_from_fde: FDE @ 0x5555572a32e0
               >_ULx86_64_dwarf_extract_proc_info_from_fde: looking
for CIE at address 55555728fe84
               >parse_cie: CIE parsed OK, augmentation = "zPLR",
handler=0x7fffeeaf3a80
               >_ULx86_64_dwarf_extract_proc_info_from_fde: FDE covers
IP 0x5555558b6640-0x5555558b6867, LSDA=0x5555574a67ec
     >_ULx86_64_fetch_frame: fetch frame ip=0x5555558b67a9
cfa=0x7fffffffca90 format=0
                >access_mem: mem[00007fffffffca88] <- 5555558b67e9
 >_ULx86_64_resume: (cursor=0x7fffffff6870)
        >establish_machine_state: copying out cursor state
                >establish_machine_state: copying RBP 6
                >access_mem: mem[00007fffffffca70] -> 7fffffffcab0
            >access_reg: RBP <- 0x00007fffffffcab0
                >establish_machine_state: copying RSP 7
            >access_reg: RSP <- 0x00007fffffffca90
                >establish_machine_state: copying R8 8
                >access_mem: mem[00007fffffff64c8] -> 555557ced700
...  All of these are wrong and cause a crash after landing in the
exception handler in main():
            >access_reg: R12 <- 0x73746e6f662f6269
                >establish_machine_state: copying R13 13
                >access_mem: mem[00007fffffffca98] -> 257c32600
            >access_reg: R13 <- 0x0000000257c32600
                >establish_machine_state: copying R14 14
                >access_mem: mem[00007fffffffcaa0] -> 555557c325d0
            >access_reg: R14 <- 0x0000555557c325d0
                >establish_machine_state: copying R15 15
                >access_mem: mem[00007fffffffcaa8] -> 17ebc2371241f
            >access_reg: R15 <- 0x00017ebc2371241f
                >establish_machine_state: copying RIP 16
                >access_mem: mem[00007fffffffca88] -> 5555558b67e9
            >access_reg: RIP <- 0x00005555558b67e9

Does anyone have any ideas? What could be causing this?
I'm tearing my hair out. Thanks a lot!

Re: Incorrect unwind when throwing exceptions - possible cause?

[Sam Varshavchik]

[-- Attachment #1: Type: text/plain, Size: 646 bytes --]

Juraj Oršulić writes:

> I have a hair-tearing problem with x86_64 gcc 9.3.0 (Ubuntu 20.04),
> -O2 and the following "minimal" example
> (the real minimal example also links in a bunch of 3rd party libraries
> and is a part of a big project, so I can't really provide a minimal example,
> but I slimmed it down to basically this):
>
> class Application() {
> public:
>   Application(int argc, char **argv) {}
>
>   int process() {
>      if (argc > 1)  { throw std::runtime_error("Exception"); }
>
>     // do some processing here
>     return 0;
>   }

This was slimmed down too much. I don't see how this would compile.


[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: Incorrect unwind when throwing exceptions - possible cause?

[Juraj Oršulić]

Hi Sam, thanks for replying. The example wasn't meant to be compile
(sure, I missed the class closing brace and omitted storing argc).
That's not the point.
To reproduce it I would need to include a bunch of other code from the
codebase and put it inside ::process().
The point is that the mere presence of this code causes it to crash
if I throw right at the beginning -- I don't even execute any of that code.
I'm trying to figure out why the DWARF unwinding code becomes incorrect,
as I described in detail.



Sam Varshavchik wrote:

> This was slimmed down too much. I don't see how this would compile.

Re: Incorrect unwind when throwing exceptions - possible cause?

[Juraj Oršulić]

One additional detail: when I build in debug (-O0), it doesn't crash.
In that case, gcc decides to use an unwind table based on offsets
against the cfa instead of against rbp, which works fine:

     >_ULx86_64_fetch_frame: fetch frame ip=0x55555589afb7
cfa=0x7fffffff94d0 format=0
               >run_cfi_program: CFA_def_cfa r7+0x8
               >run_cfi_program: CFA_offset r16 at cfa+0xfffffffffffffff8
               >run_cfi_program: CFA_advance_loc to 0x5555558855b3
               >run_cfi_program: CFA_def_cfa_offset 0x10
               >run_cfi_program: CFA_offset r6 at cfa+0xfffffffffffffff0
               >run_cfi_program: CFA_advance_loc to 0x5555558855b6
               >run_cfi_program: CFA_def_cfa_register r6
               >run_cfi_program: CFA_advance_loc to 0x5555558855e6
               >run_cfi_program: CFA_offset r13 at cfa+0xffffffffffffffe8
               >run_cfi_program: CFA_offset r12 at cfa+0xffffffffffffffe0
               >run_cfi_program: CFA_offset r3 at cfa+0xffffffffffffffd8




On Thu, Feb 3, 2022 at 2:36 AM Juraj Oršulić <juraj.orsulic@fer.hr> wrote:
>
> Hi Sam, thanks for replying. The example wasn't meant to be compile
> (sure, I missed the class closing brace and omitted storing argc).
> That's not the point.
> To reproduce it I would need to include a bunch of other code from the
> codebase and put it inside ::process().
> The point is that the mere presence of this code causes it to crash
> if I throw right at the beginning -- I don't even execute any of that code.
> I'm trying to figure out why the DWARF unwinding code becomes incorrect,
> as I described in detail.
>
>
>
> Sam Varshavchik wrote:
>
> > This was slimmed down too much. I don't see how this would compile.

Re: Incorrect unwind when throwing exceptions - possible cause?

[Florian Weimer]

* Juraj Oršulić:

> I have a hair-tearing problem with x86_64 gcc 9.3.0 (Ubuntu 20.04),
> -O2 and the following "minimal" example
> (the real minimal example also links in a bunch of 3rd party libraries
> and is a part of a big project, so I can't really provide a minimal example,
> but I slimmed it down to basically this):

Are you using the libgcc unwinder in the crashing case?  (And not
libunwind.)

Thanks,
Florian

Re: Incorrect unwind when throwing exceptions - possible cause?

[Juraj Oršulić]

Sorry for replying privately by mistake.

Thank you so much Florian, I was not aware that libunwind is not the default
unwinder. That could be the key to this and looks like we're linking it probably
due to a 3rd party dependency.

Looks like I made a mistake by reducing our project to an exception
hello-world crashing minimum example instead of building one externally.

In fact, I did, but I saw that _Unwind_RaiseException is called directly in
__cxa_throw in libstdc++, so I assumed that is how it normally goes.
Now I see that gcc provides a built-in for this function.


On Thu, Feb 3, 2022 at 1:26 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Juraj Oršulić:
>
> > I haven’t specified anything, so it’s using libunwind.so by default.
> > I have stepped through libunwind and observed the moment of curruption when it
> > prematurely commits the new rbp value.
>
> The default for GCC is its built-in unwinder.  If you are using
> libunwind, you need to talk to the libunwind developers.
>
> Please keep posting to the list.
>
> Thanks,
> Florian
>

Re: Incorrect unwind when throwing exceptions - possible cause?

[Juraj Oršulić]

Just to confirm, the problem was solved by adjusting the 3rdparty library
guilty for pulling in libunwind (in our case, Google glog, which has it as
an optional dependency) and evicting libunwind from the link list
(and using gcc's builtin unwinder instead).

It looks like that an up-to-date libunwind on Arch as of today doesn't have
this bug, but it's certainly present on Ubuntu 20.04.

Thanks again Florian!

On Thu, Feb 3, 2022 at 2:40 PM Juraj Oršulić <juraj.orsulic@fer.hr> wrote:
>
> Sorry for replying privately by mistake.
>
> Thank you so much Florian, I was not aware that libunwind is not the default
> unwinder. That could be the key to this and looks like we're linking it probably
> due to a 3rd party dependency.
>
> Looks like I made a mistake by reducing our project to an exception
> hello-world crashing minimum example instead of building one externally.
>
> In fact, I did, but I saw that _Unwind_RaiseException is called directly in
> __cxa_throw in libstdc++, so I assumed that is how it normally goes.
> Now I see that gcc provides a built-in for this function.
>
>
> On Thu, Feb 3, 2022 at 1:26 PM Florian Weimer <fweimer@redhat.com> wrote:
> >
> > * Juraj Oršulić:
> >
> > > I haven’t specified anything, so it’s using libunwind.so by default.
> > > I have stepped through libunwind and observed the moment of curruption when it
> > > prematurely commits the new rbp value.
> >
> > The default for GCC is its built-in unwinder.  If you are using
> > libunwind, you need to talk to the libunwind developers.
> >
> > Please keep posting to the list.
> >
> > Thanks,
> > Florian
> >

Re: Incorrect unwind when throwing exceptions - possible cause?

[Andrew Haley]

On 2/3/22 15:38, Juraj Oršulić wrote:
> Just to confirm, the problem was solved by adjusting the 3rdparty library
> guilty for pulling in libunwind (in our case, Google glog, which has it as
> an optional dependency) and evicting libunwind from the link list
> (and using gcc's builtin unwinder instead).
> 
> It looks like that an up-to-date libunwind on Arch as of today doesn't have
> this bug, but it's certainly present on Ubuntu 20.04.
> 
> Thanks again Florian!

Yes, I agree, that was a great catch.

-- 
Andrew Haley  (he/him)
Java Platform Lead Engineer
Red Hat UK Ltd. <https://www.redhat.com>
https://keybase.io/andrewhaley
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671