Re: [PATCH][?/n] LTO type merging cleanup

[Jan Hubicka <hubicka@ucw.cz>]

> 
> We can end up with an infinite recursion as gimple_register_type
> tries to register TYPE_MAIN_VARIANT first.  This is because we
> are being called from the LTO type-fixup code which walks the
> type graph and adjusts types to their leaders.  So we can
> be called for type SCCs that are only partially fixed up yet
> which means TYPE_MAIN_VARIANT might temporarily not honor
> the invariant that the main variant of a main variant is itself.
> Thus, simply avoid recursing more than once - we are sure that
> we will be reaching at most type duplicates in further recursion.
> 
> Bootstrap & regtest pending on x86_64-unknown-linux-gnu.

With this funcion WPA stage passes with some improvements I repported to mozilla metabug.

We now get ICE in ltrans:
#0  gimple_register_type (t=0x0) at ../../gcc/gimple.c:4616
#1  0x00000000005a0fc9 in gimple_register_canonical_type (t=0x7fffe851f498) at ../../gcc/gimple.c:4890
#2  0x000000000048f14d in lto_ft_type (t=0x7fffe851f498) at ../../gcc/lto/lto.c:401
#3  lto_fixup_types (t=0x7fffe851f498) at ../../gcc/lto/lto.c:581
#4  0x000000000048f4a0 in uniquify_nodes (node=Unhandled dwarf expression opcode 0xf3

TYPE_MAIN_VARIANT is NULL.
(gdb) up
#1  0x00000000005a0fc9 in gimple_register_canonical_type (t=0x7fffe851f498) at ../../gcc/gimple.c:4890
4890      t = gimple_register_type (TYPE_MAIN_VARIANT (t));
(gdb) p debug_generic_stmt (t)
struct _ffi_type

$1 = void
(gdb) p debug_tree (t)
 <record_type 0x7fffe851f498 _ffi_type BLK
    size <integer_cst 0x7ffff7ecf680 type <integer_type 0x7ffff7eca0a8 bit_size_type> constant 192>
    unit size <integer_cst 0x7ffff7ecf640 type <integer_type 0x7ffff7eca000> constant 24>
    align 64 symtab 0 alias set -1 structural equality
    fields <field_decl 0x7fffe87684c0 size
        type <integer_type 0x7ffff7eca690 long unsigned int public unsigned DI
            size <integer_cst 0x7ffff7ecf1e0 constant 64>
            unit size <integer_cst 0x7ffff7ecf200 constant 8>
            align 64 symtab 0 alias set -1 canonical type 0x7ffff7eca690 precision 64 min <integer_cst 0x7ffff7ecf220 0> max <integer_cst 0x7ffff7ecf1c0 18446744073709551615>
            pointer_to_this <pointer_type 0x7ffff5336150> reference_to_this <reference_type 0x7ffff0aba000>>
        used unsigned nonlocal DI file ctypes/libffi/include/ffi.h line 109 col 0 size <integer_cst 0x7ffff7ecf1e0 64> unit size <integer_cst 0x7ffff7ecf200 8>
        align 64 offset_align 128
        offset <integer_cst 0x7ffff7ebaf00 constant 0>
        bit offset <integer_cst 0x7ffff7ecf420 constant 0> context <record_type 0x7fffe851f2a0 _ffi_type>
        chain <field_decl 0x7fffe8768558 alignment type <integer_type 0x7ffff7eca3f0 short unsigned int>
            used unsigned nonlocal HI file ctypes/libffi/include/ffi.h line 110 col 0
            size <integer_cst 0x7ffff7ecf080 constant 16>
            unit size <integer_cst 0x7ffff7ecf0a0 constant 2>
            align 16 offset_align 128 offset <integer_cst 0x7ffff7ebaf00 0> bit offset <integer_cst 0x7ffff7ecf1e0 64> context <record_type 0x7fffe851f2a0 _ffi_type> chain <field_decl 0x7fffe87685f0 type>>>
    chain <type_decl 0x7fffe8966ac8 _ffi_type>>
$2 = void

Let me know if there is anything easy I could work out ;)
I think the bug may be in the recursion guard.  When you have cycle of length
greater than 2 of MVs, you won't walk them all.

Honza