internal compiler error: in referenced_var_lookup, at tree-dfa.c

internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Jay K]

So..we have a custom frontend.
That uses process boundaries to avoid GPL crossing into BSDish licensed code.
So maybe you don't want to help me. Understood.

But just in case:

We generate trees. Probably they aren't of great quality.
e.g. relatively devoid of types and do field accesses by offseting pointers and casting.
We do our own layout earlier. Not great, but that's how it is.


Currently we use gcc 4.5.1.
When I enable inlining on some architectures, e.g. Macosx/x86, much code yields:


../FPrint.m3: In function 'FPrint__xCombine':
../FPrint.m3:25:0: internal compiler error: in referenced_var_lookup, at tree-dfa.c:525
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.


Now I've spent a while debugging this. I'm not completely lazy.


The original source (again, not a standard frontend):

PROCEDURE xCombine (): INTEGER =
  BEGIN
    RETURN xFix32 (0);
  END xCombine;

PROCEDURE xFix32 (x: INTEGER): INTEGER =
  CONST a = 1;
  BEGIN
   IF Word.And (x, a) = 0 THEN
     RETURN 0;
   END;
   RETURN 0;
  END xFix32;

 -fdump-tree-all, 003t.original:

FPrint__xCombine ()
{
  int_32 D.985;
  int_32 M3_AcxOUs__result;

    int_32 M3_AcxOUs__result;
    int_32 D.985;
  D.985 = FPrint__xFix32 (0);
  <retval> = D.985;
  return <retval>;
}


FPrint__xFix32 (int_32 M3_AcxOUs_x)
{
  int_32 M3_AcxOUs__result;

    int_32 M3_AcxOUs__result;
  if ((M3_AcxOUs_x & 1) != 0)
    {
      goto <D.986>;
    }
  <retval> = 0;
  return <retval>;
  <D.986>:;
  <retval> = 0;
  return <retval>;
}



040t.release_ssa:
;; Function FPrint__xFix32 (FPrint__xFix32)

Released 0 names, 0.00%
FPrint__xFix32 (int_32 M3_AcxOUs_x)
{
  int_32 D.987;

<bb 2>:
  D.987_3 = M3_AcxOUs_x_2(D) & 1;
  if (D.987_3 != 0)
    goto <bb 4> (<L1>);
  else
    goto <bb 3>;

<bb 3>:
  <retval>_4 = 0;
  goto <bb 5>;

<L1>:
  <retval>_5 = 0;

<bb 5>:
  # <retval>_1 = PHI <<retval>_4(3), <retval>_5(4)>
  return <retval>_1;

}


Breakpoint 1, 0x0078f21a in copy_phis_for_bb (bb=0x4126a280, id=0xbffff5d8) at ../../gcc-4.5/gcc/tree-inline.c:1937
1937    {
(gdb) n
1938      basic_block const new_bb = (basic_block) bb->aux;
(gdb) 
1943      jaykrell_check_1();  doesn't do anything now, was helping debug 
(gdb) 
1945      for (si = gsi_start (phi_nodes (bb)); !gsi_end_p (si); gsi_next (&si))
(gdb) 
1947          tree res = { 0 }, new_res = { 0 };
(gdb) 
1948          gimple new_phi = { 0 };
(gdb) 
1949          edge new_edge = { 0 };
(gdb) 
1951          phi = gsi_stmt (si);
(gdb) 
1952          res = PHI_RESULT (phi);
(gdb) 
1953          new_res = res;
(gdb) 
1954          if (is_gimple_reg (res))
(gdb) 
1956          walk_tree (&new_res, copy_tree_body_r, id, NULL);
(gdb) 
1957          new_phi = create_phi_node (new_res, new_bb);  This line split up locally, ok.
(gdb) 
1958          SSA_NAME_DEF_STMT (new_res) = new_phi;
(gdb) call debug_referenced_vars()

Referenced variables in FPrint__xCombine: 7

Variable: D.1036, UID D.1036, int_32gimple_default_def 0x412130a8 1036

Variable: D.1041, UID D.1041, int_32gimple_default_def 0x412130a8 1041

Variable: .MEM, UID D.1038, <unnamed type>, is global, call clobberedgimple_default_def 0x412130a8 1038

Variable: M3_AcxOUs_x, UID D.1039, int_32gimple_default_def 0x412130a8 1039

Variable: D.1040, UID D.1040, int_32gimple_default_def 0x412130a8 1040

Variable: <retval>, UID D.979, int_32gimple_default_def 0x412130a8 979

Variable: D.985, UID D.985, int_32gimple_default_def 0x412130a8 985


(gdb) n
1959          FOR_EACH_EDGE (new_edge, ei, new_bb->preds)
(gdb) call debug_referenced_vars()

Referenced variables in FPrint__xCombine: 7

Variable: D.1036, UID D.1036, int_32gimple_default_def 0x412130a8 1036

Variable: D.1041, UID D.1041, int_32gimple_default_def 0x412130a8 1041

Variable: .MEM, UID D.1038, <unnamed type>, is global, call clobberedgimple_default_def 0x412130a8 1038

Variable: M3_AcxOUs_x, UID D.1039, int_32gimple_default_def 0x412130a8 1039

Variable: D.1093058884, UID D.1093058884, int_32gimple_default_def 0x412130a8 1093058884

Variable: <retval>, UID D.979, int_32gimple_default_def 0x412130a8 979

Variable: D.985, UID D.985, int_32gimple_default_def 0x412130a8 985


You can see D.1040 got overwritten with something else.
  And then later the assertion is that it is missing.
Is it valid for uids to be so high?


Any clues/tips?


Thanks much,
 - Jay
 		 	   		  

Re: internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Ian Lance Taylor]

Jay K <jay.krell@cornell.edu> writes:

> That uses process boundaries to avoid GPL crossing into BSDish licensed code.
> So maybe you don't want to help me. Understood.

Note that different people have different opinions as to whether a
process boundary means that your code is not a derived work.  Not that
we should get into that discussion on this mailing list.


> Variable: D.1093058884, UID D.1093058884, int_32gimple_default_def 0x412130a8 1093058884

This is clearly wrong, though I have no idea what caused it.

> Is it valid for uids to be so high?

No.

From your description, you've implemented some sort of customized tree
reader.  Does it play nicely with the garbage collector?

Ian

Re: internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Robert Dewar]

On 9/10/2010 11:08 AM, Ian Lance Taylor wrote:
> Jay K<jay.krell@cornell.edu>  writes:
>
>> That uses process boundaries to avoid GPL crossing into BSDish licensed code.
>> So maybe you don't want to help me. Understood.
>
> Note that different people have different opinions as to whether a
> process boundary means that your code is not a derived work.  Not that
> we should get into that discussion on this mailing list.

Indeed, it is important to realize that putting in an arbitrary
process boundary is not guarantee at all that you have not created
a derived work! You need to consult a knowledgable attorney before
proceeding in this direction.

RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Jay K]

Er, As  I understand, lack of a process boundary automatically implies GPL "spread" through "linkage".

  Assuming "linkage" means "ld". I'm not sure I've seen "linkage" defined. However

  if "linkage" or "derivation" includes "interaction via file or network I/O", then a lot of folks will be upset,
  (and some people very pleased :) ) File and network I/O connect all code in the world.

A process boundary at least gives you a chance.


Some of the work -- the frontend -- is clearly derived, and linked, so it is GPL.



> a derived work! You need to consult a knowledgable attorney before

> proceeding in this direction.



Most of the proceeding in this direction was done >10 years ago by others.
  Granted, I don't know what legal advise they had.

I'm proceeding not much further, e.g. merging to current gcc, making debugging better.

At some point I might generate C to fix a number of problems (including this assert and
licensing, and debugging, and efficient exception handling, etc.), but that is a different matter.



Anyway, I put this out there to give folks a chance to not "like" me and not help me.

I'll address the technical part separately.


Thanks,

 - Jay

----------------------------------------
> Date: Fri, 10 Sep 2010 11:17:59 -0400
> From: dewar@adacore.com
> To: iant@google.com
> CC: jay.krell@cornell.edu; gcc@gcc.gnu.org
> Subject: Re: internal compiler error: in referenced_var_lookup, at tree-dfa.c
>
> On 9/10/2010 11:08 AM, Ian Lance Taylor wrote:
> > Jay K writes:
> >
> >> That uses process boundaries to avoid GPL crossing into BSDish licensed code.
> >> So maybe you don't want to help me. Understood.
> >
> > Note that different people have different opinions as to whether a
> > process boundary means that your code is not a derived work. Not that
> > we should get into that discussion on this mailing list.
>
> Indeed, it is important to realize that putting in an arbitrary
> process boundary is not guarantee at all that you have not created
> a derived work! You need to consult a knowledgable attorney before
> proceeding in this direction.
 		 	   		  

RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Jay K]

[licensing dealt with separately]



> > Variable: D.1093058884, UID D.1093058884, int_32gimple_default_def 0x412130a8 1093058884
> This is clearly wrong, though I have no idea what caused it.
> > Is it valid for uids to be so high?
> No.

Thanks, that helps.


> From your description, you've implemented some sort of customized tree
> reader.


Not exactly, by my understanding of terminology.
We do end up making gcc trees but we serialize something that is separately
specified and not really the same, though some level of resemblance is
unavoidable since they both are involved with compilation, i.e. they both
have operations like "add" and notions of locals, parameters, functions, etc.
Ours is stack-based though for example (not my preference, but it was already there).


 > Does it play nicely with the garbage collector?


I think so.
We have the GTY annotations, I've managed to crash things when I got them
wrong/missing. I haven't moved all targets from 4.3.x to 4.5.x so I even have
to hack on the code a bit because GTY has to be in a different place.
I put the type declarations in seperate .h files, maintain both, and copy one
over the other before compilation.


We do have an open bug report about causing the gcc garbage collector
consuming infinite memory, maybe due to a cycle in our data.
But really the system works a ton. I can compile and run tens of thousands of lines
of code, for multiple architectures. I "just" have to turn off inlining, and 
a small number of other optimizations. Clearly we are pretty good, and flawed.


Notice that the gcc middle end seemed to have created this variable with the high uid.
I checked the globals that guide uid assignment, found them after sending the
first mail. They aren't so high.
I haven't yet found where this uid comes from.
I kind of suspect it might be a type mismatch, overwriting part of a tree node
  with the wrong type or such.
I'll have to dig more.

I know it comes from here:
copy_phis_for_bb:
...
      SSA_NAME_DEF_STMT (new_res)
        = new_phi = create_phi_node (new_res, new_bb);



Thanks,
 - Jay
 		 	   		  

RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Jay K]

> I kind of suspect it might be a type mismatch, overwriting part of a tree node

configure -enable-checking:

?"Bmr?: In function 'FPrint__xCombine':
`?"Bmr?:13:0: internal compiler error: tree check: expected ssa_name, have var_decl in copy_phis_for_bb, at tree-inline.c:1950

and some other problems..I really need to fix those...

 - Jay

----------------------------------------
> From: jay.krell@cornell.edu
> To: iant@google.com
> CC: gcc@gcc.gnu.org
> Subject: RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c
> Date: Fri, 10 Sep 2010 20:38:58 +0000
>
>
> [licensing dealt with separately]
>
>
>
> > > Variable: D.1093058884, UID D.1093058884, int_32gimple_default_def 0x412130a8 1093058884
> > This is clearly wrong, though I have no idea what caused it.
> > > Is it valid for uids to be so high?
> > No.
>
> Thanks, that helps.
>
>
> > From your description, you've implemented some sort of customized tree
> > reader.
>
>
> Not exactly, by my understanding of terminology.
> We do end up making gcc trees but we serialize something that is separately
> specified and not really the same, though some level of resemblance is
> unavoidable since they both are involved with compilation, i.e. they both
> have operations like "add" and notions of locals, parameters, functions, etc.
> Ours is stack-based though for example (not my preference, but it was already there).
>
>
>  > Does it play nicely with the garbage collector?
>
>
> I think so.
> We have the GTY annotations, I've managed to crash things when I got them
> wrong/missing. I haven't moved all targets from 4.3.x to 4.5.x so I even have
> to hack on the code a bit because GTY has to be in a different place.
> I put the type declarations in seperate .h files, maintain both, and copy one
> over the other before compilation.
>
>
> We do have an open bug report about causing the gcc garbage collector
> consuming infinite memory, maybe due to a cycle in our data.
> But really the system works a ton. I can compile and run tens of thousands of lines
> of code, for multiple architectures. I "just" have to turn off inlining, and
> a small number of other optimizations. Clearly we are pretty good, and flawed.
>
>
> Notice that the gcc middle end seemed to have created this variable with the high uid.
> I checked the globals that guide uid assignment, found them after sending the
> first mail. They aren't so high.
> I haven't yet found where this uid comes from.
> I kind of suspect it might be a type mismatch, overwriting part of a tree node
>   with the wrong type or such.
> I'll have to dig more.
>
> I know it comes from here:
> copy_phis_for_bb:
> ...
>       SSA_NAME_DEF_STMT (new_res)
>         = new_phi = create_phi_node (new_res, new_bb);
>
>
>
> Thanks,
>  - Jay
>
 		 	   		  

RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Jay K]

arg..well, I had replaced xmalloc with alloca, leading to some of the garbage below, but
I am indeed still running afoul of the garbage collector.
I don't know if that is my original problem but I should probably fix this first.
 ie: now that I'm using -enable-checking, and I think it collects earlier/more often.


I need to "map" my internal unsigned long arbitrary integers, to trees.
So I just have an array of struct {unsigned id; tree t };
I put GTY on the struct, on the field, and on the VEC of them.
When I append I mark it dirty.
When I search I qsort if dirty, then bsearch.


typedef struct GTY(()) m3type_t
{
  unsigned long id;
  tree GTY (()) t;
} m3type_t;


static GTY (()) VEC (m3type_t,gc)* m3type_table;

seems reasonable eh?
The files are in gtfiles. When I put the GTY in the wrong place I get compile errors.

I guess I can try rolling my own "VEC" or even use a fixed size and see what happens..

Thanks,
 - Jay

----------------------------------------
> From: jay.krell@cornell.edu
> To: iant@google.com
> CC: gcc@gcc.gnu.org
> Subject: RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c
> Date: Sat, 11 Sep 2010 08:48:08 +0000
>
>
> > I kind of suspect it might be a type mismatch, overwriting part of a tree node
>
> configure -enable-checking:
>
> ?"Bmr?: In function 'FPrint__xCombine':
> `?"Bmr?:13:0: internal compiler error: tree check: expected ssa_name, have var_decl in copy_phis_for_bb, at tree-inline.c:1950
>
> and some other problems..I really need to fix those...
>
>  - Jay
>
> ----------------------------------------
> > From: jay.krell@cornell.edu
> > To: iant@google.com
> > CC: gcc@gcc.gnu.org
> > Subject: RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c
> > Date: Fri, 10 Sep 2010 20:38:58 +0000
> >
> >
> > [licensing dealt with separately]
> >
> >
> >
> > > > Variable: D.1093058884, UID D.1093058884, int_32gimple_default_def 0x412130a8 1093058884
> > > This is clearly wrong, though I have no idea what caused it.
> > > > Is it valid for uids to be so high?
> > > No.
> >
> > Thanks, that helps.
> >
> >
> > > From your description, you've implemented some sort of customized tree
> > > reader.
> >
> >
> > Not exactly, by my understanding of terminology.
> > We do end up making gcc trees but we serialize something that is separately
> > specified and not really the same, though some level of resemblance is
> > unavoidable since they both are involved with compilation, i.e. they both
> > have operations like "add" and notions of locals, parameters, functions, etc.
> > Ours is stack-based though for example (not my preference, but it was already there).
> >
> >
> > > Does it play nicely with the garbage collector?
> >
> >
> > I think so.
> > We have the GTY annotations, I've managed to crash things when I got them
> > wrong/missing. I haven't moved all targets from 4.3.x to 4.5.x so I even have
> > to hack on the code a bit because GTY has to be in a different place.
> > I put the type declarations in seperate .h files, maintain both, and copy one
> > over the other before compilation.
> >
> >
> > We do have an open bug report about causing the gcc garbage collector
> > consuming infinite memory, maybe due to a cycle in our data.
> > But really the system works a ton. I can compile and run tens of thousands of lines
> > of code, for multiple architectures. I "just" have to turn off inlining, and
> > a small number of other optimizations. Clearly we are pretty good, and flawed.
> >
> >
> > Notice that the gcc middle end seemed to have created this variable with the high uid.
> > I checked the globals that guide uid assignment, found them after sending the
> > first mail. They aren't so high.
> > I haven't yet found where this uid comes from.
> > I kind of suspect it might be a type mismatch, overwriting part of a tree node
> > with the wrong type or such.
> > I'll have to dig more.
> >
> > I know it comes from here:
> > copy_phis_for_bb:
> > ...
> > SSA_NAME_DEF_STMT (new_res)
> > = new_phi = create_phi_node (new_res, new_bb);
> >
> >
> >
> > Thanks,
> > - Jay
> >
>
 		 	   		  

RE: internal compiler error: in referenced_var_lookup, at tree-dfa.c

[Jay K]

I have it seemingly working now, much better, thanks for the nudges -- that indeed high id is invalid, and to look again at my GTY use.
I don't know if it made the difference but I changed some whitespace to match others, and
typedef struct foo_t { ... } foo_t;
to
typedef struct foo { ... } foo_t; without the _t on the struct tag.

 - Jay

----------------------------------------
...snip...