Re: VTA guality assessment: better than -O0 ;-)

[Richard Guenther <richard.guenther@gmail.com>]

On Sat, Jun 13, 2009 at 9:29 AM, Alexandre Oliva<aoliva@redhat.com> wrote:
> So, after I tested and installed this patch
> http://gcc.gnu.org/ml/gcc-patches/2009-06/msg00903.html I started
> looking closely at the guality (debug info quality) test results.
>
> So far, I have only added two very simple tests to the guality
> testsuite, but they already show very promising results.
>
> GUALCHKXPRVAL("expr", value, maybe_dead_p) checks whether expr,
> evaluated by the debugger, matches value, evaluated at run time.
> maybe_dead_p indicates whether, if the debugger fails to compute expr
> (say, optimized away or missing debug info), we get an UNRESOLVED or a
> FAIL.  If expr is evaluated successfully but it doesn't match the
> expected value, we get a FAIL, otherwise a PASS.
>
> GUALCHKXPR(expr) is the same as GUALCHKXPRVAL("expr",(expr),1)
>
> GUALCHKFLA(expr) is nearly equivalent to GUALCHKXPRVAL("expr",(expr),0),
> except that (expr) is saved in a temporary and stored in a volatile
> memory location after the check, so expr *must* be live at the point of
> check.  FLA stands for Forced Live After.
>
> Ok, on to some test results (on x86_64-linux-gnu):
>
> guality.c:
>
> int
> main (int argc, char *argv[])
> {
>  int i = argc+1;
>  int j = argc-2;
>  int k = 5;
>
>  GUALCHKXPR (argc);
>  GUALCHKXPR (i);
>  GUALCHKXPR (j);
>  GUALCHKXPR (k);
>  GUALCHKXPR (&i);
>  GUALCHKFLA (argc);
>  GUALCHKFLA (i);
>  GUALCHKFLA (j);
>  GUALCHKXPR (i);
>  GUALCHKXPR (j);
>  GUALCHKXPRVAL ("k", 5, 1);
>  GUALCHKXPRVAL ("0x40", 64, 0);
> }
>
> -O0 (implied -fno-var-tracking-assignments)
>
> PASS: argc is 1
> PASS: i is 2
> FAIL: j is 32767, not -1
> FAIL: k is 2028276576, not 5
> PASS: &i is 140735221664248
> PASS: argc is 1
> PASS: i is 2
> FAIL: j is 32767, not -1
> PASS: i is 2
> FAIL: j is 32767, not -1
> FAIL: k is 2028276576, not 5
> PASS: 0x40 is 64
> FAIL: 7 PASS, 5 FAIL, 0 UNRESOLVED
>
> -O1 -fno-var-tracking-assignments
>
> PASS: argc is 1
> FAIL: i is 0, not 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> UNRESOLVED: &i is not computable, expected 140733781777644
> PASS: argc is 1
> PASS: i is 2
> PASS: j is -1
> FAIL: i is 0, not 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> PASS: 0x40 is 64
> FAIL: 7 PASS, 2 FAIL, 3 UNRESOLVED
>
> We see that debug info got better for j, and k is no longer wrong: it is
> completely dropped from debug information, even though it could have
> been encoded in standard debug info, for its value is the same constant
> throughout its entire lifetime.  For some reason, although i is
> addressable, it's not uniformly computable: taking its address doesn't
> work in between two sucessful evaluations of i, and &i actually works in
> the debugger at those surrounding execution points.
>
> -O1 -fvar-tracking-assignments
>
> PASS: argc is 1
> UNRESOLVED: i is optimized away, expected 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> UNRESOLVED: &i is not computable, expected 140735863643740
> PASS: argc is 1
> PASS: i is 2
> PASS: j is -1
> UNRESOLVED: i is optimized away, expected 2
> PASS: j is -1
> UNRESOLVED: k is not computable, expected 5
> PASS: 0x40 is 64
> PASS: 7 PASS, 0 FAIL, 5 UNRESOLVED
>
> Yay, PASS!  With Jakub's patch to encode constants in location lists,
> the two UNRESOLVED tests for k will become PASS.  I haven't looked into
> why i is taken as optimized away.  What I do know is that i isn't
> tracked by VTA, for VTA only tracks variables that aren't addressable.
>
> Not convinced yet?  Why, sure, VTA (+ DW_OP_implicit_value) has *only*
> gone from totally broken j and k at -O0 to totally correct debug info,
> while somehow *fixing* additional errors for i that is not even tracked
> by VTA.  From 0% to 100% correctness, and 100% completeness for all
> VTA-tracked variables.
>
> But it gets better.
>
> Remember those examples from the slides in the VTA presentation in last
> year's GCC Summit?  http://people.redhat.com/~aoliva/papers/vta/
>
> I've turned them into another guality test, that runs with GUALCHK(expr)
> defined GUALCHKXPRVAL("expr", expr, 0), i.e., it requires variables to
> be available and computable.
>
> typedef struct list {
>  struct list *n;
>  int v;
> } elt, *node;
>
> node
> find_val (node c, int v, node e)
> {
>  while (c < e)
>    {
>      GUALCHK (c);
>      GUALCHK (v);
>      GUALCHK (e);
>      if (c->v == v)
>        return c;
>      GUALCHK (c);
>      GUALCHK (v);
>      GUALCHK (e);
>      c++;
>    }
>  return NULL;
> }
>
> node
> find_prev (node c, node w)
> {
>  while (c)
>    {
>      node o = c;
>      c = c->n;
>      GUALCHK (c);
>      GUALCHK (o);
>      GUALCHK (w);
>      if (c == w)
>        return o;
>      GUALCHK (c);
>      GUALCHK (o);
>      GUALCHK (w);
>    }
>  return NULL;
> }
>
> node
> check_arr (node c, node e)
> {
>  if (c == e)
>    return NULL;
>  e--;
>  while (c < e)
>    {
>      GUALCHK (c);
>      GUALCHK (e);
>      if (c->v > (c+1)->v)
>        return c;
>      GUALCHK (c);
>      GUALCHK (e);
>      c++;
>    }
>  return NULL;
> }
>
> node
> check_list (node c, node t)
> {
>  while (c != t)
>    {
>      node n = c->n;
>      GUALCHK (c);
>      GUALCHK (n);
>      GUALCHK (t);
>      if (c->v > n->v)
>        return c;
>      GUALCHK (c);
>      GUALCHK (n);
>      GUALCHK (t);
>      c = n;
>    }
>  return NULL;
> }
>
> struct list testme[] = {
>  { &testme[1],  2 },
>  { &testme[2],  3 },
>  { &testme[3],  5 },
>  { &testme[4],  7 },
>  { &testme[5], 11 },
>  { NULL, 13 },
> };
>
> int
> main (int argc, char *argv[])
> {
>  int n = sizeof (testme) / sizeof (*testme);
>  node first, last, begin, end, ret;
>
>  GUALCHKXPR (n);
>
>  begin = first = &testme[0];
>  last = &testme[n-1];
>  end = &testme[n];
>
>  GUALCHKXPR (first);
>  GUALCHKXPR (last);
>  GUALCHKXPR (begin);
>  GUALCHKXPR (end);
>
>  ret = find_val (begin, 13, end);
>  GUALCHK (ret);
>  assert (ret == last);
>
>  ret = find_prev (first, last);
>  GUALCHK (ret);
>  assert (ret == &testme[n-2]);
>
>  ret = check_arr (begin, end);
>  GUALCHK (ret);
>  assert (!ret);
>
>  ret = check_list (first, last);
>  GUALCHK (ret);
>  assert (!ret);
> }
>
>
> -O0 (-fno-var-tracking-assignments implied)
>
> FAIL: n is 0, not 6
> FAIL: first is 16187408, not 6297888
> FAIL: last is 249832090421, not 6297968
> FAIL: begin is 140735655442656, not 6297888
> FAIL: end is 139872232800256, not 6297984
> PASS: c is 6297888
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297888
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297904
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297904
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297920
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297920
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297936
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297936
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297952
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297952
> PASS: v is 13
> PASS: e is 6297984
> PASS: c is 6297968
> PASS: v is 13
> PASS: e is 6297984
> FAIL: ret is 66, not 6297968
> PASS: c is 6297904
> FAIL: o is 55840872688, not 6297888
> PASS: w is 6297968
> PASS: c is 6297904
> FAIL: o is 55840872688, not 6297888
> PASS: w is 6297968
> PASS: c is 6297920
> FAIL: o is 55840872688, not 6297904
> PASS: w is 6297968
> PASS: c is 6297920
> FAIL: o is 55840872688, not 6297904
> PASS: w is 6297968
> PASS: c is 6297936
> FAIL: o is 55840872688, not 6297920
> PASS: w is 6297968
> PASS: c is 6297936
> FAIL: o is 55840872688, not 6297920
> PASS: w is 6297968
> PASS: c is 6297952
> FAIL: o is 55840872688, not 6297936
> PASS: w is 6297968
> PASS: c is 6297952
> FAIL: o is 55840872688, not 6297936
> PASS: w is 6297968
> PASS: c is 6297968
> FAIL: o is 55840872688, not 6297952
> PASS: w is 6297968
> FAIL: ret is 66, not 6297952
> PASS: c is 6297888
> PASS: e is 6297968
> PASS: c is 6297888
> PASS: e is 6297968
> PASS: c is 6297904
> PASS: e is 6297968
> PASS: c is 6297904
> PASS: e is 6297968
> PASS: c is 6297920
> PASS: e is 6297968
> PASS: c is 6297920
> PASS: e is 6297968
> PASS: c is 6297936
> PASS: e is 6297968
> PASS: c is 6297936
> PASS: e is 6297968
> PASS: c is 6297952
> PASS: e is 6297968
> PASS: c is 6297952
> PASS: e is 6297968
> FAIL: ret is 66, not 0
> PASS: c is 6297888
> FAIL: n is 6297968, not 6297904
> PASS: t is 6297968
> PASS: c is 6297888
> FAIL: n is 6297968, not 6297904
> PASS: t is 6297968
> PASS: c is 6297904
> FAIL: n is 6297968, not 6297920
> PASS: t is 6297968
> PASS: c is 6297904
> FAIL: n is 6297968, not 6297920
> PASS: t is 6297968
> PASS: c is 6297920
> FAIL: n is 6297968, not 6297936
> PASS: t is 6297968
> PASS: c is 6297920
> FAIL: n is 6297968, not 6297936
> PASS: t is 6297968
> PASS: c is 6297936
> FAIL: n is 6297968, not 6297952
> PASS: t is 6297968
> PASS: c is 6297936
> FAIL: n is 6297968, not 6297952
> PASS: t is 6297968
> PASS: c is 6297952
> PASS: n is 6297968
> PASS: t is 6297968
> PASS: c is 6297952
> PASS: n is 6297968
> PASS: t is 6297968
> FAIL: ret is 66, not 0
> FAIL: 93 PASS, 26 FAIL, 0 UNRESOLVED
>
> -O2 -fno-var-tracking-assignments:
>
> UNRESOLVED: n is not computable, expected 6
> UNRESOLVED: first is not computable, expected 6297408
> UNRESOLVED: last is not computable, expected 6297488
> UNRESOLVED: begin is not computable, expected 6297408
> UNRESOLVED: end is not computable, expected 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297488
> PASS: v is 13
> PASS: e is 6297504
> FAIL: ret is not computable, expected 6297488
> PASS: c is 6297424
> FAIL: o is optimized away, expected 6297408
> PASS: w is 6297488
> PASS: c is 6297424
> FAIL: o is optimized away, expected 6297408
> PASS: w is 6297488
> PASS: c is 6297440
> FAIL: o is optimized away, expected 6297424
> PASS: w is 6297488
> PASS: c is 6297440
> FAIL: o is optimized away, expected 6297424
> PASS: w is 6297488
> PASS: c is 6297456
> FAIL: o is optimized away, expected 6297440
> PASS: w is 6297488
> PASS: c is 6297456
> FAIL: o is optimized away, expected 6297440
> PASS: w is 6297488
> PASS: c is 6297472
> FAIL: o is optimized away, expected 6297456
> PASS: w is 6297488
> PASS: c is 6297472
> FAIL: o is optimized away, expected 6297456
> PASS: w is 6297488
> PASS: c is 6297488
> FAIL: o is optimized away, expected 6297472
> PASS: w is 6297488
> FAIL: ret is not computable, expected 6297472
> FAIL: c is 6297424, not 6297408
> PASS: e is 6297488
> FAIL: c is 6297424, not 6297408
> PASS: e is 6297488
> FAIL: c is 6297440, not 6297424
> PASS: e is 6297488
> FAIL: c is 6297440, not 6297424
> PASS: e is 6297488
> FAIL: c is 6297456, not 6297440
> PASS: e is 6297488
> FAIL: c is 6297456, not 6297440
> PASS: e is 6297488
> FAIL: c is 6297472, not 6297456
> PASS: e is 6297488
> FAIL: c is 6297472, not 6297456
> PASS: e is 6297488
> FAIL: c is 6297488, not 6297472
> PASS: e is 6297488
> FAIL: c is 6297488, not 6297472
> PASS: e is 6297488
> FAIL: ret is not computable, expected 0
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> FAIL: ret is not computable, expected 0
> FAIL: 91 PASS, 23 FAIL, 5 UNRESOLVED
>
> Not much worse than -O0...  Not only because some variables got
> optimized away, but also because they got optimized in ways that I
> “predicted” in the Summit, causing debug info to map to incorrect
> locations.
>
> -O2 -fvar-tracking-assignments:
>
> UNRESOLVED: n is not computable, expected 6
> UNRESOLVED: first is not computable, expected 6297408
> UNRESOLVED: last is not computable, expected 6297488
> UNRESOLVED: begin is not computable, expected 6297408
> UNRESOLVED: end is not computable, expected 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297408
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297424
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297440
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297456
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297472
> PASS: v is 13
> PASS: e is 6297504
> PASS: c is 6297488
> PASS: v is 13
> PASS: e is 6297504
> FAIL: ret is 1, not 6297488
> PASS: c is 6297424
> PASS: o is 6297408
> PASS: w is 6297488
> PASS: c is 6297424
> PASS: o is 6297408
> PASS: w is 6297488
> PASS: c is 6297440
> PASS: o is 6297424
> PASS: w is 6297488
> PASS: c is 6297440
> PASS: o is 6297424
> PASS: w is 6297488
> PASS: c is 6297456
> PASS: o is 6297440
> PASS: w is 6297488
> PASS: c is 6297456
> PASS: o is 6297440
> PASS: w is 6297488
> PASS: c is 6297472
> PASS: o is 6297456
> PASS: w is 6297488
> PASS: c is 6297472
> PASS: o is 6297456
> PASS: w is 6297488
> PASS: c is 6297488
> PASS: o is 6297472
> PASS: w is 6297488
> FAIL: ret is 1, not 6297472
> PASS: c is 6297408
> PASS: e is 6297488
> PASS: c is 6297408
> PASS: e is 6297488
> PASS: c is 6297424
> PASS: e is 6297488
> PASS: c is 6297424
> PASS: e is 6297488
> PASS: c is 6297440
> PASS: e is 6297488
> PASS: c is 6297440
> PASS: e is 6297488
> PASS: c is 6297456
> PASS: e is 6297488
> PASS: c is 6297456
> PASS: e is 6297488
> PASS: c is 6297472
> PASS: e is 6297488
> PASS: c is 6297472
> PASS: e is 6297488
> FAIL: ret is 1, not 0
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297408
> PASS: n is 6297424
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297424
> PASS: n is 6297440
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297440
> PASS: n is 6297456
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297456
> PASS: n is 6297472
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> PASS: c is 6297472
> PASS: n is 6297488
> PASS: t is 6297488
> FAIL: ret is 1, not 0
> FAIL: 110 PASS, 4 FAIL, 5 UNRESOLVED
>
> Pretty good, eh?  And the 4 failures are easy to fix: ret is noted in
> debug info as being held in a call-clobbered register at the time of the
> call to the function that implements GUALCHK.  It was saved in another
> call-saved register, but the annotation that says so, right after the
> call, only takes effect after the call.
>
> Although this could be fixed by changing the logic that emits debug info
> before calls, so as to note the change in location earlier, this would
> be wrong on two accounts: if you're at the pc of the call, you want to
> be able to inspect the variable, and you want all locations in which
> it's live, so that they can all be modified.
>
> So the proper fix would be to emit annotations right after the call that
> take effect after we perform the call, but before it returns, i.e.,
> annotations at next_pc-1, rather than at next_pc.  This would fix the
> only 4 errors above.
>
> As for the UNRESOLVED bits, n would be fixed with DW_OP_implicit_value,
> but the other four would require DW_OP_stack_value and some additional
> fix in cfgexpand.c, to expand to (plus (symbol_ref) (const_int)) the
> debug stmts that bind first, last, begin and end to &testme[<index>] at
> the tree optimized dump.  Then it would be another 100% pass rate,
> without any UNRESOLVED whatsoever (all variables are tracked by VTA in
> this testcase).
>
> Could it get any better?

Yes, I don't like -O0 producing worse debug info - what does
the -O0 -fvar-tracking-assignments results look like?

Richard.

> --
> Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/   FSF Latin America board member
> Free Software Evangelist      Red Hat Brazil Compiler Engineer
>