VTA guality assessment: better than -O0 ;-)

[Alexandre Oliva <aoliva@redhat.com>]

So, after I tested and installed this patch
http://gcc.gnu.org/ml/gcc-patches/2009-06/msg00903.html I started
looking closely at the guality (debug info quality) test results.

So far, I have only added two very simple tests to the guality
testsuite, but they already show very promising results.

GUALCHKXPRVAL("expr", value, maybe_dead_p) checks whether expr,
evaluated by the debugger, matches value, evaluated at run time.
maybe_dead_p indicates whether, if the debugger fails to compute expr
(say, optimized away or missing debug info), we get an UNRESOLVED or a
FAIL.  If expr is evaluated successfully but it doesn't match the
expected value, we get a FAIL, otherwise a PASS.

GUALCHKXPR(expr) is the same as GUALCHKXPRVAL("expr",(expr),1)

GUALCHKFLA(expr) is nearly equivalent to GUALCHKXPRVAL("expr",(expr),0),
except that (expr) is saved in a temporary and stored in a volatile
memory location after the check, so expr *must* be live at the point of
check.  FLA stands for Forced Live After.

Ok, on to some test results (on x86_64-linux-gnu):

guality.c:

int
main (int argc, char *argv[])
{
  int i = argc+1;
  int j = argc-2;
  int k = 5;

  GUALCHKXPR (argc);
  GUALCHKXPR (i);
  GUALCHKXPR (j);
  GUALCHKXPR (k);
  GUALCHKXPR (&i);
  GUALCHKFLA (argc);
  GUALCHKFLA (i);
  GUALCHKFLA (j);
  GUALCHKXPR (i);
  GUALCHKXPR (j);
  GUALCHKXPRVAL ("k", 5, 1);
  GUALCHKXPRVAL ("0x40", 64, 0);
}

-O0 (implied -fno-var-tracking-assignments)

PASS: argc is 1
PASS: i is 2
FAIL: j is 32767, not -1
FAIL: k is 2028276576, not 5
PASS: &i is 140735221664248
PASS: argc is 1
PASS: i is 2
FAIL: j is 32767, not -1
PASS: i is 2
FAIL: j is 32767, not -1
FAIL: k is 2028276576, not 5
PASS: 0x40 is 64
FAIL: 7 PASS, 5 FAIL, 0 UNRESOLVED

-O1 -fno-var-tracking-assignments

PASS: argc is 1
FAIL: i is 0, not 2
PASS: j is -1
UNRESOLVED: k is not computable, expected 5
UNRESOLVED: &i is not computable, expected 140733781777644
PASS: argc is 1
PASS: i is 2
PASS: j is -1
FAIL: i is 0, not 2
PASS: j is -1
UNRESOLVED: k is not computable, expected 5
PASS: 0x40 is 64
FAIL: 7 PASS, 2 FAIL, 3 UNRESOLVED

We see that debug info got better for j, and k is no longer wrong: it is
completely dropped from debug information, even though it could have
been encoded in standard debug info, for its value is the same constant
throughout its entire lifetime.  For some reason, although i is
addressable, it's not uniformly computable: taking its address doesn't
work in between two sucessful evaluations of i, and &i actually works in
the debugger at those surrounding execution points.

-O1 -fvar-tracking-assignments

PASS: argc is 1
UNRESOLVED: i is optimized away, expected 2
PASS: j is -1
UNRESOLVED: k is not computable, expected 5
UNRESOLVED: &i is not computable, expected 140735863643740
PASS: argc is 1
PASS: i is 2
PASS: j is -1
UNRESOLVED: i is optimized away, expected 2
PASS: j is -1
UNRESOLVED: k is not computable, expected 5
PASS: 0x40 is 64
PASS: 7 PASS, 0 FAIL, 5 UNRESOLVED

Yay, PASS!  With Jakub's patch to encode constants in location lists,
the two UNRESOLVED tests for k will become PASS.  I haven't looked into
why i is taken as optimized away.  What I do know is that i isn't
tracked by VTA, for VTA only tracks variables that aren't addressable.

Not convinced yet?  Why, sure, VTA (+ DW_OP_implicit_value) has *only*
gone from totally broken j and k at -O0 to totally correct debug info,
while somehow *fixing* additional errors for i that is not even tracked
by VTA.  From 0% to 100% correctness, and 100% completeness for all
VTA-tracked variables.

But it gets better.

Remember those examples from the slides in the VTA presentation in last
year's GCC Summit?  http://people.redhat.com/~aoliva/papers/vta/

I've turned them into another guality test, that runs with GUALCHK(expr)
defined GUALCHKXPRVAL("expr", expr, 0), i.e., it requires variables to
be available and computable.

typedef struct list {
  struct list *n;
  int v;
} elt, *node;

node
find_val (node c, int v, node e)
{
  while (c < e)
    {
      GUALCHK (c);
      GUALCHK (v);
      GUALCHK (e);
      if (c->v == v)
	return c;
      GUALCHK (c);
      GUALCHK (v);
      GUALCHK (e);
      c++;
    }
  return NULL;
}

node
find_prev (node c, node w)
{
  while (c)
    {
      node o = c;
      c = c->n;
      GUALCHK (c);
      GUALCHK (o);
      GUALCHK (w);
      if (c == w)
	return o;
      GUALCHK (c);
      GUALCHK (o);
      GUALCHK (w);
    }
  return NULL;
}

node
check_arr (node c, node e)
{
  if (c == e)
    return NULL;
  e--;
  while (c < e)
    {
      GUALCHK (c);
      GUALCHK (e);
      if (c->v > (c+1)->v)
	return c;
      GUALCHK (c);
      GUALCHK (e);
      c++;
    }
  return NULL;
}

node
check_list (node c, node t)
{
  while (c != t)
    {
      node n = c->n;
      GUALCHK (c);
      GUALCHK (n);
      GUALCHK (t);
      if (c->v > n->v)
	return c;
      GUALCHK (c);
      GUALCHK (n);
      GUALCHK (t);
      c = n;
    }
  return NULL;
}

struct list testme[] = {
  { &testme[1],  2 },
  { &testme[2],  3 },
  { &testme[3],  5 },
  { &testme[4],  7 },
  { &testme[5], 11 },
  { NULL, 13 },
};

int
main (int argc, char *argv[])
{
  int n = sizeof (testme) / sizeof (*testme);
  node first, last, begin, end, ret;

  GUALCHKXPR (n);

  begin = first = &testme[0];
  last = &testme[n-1];
  end = &testme[n];

  GUALCHKXPR (first);
  GUALCHKXPR (last);
  GUALCHKXPR (begin);
  GUALCHKXPR (end);

  ret = find_val (begin, 13, end);
  GUALCHK (ret);
  assert (ret == last);

  ret = find_prev (first, last);
  GUALCHK (ret);
  assert (ret == &testme[n-2]);

  ret = check_arr (begin, end);
  GUALCHK (ret);
  assert (!ret);

  ret = check_list (first, last);
  GUALCHK (ret);
  assert (!ret);
}


-O0 (-fno-var-tracking-assignments implied)

FAIL: n is 0, not 6
FAIL: first is 16187408, not 6297888
FAIL: last is 249832090421, not 6297968
FAIL: begin is 140735655442656, not 6297888
FAIL: end is 139872232800256, not 6297984
PASS: c is 6297888
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297888
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297904
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297904
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297920
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297920
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297936
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297936
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297952
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297952
PASS: v is 13
PASS: e is 6297984
PASS: c is 6297968
PASS: v is 13
PASS: e is 6297984
FAIL: ret is 66, not 6297968
PASS: c is 6297904
FAIL: o is 55840872688, not 6297888
PASS: w is 6297968
PASS: c is 6297904
FAIL: o is 55840872688, not 6297888
PASS: w is 6297968
PASS: c is 6297920
FAIL: o is 55840872688, not 6297904
PASS: w is 6297968
PASS: c is 6297920
FAIL: o is 55840872688, not 6297904
PASS: w is 6297968
PASS: c is 6297936
FAIL: o is 55840872688, not 6297920
PASS: w is 6297968
PASS: c is 6297936
FAIL: o is 55840872688, not 6297920
PASS: w is 6297968
PASS: c is 6297952
FAIL: o is 55840872688, not 6297936
PASS: w is 6297968
PASS: c is 6297952
FAIL: o is 55840872688, not 6297936
PASS: w is 6297968
PASS: c is 6297968
FAIL: o is 55840872688, not 6297952
PASS: w is 6297968
FAIL: ret is 66, not 6297952
PASS: c is 6297888
PASS: e is 6297968
PASS: c is 6297888
PASS: e is 6297968
PASS: c is 6297904
PASS: e is 6297968
PASS: c is 6297904
PASS: e is 6297968
PASS: c is 6297920
PASS: e is 6297968
PASS: c is 6297920
PASS: e is 6297968
PASS: c is 6297936
PASS: e is 6297968
PASS: c is 6297936
PASS: e is 6297968
PASS: c is 6297952
PASS: e is 6297968
PASS: c is 6297952
PASS: e is 6297968
FAIL: ret is 66, not 0
PASS: c is 6297888
FAIL: n is 6297968, not 6297904
PASS: t is 6297968
PASS: c is 6297888
FAIL: n is 6297968, not 6297904
PASS: t is 6297968
PASS: c is 6297904
FAIL: n is 6297968, not 6297920
PASS: t is 6297968
PASS: c is 6297904
FAIL: n is 6297968, not 6297920
PASS: t is 6297968
PASS: c is 6297920
FAIL: n is 6297968, not 6297936
PASS: t is 6297968
PASS: c is 6297920
FAIL: n is 6297968, not 6297936
PASS: t is 6297968
PASS: c is 6297936
FAIL: n is 6297968, not 6297952
PASS: t is 6297968
PASS: c is 6297936
FAIL: n is 6297968, not 6297952
PASS: t is 6297968
PASS: c is 6297952
PASS: n is 6297968
PASS: t is 6297968
PASS: c is 6297952
PASS: n is 6297968
PASS: t is 6297968
FAIL: ret is 66, not 0
FAIL: 93 PASS, 26 FAIL, 0 UNRESOLVED

-O2 -fno-var-tracking-assignments:

UNRESOLVED: n is not computable, expected 6
UNRESOLVED: first is not computable, expected 6297408
UNRESOLVED: last is not computable, expected 6297488
UNRESOLVED: begin is not computable, expected 6297408
UNRESOLVED: end is not computable, expected 6297504
PASS: c is 6297408
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297408
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297424
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297424
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297440
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297440
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297456
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297456
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297472
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297472
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297488
PASS: v is 13
PASS: e is 6297504
FAIL: ret is not computable, expected 6297488
PASS: c is 6297424
FAIL: o is optimized away, expected 6297408
PASS: w is 6297488
PASS: c is 6297424
FAIL: o is optimized away, expected 6297408
PASS: w is 6297488
PASS: c is 6297440
FAIL: o is optimized away, expected 6297424
PASS: w is 6297488
PASS: c is 6297440
FAIL: o is optimized away, expected 6297424
PASS: w is 6297488
PASS: c is 6297456
FAIL: o is optimized away, expected 6297440
PASS: w is 6297488
PASS: c is 6297456
FAIL: o is optimized away, expected 6297440
PASS: w is 6297488
PASS: c is 6297472
FAIL: o is optimized away, expected 6297456
PASS: w is 6297488
PASS: c is 6297472
FAIL: o is optimized away, expected 6297456
PASS: w is 6297488
PASS: c is 6297488
FAIL: o is optimized away, expected 6297472
PASS: w is 6297488
FAIL: ret is not computable, expected 6297472
FAIL: c is 6297424, not 6297408
PASS: e is 6297488
FAIL: c is 6297424, not 6297408
PASS: e is 6297488
FAIL: c is 6297440, not 6297424
PASS: e is 6297488
FAIL: c is 6297440, not 6297424
PASS: e is 6297488
FAIL: c is 6297456, not 6297440
PASS: e is 6297488
FAIL: c is 6297456, not 6297440
PASS: e is 6297488
FAIL: c is 6297472, not 6297456
PASS: e is 6297488
FAIL: c is 6297472, not 6297456
PASS: e is 6297488
FAIL: c is 6297488, not 6297472
PASS: e is 6297488
FAIL: c is 6297488, not 6297472
PASS: e is 6297488
FAIL: ret is not computable, expected 0
PASS: c is 6297408
PASS: n is 6297424
PASS: t is 6297488
PASS: c is 6297408
PASS: n is 6297424
PASS: t is 6297488
PASS: c is 6297424
PASS: n is 6297440
PASS: t is 6297488
PASS: c is 6297424
PASS: n is 6297440
PASS: t is 6297488
PASS: c is 6297440
PASS: n is 6297456
PASS: t is 6297488
PASS: c is 6297440
PASS: n is 6297456
PASS: t is 6297488
PASS: c is 6297456
PASS: n is 6297472
PASS: t is 6297488
PASS: c is 6297456
PASS: n is 6297472
PASS: t is 6297488
PASS: c is 6297472
PASS: n is 6297488
PASS: t is 6297488
PASS: c is 6297472
PASS: n is 6297488
PASS: t is 6297488
FAIL: ret is not computable, expected 0
FAIL: 91 PASS, 23 FAIL, 5 UNRESOLVED

Not much worse than -O0...  Not only because some variables got
optimized away, but also because they got optimized in ways that I
“predicted” in the Summit, causing debug info to map to incorrect
locations.

-O2 -fvar-tracking-assignments:

UNRESOLVED: n is not computable, expected 6
UNRESOLVED: first is not computable, expected 6297408
UNRESOLVED: last is not computable, expected 6297488
UNRESOLVED: begin is not computable, expected 6297408
UNRESOLVED: end is not computable, expected 6297504
PASS: c is 6297408
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297408
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297424
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297424
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297440
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297440
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297456
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297456
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297472
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297472
PASS: v is 13
PASS: e is 6297504
PASS: c is 6297488
PASS: v is 13
PASS: e is 6297504
FAIL: ret is 1, not 6297488
PASS: c is 6297424
PASS: o is 6297408
PASS: w is 6297488
PASS: c is 6297424
PASS: o is 6297408
PASS: w is 6297488
PASS: c is 6297440
PASS: o is 6297424
PASS: w is 6297488
PASS: c is 6297440
PASS: o is 6297424
PASS: w is 6297488
PASS: c is 6297456
PASS: o is 6297440
PASS: w is 6297488
PASS: c is 6297456
PASS: o is 6297440
PASS: w is 6297488
PASS: c is 6297472
PASS: o is 6297456
PASS: w is 6297488
PASS: c is 6297472
PASS: o is 6297456
PASS: w is 6297488
PASS: c is 6297488
PASS: o is 6297472
PASS: w is 6297488
FAIL: ret is 1, not 6297472
PASS: c is 6297408
PASS: e is 6297488
PASS: c is 6297408
PASS: e is 6297488
PASS: c is 6297424
PASS: e is 6297488
PASS: c is 6297424
PASS: e is 6297488
PASS: c is 6297440
PASS: e is 6297488
PASS: c is 6297440
PASS: e is 6297488
PASS: c is 6297456
PASS: e is 6297488
PASS: c is 6297456
PASS: e is 6297488
PASS: c is 6297472
PASS: e is 6297488
PASS: c is 6297472
PASS: e is 6297488
FAIL: ret is 1, not 0
PASS: c is 6297408
PASS: n is 6297424
PASS: t is 6297488
PASS: c is 6297408
PASS: n is 6297424
PASS: t is 6297488
PASS: c is 6297424
PASS: n is 6297440
PASS: t is 6297488
PASS: c is 6297424
PASS: n is 6297440
PASS: t is 6297488
PASS: c is 6297440
PASS: n is 6297456
PASS: t is 6297488
PASS: c is 6297440
PASS: n is 6297456
PASS: t is 6297488
PASS: c is 6297456
PASS: n is 6297472
PASS: t is 6297488
PASS: c is 6297456
PASS: n is 6297472
PASS: t is 6297488
PASS: c is 6297472
PASS: n is 6297488
PASS: t is 6297488
PASS: c is 6297472
PASS: n is 6297488
PASS: t is 6297488
FAIL: ret is 1, not 0
FAIL: 110 PASS, 4 FAIL, 5 UNRESOLVED

Pretty good, eh?  And the 4 failures are easy to fix: ret is noted in
debug info as being held in a call-clobbered register at the time of the
call to the function that implements GUALCHK.  It was saved in another
call-saved register, but the annotation that says so, right after the
call, only takes effect after the call.

Although this could be fixed by changing the logic that emits debug info
before calls, so as to note the change in location earlier, this would
be wrong on two accounts: if you're at the pc of the call, you want to
be able to inspect the variable, and you want all locations in which
it's live, so that they can all be modified.

So the proper fix would be to emit annotations right after the call that
take effect after we perform the call, but before it returns, i.e.,
annotations at next_pc-1, rather than at next_pc.  This would fix the
only 4 errors above.

As for the UNRESOLVED bits, n would be fixed with DW_OP_implicit_value,
but the other four would require DW_OP_stack_value and some additional
fix in cfgexpand.c, to expand to (plus (symbol_ref) (const_int)) the
debug stmts that bind first, last, begin and end to &testme[<index>] at
the tree optimized dump.  Then it would be another 100% pass rate,
without any UNRESOLVED whatsoever (all variables are tracked by VTA in
this testcase).

Could it get any better?

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist      Red Hat Brazil Compiler Engineer