frameworklet to assess the quality of debug information

frameworklet to assess the quality of debug information

[Alexandre Oliva]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

Here's my first cut at trying to tell how well or how bad we perform
in terms of debug info, that can be dropped into the GCC run-time test
infrastructure and used by means of #include in new tests that add
GUALCHK* annotations (or with separate compilation, if some stuff is
moved into a separate header).

Thoughts, comments, suggestions, tomatoes, eggs? :-)


[-- Attachment #2: guality.c --]
[-- Type: text/plain, Size: 10336 bytes --]

/* Infrastructure to test the quality of debug information.
   Copyright (C) 2008 Free Software Foundation, Inc.
   Contributed by Alexandre Oliva <dnovillo@redhat.com>.

This file is part of GCC.

GCC is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.

GCC is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3.  If not see
<http://www.gnu.org/licenses/>.  */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

/* This is a first cut at checking that debug information matches
   run-time.  The idea is to annotate programs with GUALCHK* macros
   that guide the tests.

   In the current implementation, all of the macros expand to function
   calls.  On the one hand, this interferes with optimizations; on the
   other hand, it establishes an optimization barrier and a clear
   inspection point, where previous operations (as in the abstract
   machine) should have been completed and have their effects visible,
   and future operations shouldn't have started yet.

   In the current implementation of guality_check(), we fork a child
   process that runs gdb, attaches to the parent process (the one that
   called guality_check), moves up one stack frame (to the caller of
   guality_check) and then examines the given expression.

   If it matches the expected value, we have a PASS.  If it differs,
   we have a FAILure.  If it is missing, we'll have a FAIL or an
   UNRESOLVED depending on whether the variable or expression might be
   unavailable at that point, as indicated by the third argument.

   We envision a future alternate implementation with two compilation
   and execution cycles, say one that runs the program and uses the
   macros to log expressions and expected values, another in which the
   macros expand to nothing and the logs are used to guide a debug
   session that tests the values.  How to identify the inspection
   points in the second case is yet to be determined.  It is
   recommended that GUALCHK* macros be by themselves in source lines,
   so that __FILE__ and __LINE__ will be usable to identify them.
*/

/* Attach a debugger to the current process and verify that the string
   EXPR, evaluated by the debugger, yields the long long number VAL.
   If the debugger cannot compute the expression, say because the
   variable is unavailable, this will count as an error, unless unkok
   is nonzero.  */

#define GUALCHKXPRVAL(expr, val, unkok) \
  guality_check ((expr), (val), (unkok))

/* Check that a debugger knows that EXPR evaluates to the run-time
   value of EXPR.  Unknown values are marked as acceptable,
   considering that EXPR may die right after this call.  This will
   affect the generated code in that EXPR will be evaluated and forced
   to remain live at least until right before the call to
   guality_check, although not necessarily after the call.  */

#define GUALCHKXPR(expr) \
  GUALCHKXPRVAL (#expr, (long long)(expr), 1)

/* Check that a debugger knows that EXPR evaluates to the run-time
   value of EXPR.  Unknown values are marked as errors, because the
   value of EXPR is forced to be available right after the call, for a
   range of at least one instruction.  This will affect the generated
   code, in that EXPR *will* be evaluated both before and after the
   call to guality_check.  */

#define GUALCHKFLA(expr) do {				\
    GUALCHKXPRVAL (#expr, (long long)(expr), 0);	\
    guality_forced_live = (long long)(expr);		\
  } while (0)

/* GUALCHK is the simplest way to assert that debug information for an
   expression matches its run-time value.  Whether to force the
   expression live after the call, so as to flag incompleteness
   errors, can be disabled by defining DONT_FORCE_LIVE_AFTER.  */

#ifndef DONT_FORCE_LIVE_AFTER
#define GUALCHK(var) GUALCHKFLA(var)
#else
#define GUALCHK(var) GUALCHKXPR(var)
#endif

/* The program that GDB is going to be told to attach to.  */

static char *guality_program_name;

/* The pid of the process it should attach to, as a string.  */

char guality_pid_str[30];

/* The name of the GDB program.  This is gdb by default, but it can be
   overridden by setting the GUALITY_GDB environment variable.  */

static const char *guality_gdb_name;

/* Kinds of results communicated as exit status from child process
   that runs gdb to the parent process that's being monitored.  */

enum guality_counter { PASS, INCORRECT, INCOMPLETE };

/* Count of passes and errors.  */

static int guality_count[INCOMPLETE+1];

/* If --guality-skip is given in the command line, all the monitoring,
   forking and debugger-attaching action will be disabled.  This is
   useful to run the monitor program within a debugger.  */

static int guality_skip;

/* This variable is holds the last value that was forced live after a
   call.  */

static long long guality_forced_live;

/* This function is the main guality program.  It may actually be
   defined as main, because we #define main to it afterwards.  Because
   of this wrapping, guality_main may not have an empty argument
   list.  */

extern int guality_main (int argc, char *argv[]);

/* Set things up, run guality_main, then print a summary and quit.  */

int
main (int argc, char *argv[])
{
  int i;

  guality_program_name = argv[0];
  sprintf (guality_pid_str, "%i", getpid ());
  guality_gdb_name = getenv ("GUALITY_GDB");
  if (!guality_gdb_name)
    guality_gdb_name = "gdb";

  for (i = 1; i < argc; i++)
    if (strcmp (argv[i], "--guality-skip") == 0)
      guality_skip = 1;
    else
      break;

  argv[--i] = guality_program_name;

  guality_main (argc - i, argv + i);

  i = guality_count[INCORRECT];

  printf ("%s: %i PASS, %i FAIL, %i UNRESOLVED\n",
	  i ? "FAIL" : "PASS",
	  guality_count[PASS], guality_count[INCORRECT],
	  guality_count[INCOMPLETE]);

  return i;
}

#define main guality_main

/* This is run in the child process, to attach gdb to the parent and
   verify that name evaluates to value or unknown.  */

static void
attach (const char *name, long long value, int unknown_ok)
{
  FILE *p;
  const char *cmd_arr[] = {
    guality_gdb_name,
    " -nx -q --batch"
    " --eval-command 'set var loop=0' --eval-command 'up'"
    " --eval-command 'p ", name,
    "' --eval-command detach --eval-command quit ",
    guality_program_name, " ", guality_pid_str
  };
  int i, len;
  long long value_in;
  int base;
  int sign;

  for (i = 0, len = 1; i < sizeof (cmd_arr) / sizeof(*cmd_arr); i++)
    len += strlen (cmd_arr[i]);

  {
    char cmd[len];

    cmd[0] = '\0';
    for (i = 0; i < sizeof (cmd_arr) / sizeof(*cmd_arr); i++)
      strcat (cmd, cmd_arr[i]);

    p = popen (cmd, "r");
  }

  /* Look for "$1 = ".  Assume unknown if we don't find it.  */
  sign = 0;

  while ((i = getc (p)) != '$')
    if (i == EOF)
      goto reached_eof;
  
  if ((i = getc (p)) != '1')
    abort ();

  if ((i = getc (p)) != ' ')
    abort ();

  if ((i = getc (p)) != '=')
    abort ();

  if ((i = getc (p)) != ' ')
    abort ();

  /* Parse the value.  We support decimal, hex led by "0x", with zero
     or more leading "-" signs.  "<unavailable>" is recognized by the
     leading '<'.  "(type)" is recognized by the leading paren, and
     skipped up to a matching paren, that must be followed by a
     blank.  */

  value_in = 0;
  base = 0;
  sign = 1;
  while ((i = getc (p)) != '\n')
    {
      int zero;

      if (i == EOF)
	abort ();

      if (base == 0)
	{
	  if (i == '<')
	    {
	      sign = 0;
	      break;
	    }
      
	  if (i == '(')
	    {
	      int unmatched = 1;
	      while (unmatched && (i = getc (p)) != EOF)
		if (i == '(')
		  unmatched++;
		else if (i == ')')
		  unmatched--;

	      if (i == EOF || (i = getc (p)) != ' ')
		abort ();
	      continue;
	    }

	  if (i == '-')
	    {
	      sign = -sign;
	      continue;
	    }

	  if (i == 'x')
	    {
	      base = 16;
	      continue;
	    }

	  if (i != '0')
	    base = 10;
	}

      zero = '0';

      if (i < '0' || i > '9')
	{
	  if (base != 16)
	    abort ();

	  if (i >= 'A' && i <= 'F')
	    zero = 'A' - 10;
	  else if (i >= 'a' && i <= 'f')
	    zero = 'a' - 10;
	  else
	    abort ();
	}

      value_in *= base;
      value_in += i - zero;
    }
	
  /* Skip any remaining junk.  */

  while ((i = getc (p)) != EOF)
    ;

 reached_eof:
  pclose (p);

  if (!sign)
    {
      printf ("%s: %s is unavailable, expected %lli\n",
	      unknown_ok ? "UNRESOLVED" : "FAIL", name, value);
      exit (unknown_ok ? INCOMPLETE : INCORRECT);
    }

  value_in *= sign;

  if (value != value_in)
    {
      printf ("FAIL: %s is %lli, expected %lli\n", name, value_in, value);
      exit (INCORRECT);
    }

  printf ("PASS: %s is %lli, as expected\n", name, value);
  exit (PASS);
}

/* Fork a child process and attach a debugger to the parent to
   evaluate NAME in the caller.  If it matches VALUE, we have a PASS;
   if it's unknown and UNKNOWN_OK, we have an UNRESOLVED.  Otherwise,
   it's a FAIL.  */

static void __attribute__((noinline))
guality_check (const char *name, long long value, int unknown_ok)
{
  pid_t pid;
  volatile int loop;
  int status;
  
  if (guality_skip)
    return;

  switch ((pid = fork ()))
    {
    case 0:
      attach (name, value, unknown_ok);

    case -1:
      abort ();

    default:
      break;
    }

  loop = 1;
  while (loop)
    ;

  if (wait (&status) != pid)
    abort ();

  if (!WIFEXITED (status))
    abort ();

  switch (WEXITSTATUS (status))
    {
    case PASS:
    case INCORRECT:
    case INCOMPLETE:
      ++guality_count[WEXITSTATUS (status)];
      break;

    default:
      abort ();
    }
}

#ifdef GUALCHK_SELF_TEST
/* Some silly sanity checking.  */

int
main (int argc, char *argv[])
{
  int i = argc+1;
  int j = argc-2;
  int k = 5;

  GUALCHK (argc);
  GUALCHK (&i);
  GUALCHK (i);
  GUALCHK (j);
  GUALCHKXPRVAL ("0x40", 64, 0);
  GUALCHKXPRVAL ("k", 5, 1);
}
#endif

[-- Attachment #3: Type: text/plain, Size: 257 bytes --]


-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: frameworklet to assess the quality of debug information

[Alexandre Oliva]

On Jul 28, 2008, Alexandre Oliva <aoliva@redhat.com> wrote:

> Thoughts, comments, suggestions, tomatoes, eggs? :-)

>    Contributed by Alexandre Oliva <dnovillo@redhat.com>.

                                     ^^^^^^^^

Heh.  So now you know I copied the header from some other file :-)

Fixed, thanks Dodji for pointing it out.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: frameworklet to assess the quality of debug information

[Joseph S. Myers]

On Mon, 28 Jul 2008, Alexandre Oliva wrote:

> Here's my first cut at trying to tell how well or how bad we perform
> in terms of debug info, that can be dropped into the GCC run-time test
> infrastructure and used by means of #include in new tests that add
> GUALCHK* annotations (or with separate compilation, if some stuff is
> moved into a separate header).
> 
> Thoughts, comments, suggestions, tomatoes, eggs? :-)

For the GCC testsuite parts of this will of course need to be written in 
Tcl to use the DejaGnu interfaces to execute both the debugger (on a 
possibly remote host) and the test program (on a possibly remote, 
simulator, etc. target), rather than fork/exec.  (For experimenting with 
different approaches, assessing the current state of the compiler, etc., 
this isn't needed so much.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: frameworklet to assess the quality of debug information

[Ian Lance Taylor]

Alexandre Oliva <aoliva@redhat.com> writes:

> Here's my first cut at trying to tell how well or how bad we perform
> in terms of debug info, that can be dropped into the GCC run-time test
> infrastructure and used by means of #include in new tests that add
> GUALCHK* annotations (or with separate compilation, if some stuff is
> moved into a separate header).
>
> Thoughts, comments, suggestions, tomatoes, eggs? :-)

It's an interesting approach.  Have you considered an approach more
like Dejagnu's dg support?  It would be harder to write tests, but
would permit testing debugging info less invasively.

int
main (int argc, char *argv[])
{
  int i = argc+1;
  int j = argc-2;
  int k = 5;

  /* { gualchk argc  1 } */
  /* { gualchk i 2 } */
  /* { gualchk j -1 }  */
}

Ian

Re: frameworklet to assess the quality of debug information

[Richard Guenther]

On Tue, Jul 29, 2008 at 5:17 AM, Ian Lance Taylor <iant@google.com> wrote:
> Alexandre Oliva <aoliva@redhat.com> writes:
>
>> Here's my first cut at trying to tell how well or how bad we perform
>> in terms of debug info, that can be dropped into the GCC run-time test
>> infrastructure and used by means of #include in new tests that add
>> GUALCHK* annotations (or with separate compilation, if some stuff is
>> moved into a separate header).
>>
>> Thoughts, comments, suggestions, tomatoes, eggs? :-)
>
> It's an interesting approach.  Have you considered an approach more
> like Dejagnu's dg support?  It would be harder to write tests, but
> would permit testing debugging info less invasively.
>
> int
> main (int argc, char *argv[])
> {
>  int i = argc+1;
>  int j = argc-2;
>  int k = 5;
>
>  /* { gualchk argc  1 } */
>  /* { gualchk i 2 } */
>  /* { gualchk j -1 }  */
> }

I am less concerned about this - we are writing debug testcases after
all.  I am more concerned that the current implementation has an
effect on the generated code we want to check (yes, I recognized
the comment about a two-stage approach ;)).  In that light, why not
pair the testcase with a gdb script directly?  (can one conditionally
exit gdb with a non-zero exit code?)

Thanks,
Richard.

Re: frameworklet to assess the quality of debug information

[Alexandre Oliva]

On Jul 29, 2008, Ian Lance Taylor <iant@google.com> wrote:

> Alexandre Oliva <aoliva@redhat.com> writes:
>> Here's my first cut at trying to tell how well or how bad we perform
>> in terms of debug info, that can be dropped into the GCC run-time test
>> infrastructure and used by means of #include in new tests that add
>> GUALCHK* annotations (or with separate compilation, if some stuff is
>> moved into a separate header).
>> 
>> Thoughts, comments, suggestions, tomatoes, eggs? :-)

> It's an interesting approach.  Have you considered an approach more
> like Dejagnu's dg support?

Yep.  Using comments only works for trivial testcases in which the
value is predictable, but it doesn't work at all once you start
inspecting loops.  That's once of the reasons why I went with macros
that expand to actual calls at run-time.

In an upcoming development of this infrastructure, I expect to drop
the 'call gdb on its own' stuff and have gdb start the program from
within a dejagnu script, set a breakpoint in the function called by
the macro with the expected value, and inspect that.  Then fix debug
info until we get at least that right, in the presence of calls to
sync things up and simplify the problem a bit.

Then, at a later stage, move to the logging approach that leaves no
markers whatsoever in the second run, and then somehow get the
debugger to inspect things.  Maybe have the macros generate tables
that the debug script would consult or some such.

I haven't thought this through, but I'm trying to come up with an
interface to code these tests that's generic enough to provide the
inspection features we're going to need, but that won't stop us from
evolving the underlying implementation, as we gain guality, so as to
progressively reduce the influence of the testing infrastructure on
the generate code we want to test.

ATM GCC sucks rocks even with the function calls and the forced-live
annotations in place :-(

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: frameworklet to assess the quality of debug information

[Alexandre Oliva]

On Jul 29, 2008, "Richard Guenther" <richard.guenther@gmail.com> wrote:

> why not pair the testcase with a gdb script directly?

Mostly a matter of convenience.  Writing code and adding "test this
here, etc", and not having to adjust a testcase all over just because
you have to add an #include feels so much better than the alternative.
And then, having it all in a single file that uses shared
infrastructure feels better than having to write one script per test,
but maybe that's just me.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: frameworklet to assess the quality of debug information

[Richard Guenther]

On Tue, Jul 29, 2008 at 9:01 PM, Alexandre Oliva <aoliva@redhat.com> wrote:
> On Jul 29, 2008, "Richard Guenther" <richard.guenther@gmail.com> wrote:
>
>> why not pair the testcase with a gdb script directly?
>
> Mostly a matter of convenience.  Writing code and adding "test this
> here, etc", and not having to adjust a testcase all over just because
> you have to add an #include feels so much better than the alternative.
> And then, having it all in a single file that uses shared
> infrastructure feels better than having to write one script per test,
> but maybe that's just me.

Ok, but maybe there's a trick to embed the gdb script in the source
like you can embed a Makefile in the source ...

Richard.

Re: frameworklet to assess the quality of debug information

[Daniel Jacobowitz]

On Mon, Jul 28, 2008 at 06:39:40PM -0300, Alexandre Oliva wrote:
> Here's my first cut at trying to tell how well or how bad we perform
> in terms of debug info, that can be dropped into the GCC run-time test
> infrastructure and used by means of #include in new tests that add
> GUALCHK* annotations (or with separate compilation, if some stuff is
> moved into a separate header).

FWIW, I think this is a good approach.

-- 
Daniel Jacobowitz
CodeSourcery