gdb doesn't work very well with dynamic linked binaries

gdb doesn't work very well with dynamic linked binaries

[H . J . Lu]

On Mon, Sep 04, 2000 at 11:47:13AM +0100, Joern Rennecke wrote:
> > It is too bad that not many gcc developers using --enable-shared to
> > configure gcc under ia32. See
> 
> Well, on Linux, gdb fails to restart a cc1 / cc1plus executable that is
> statically linked, which makes debugging very tedious.
> So I always patch my Makefile to use static linking.

That is a very annoying bug in gdb 5.0. When I set a breakpoint in
the shared library, I have to disable them before restart. Also it is
very hard to debug ld-linux.so.2:

# gdb ld-linux.so.2



H.J.

Re: gdb doesn't work very well with dynamic linked binaries

[Mark Kettenis]

   Date: Mon, 4 Sep 2000 08:49:34 -0700
   From: "H . J . Lu" <hjl@lucon.org>

   On Mon, Sep 04, 2000 at 11:47:13AM +0100, Joern Rennecke wrote:
   > > It is too bad that not many gcc developers using --enable-shared to
   > > configure gcc under ia32. See
   > 
   > Well, on Linux, gdb fails to restart a cc1 / cc1plus executable that is
   > statically linked, which makes debugging very tedious.
   > So I always patch my Makefile to use static linking.

   That is a very annoying bug in gdb 5.0. When I set a breakpoint in
   the shared library, I have to disable them before restart.

Getting this bug fixed is one of the release criteria for GDB 5.1.
All I have now is a hack that works around the problems, see the GDB
TODO file for more info.

   Also it is very hard to debug ld-linux.so.2:

   # gdb ld-linux.so.2

AFAIK this never worked.  Patches to make it work are certainly
welcome.  Uli mentioned he submitted some patches, but I cannot find
any trace of them.  Perhaps you re-submit them Uli?

Mark

Re: gdb doesn't work very well with dynamic linked binaries

[Daniel Berlin]

> 
> AFAIK this never worked.  Patches to make it work are certainly
> welcome.  Uli mentioned he submitted some patches, but I cannot find
> any trace of them.  Perhaps you re-submit them Uli?

If they were the same ones he sent me, i forwarded them along to various
gdb people, but the consensus was that it didn't actually fix the real
problem.
--Dan
> > Mark
> 

Re: gdb doesn't work very well with dynamic linked binaries

[Ulrich Drepper]

Daniel Berlin <dan@cgsoftware.com> writes:

> If they were the same ones he sent me,

They are

> i forwarded them along to various gdb people, but the consensus was
> that it didn't actually fix the real problem.

Well, then fix it correctly.  I'm using the patches for years without
experiencing negative side effects.  Only with them is it possible to
debug ld.so.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: gdb doesn't work very well with dynamic linked binaries

[Kevin Buettner]

On Sep 4, 11:16am, Ulrich Drepper wrote:

> > i forwarded them along to various gdb people, but the consensus was
> > that it didn't actually fix the real problem.
> 
> Well, then fix it correctly.  I'm using the patches for years without
> experiencing negative side effects.  Only with them is it possible to
> debug ld.so.

I took a look at these patches in late July in the hope that they
would fix some problems that I was seeing (on a not-to-be-named
platform) with relocating the main executable.  Below is a portion of
a message that I sent to one of the internal Red Hat lists concerning
Uli's solib.c patches.  In order to make sense of some of my comments,
it helps to know that I needed the read-only and read/write sections
to be relocated by different amounts.  This isn't terribly relevant
for the discussion at hand, but I think that any solution we come
up with needs to handle this case.  (My comments immediately preceding
Uli's patch *are* relevant though.)

......................................................................

I tried them and they didn't work for me.  There are several problems
with these patches for my situation:

    1)  The exec_bfd isn't marked DYNAMIC.  (But the OS is relocating
	it anyway; according to the ABI, this is okay.) Anyway, since
	it isn't marked DYNAMIC, Uli's code for relocating the symbols
	doesn't get a chance to run.

    2)  The stop_pc when this code is hit is at the _start symbol in
        the dynamic linker.  But I want to relocate the main executable
	whose _start symbol hasn't been hit yet.

    3)  Even if the preceding two problems could be reconciled, the
        .text and .data sections need to be relocated by different
	amounts.

[...]

I'm going to back out Uli's patch from my sandbox.  It didn't build
cleanly in my sandbox, so I'm posting below a cleaned up version which
does build.  We'll need to incorporate something like this into gdb at
some point.  Before we do though, I'd like to understand why the
changes to breakpoint.c are necessary.  Also, we need to consider the
situation where exec_bfd is not the dynamic linker, but is marked
DYNAMIC for some reason.  If this should happen, Uli's code will get
hit and the symbols in exec_bfd / symfile_objfile will get improperly
relocated.

Index: solib.c
===================================================================
RCS file: /cvs/cvsfiles/devo/gdb/solib.c,v
retrieving revision 1.146
diff -u -p -r1.146 solib.c
--- solib.c	2000/05/28 01:25:33	1.146
+++ solib.c	2000/07/26 17:50:36
@@ -54,6 +54,7 @@
 #include "environ.h"
 #include "language.h"
 #include "gdbcmd.h"
+#include "objfiles.h"
 
 #define MAX_PATH_SIZE 512	/* FIXME: Should be dynamic */
 
@@ -1984,6 +1985,39 @@ solib_create_inferior_hook ()
     {
       warning ("shared library handler failed to enable breakpoint");
       return;
+    }
+
+  if ((bfd_get_file_flags (exec_bfd) & DYNAMIC) != 0
+      && bfd_get_start_address (exec_bfd) != stop_pc)
+    {
+      /* We have to relocate the debug information.  */
+      static CORE_ADDR last_displacement;
+      CORE_ADDR displacement = stop_pc - exec_bfd->start_address;
+
+      if (last_displacement != displacement)
+	{
+	  CORE_ADDR correction = displacement - last_displacement;
+	  struct section_offsets *new_offsets;
+	  int i;
+
+	  new_offsets = alloca (symfile_objfile->num_sections
+				* sizeof (*new_offsets));
+
+	  for (i = 0; i < symfile_objfile->num_sections; ++i)
+	    ANOFFSET (new_offsets, i) =
+	      ANOFFSET (symfile_objfile->section_offsets, i);
+
+	  ANOFFSET (new_offsets, SECT_OFF_TEXT (symfile_objfile)) += displacement;
+	  ANOFFSET (new_offsets, SECT_OFF_DATA (symfile_objfile)) += displacement;
+	  ANOFFSET (new_offsets, SECT_OFF_BSS (symfile_objfile)) += displacement;
+	  ANOFFSET (new_offsets, SECT_OFF_RODATA (symfile_objfile)) += displacement;
+
+	  objfile_relocate (symfile_objfile, new_offsets);
+	  breakpoint_re_set ();
+
+	  /* Remember the current displacement.  */
+	  last_displacement = displacement;
+	}
     }
 
 #if !defined(SVR4_SHARED_LIBS) || defined(_SCO_DS)
Index: breakpoint.c
===================================================================
RCS file: /cvs/cvsfiles/devo/gdb/breakpoint.c,v
retrieving revision 1.294
diff -u -p -r1.294 breakpoint.c
--- breakpoint.c	2000/06/04 00:35:16	1.294
+++ breakpoint.c	2000/07/26 17:50:44
@@ -7108,6 +7108,7 @@ breakpoint_re_set_one (bint)
       return 0;
     case bp_breakpoint:
     case bp_hardware_breakpoint:
+    case bp_shlib_event:
     case bp_catch_load:
     case bp_catch_unload:
       if (b->addr_string == NULL)
@@ -7246,10 +7247,6 @@ breakpoint_re_set_one (bint)
 
       /* This breakpoint is special, it's set up when the inferior
          starts and we really don't want to touch it.  */
-    case bp_shlib_event:
-
-      /* Like bp_shlib_event, this breakpoint type is special.
-	 Once it is set up, we do not want to touch it.  */
     case bp_thread_event:
 
       /* Keep temporary breakpoints, which can be encountered when we step

Re: gdb doesn't work very well with dynamic linked binaries

[H . J . Lu]

On Mon, Sep 04, 2000 at 07:51:53PM +0200, Mark Kettenis wrote:
>    Date: Mon, 4 Sep 2000 08:49:34 -0700
>    From: "H . J . Lu" <hjl@lucon.org>
> 
>    On Mon, Sep 04, 2000 at 11:47:13AM +0100, Joern Rennecke wrote:
>    > > It is too bad that not many gcc developers using --enable-shared to
>    > > configure gcc under ia32. See
>    > 
>    > Well, on Linux, gdb fails to restart a cc1 / cc1plus executable that is
>    > statically linked, which makes debugging very tedious.
>    > So I always patch my Makefile to use static linking.
> 
>    That is a very annoying bug in gdb 5.0. When I set a breakpoint in
>    the shared library, I have to disable them before restart.
> 
> Getting this bug fixed is one of the release criteria for GDB 5.1.
> All I have now is a hack that works around the problems, see the GDB
> TODO file for more info.
> 

How about the hardware watchpoints on ia32? I mean

1. Delete hardware watchpoints to free hardware debug registers. Set 4
hardware watchpoints. Then delete/disable one hardware watchpoint. Set
another hardware watchpoint. Can gdb free a hardware debug register
when I delete/disable the hardware watchpoint which uses it?
2. Watch for different values on a viariable with one hardware debug
register. That is do

(gdb) watch foobar == 1
(gdb) watch foobar == 2
(gdb) watch foobar == 3
(gdb) watch foobar == 4
(gdb) watch foobar == 5

only using one hardware debug register.

I have reported them long before 5.0 was released. But at least #1
still doesn't work right in 5.0 under Linux/ia32.

Thanks.

H.J.

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

> 
> Date: Mon, 4 Sep 2000 16:44:58 -0700
> From: "H . J . Lu" <hjl@lucon.org>
> 
> 1. Delete hardware watchpoints to free hardware debug registers. Set 4
> hardware watchpoints. Then delete/disable one hardware watchpoint. Set
> another hardware watchpoint. Can gdb free a hardware debug register
> when I delete/disable the hardware watchpoint which uses it?
> 2. Watch for different values on a viariable with one hardware debug
> register. That is do
> 
> (gdb) watch foobar == 1
> (gdb) watch foobar == 2
> (gdb) watch foobar == 3
> (gdb) watch foobar == 4
> (gdb) watch foobar == 5
> 
> only using one hardware debug register.
> 
> I have reported them long before 5.0 was released. But at least #1
> still doesn't work right in 5.0 under Linux/ia32.

These are not GDB/ia32 issues per se: the above features are all
implemented in the DJGPP port of GDB and work in v5.0.  Every
x86-based target should be able to lift the relevant parts of
go32-nat.c and use them almost verbatim.  You get debug register
sharing through reference counts, and the ability to watch large
regions (up to 16 bytes) using multiple registers.  (The required
infrastructure in high-level GDB application code, mostly in
breakpoint.c, is also working since v5.0.)

What is missing is something that we discussed here some time ago: a
unified handling for debug registers common for ALL ia32 targets.  If
you want to get this done before 5.1 is out, I'm for it.  I said in
the past that I'm willing to volunteer to pull the code out of
go32-nat.c and generalize it as appropriate, as the first step towards
this goal.  Provided that it's decided to do that for 5.1, of course
(otherwise, I have too many other important things to do ;-).

Re: gdb doesn't work very well with dynamic linked binaries

[H . J . Lu]

On Tue, Sep 05, 2000 at 01:48:56AM -0400, Eli Zaretskii wrote:
> > 
> > Date: Mon, 4 Sep 2000 16:44:58 -0700
> > From: "H . J . Lu" <hjl@lucon.org>
> > 
> > 1. Delete hardware watchpoints to free hardware debug registers. Set 4
> > hardware watchpoints. Then delete/disable one hardware watchpoint. Set
> > another hardware watchpoint. Can gdb free a hardware debug register
> > when I delete/disable the hardware watchpoint which uses it?
> > 2. Watch for different values on a viariable with one hardware debug
> > register. That is do
> > 
> > (gdb) watch foobar == 1
> > (gdb) watch foobar == 2
> > (gdb) watch foobar == 3
> > (gdb) watch foobar == 4
> > (gdb) watch foobar == 5
> > 
> > only using one hardware debug register.
> > 
> > I have reported them long before 5.0 was released. But at least #1
> > still doesn't work right in 5.0 under Linux/ia32.
> 
> These are not GDB/ia32 issues per se: the above features are all
> implemented in the DJGPP port of GDB and work in v5.0.  Every
> x86-based target should be able to lift the relevant parts of
> go32-nat.c and use them almost verbatim.  You get debug register
> sharing through reference counts, and the ability to watch large
> regions (up to 16 bytes) using multiple registers.  (The required
> infrastructure in high-level GDB application code, mostly in
> breakpoint.c, is also working since v5.0.)
> 
> What is missing is something that we discussed here some time ago: a
> unified handling for debug registers common for ALL ia32 targets.  If
> you want to get this done before 5.1 is out, I'm for it.  I said in
> the past that I'm willing to volunteer to pull the code out of
> go32-nat.c and generalize it as appropriate, as the first step towards
> this goal.  Provided that it's decided to do that for 5.1, of course
> (otherwise, I have too many other important things to do ;-).
> 

If you can generalize it for ia32, I will implement it for Linux/ia32.
If it won't be fixed in 5.1, I will follow your hints and implement a
Linux only solution when it happens to me again. It is just one of
those things which makes me to roll my own stuff. Hardware watchpoints
have been known to be broken on Linux/ia32 for a long time and nothing
has been done to it.


H.J.

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

>   Date: Mon, 4 Sep 2000 23:32:22 -0700
>   From: "H . J . Lu" <hjl@lucon.org>
>
>   If you can generalize it for ia32, I will implement it for Linux/ia32.

The code on go32-nat.c manipulates an array which represents the ia32
debug registers, including the status and control registers.  It
leaves it to resume() and its subroutines on the target end to
actually insert the watchpoints when the inferior is resumed and
remove them when the inferior stops and control is passed to GDB.

If this model suits most or all ia32 targets, pulling the code from
go32-nat.c into a separate module (probably, as part of i386-nat.c)
would be very easy for me.  If not, I'd ask the relevant maintainers
to tell what provisions should I make for other platforms to fit in.

>   If it won't be fixed in 5.1, I will follow your hints and implement a
>   Linux only solution when it happens to me again.

I can do this Very Soon (tm) provided that I hear a GO from The Powers
That Be.  Andrew?  Stan?  What say you?

>   Hardware watchpoints
>   have been known to be broken on Linux/ia32 for a long time and nothing
>   has been done to it.

That's not 100% true: a few important patches related to watchpoints
went into mainstream sources (mainly in breakpoint.c) in preparation
for v5.0.  This is the infrastructure I was talking about in my
previous message; without those patches the watchpoint support in
go32-nat.c could not work reliably.  (IIRC, a major part of these
patches were resubmitted by me and approved by Michael Snyder as a
result of a discussion in which you participated.)

Re: gdb doesn't work very well with dynamic linked binaries

[Mark Kettenis]

   Date: Tue, 5 Sep 2000 06:35:52 -0400 (EDT)
   From: Eli Zaretskii <eliz@delorie.com>

   >   Date: Mon, 4 Sep 2000 23:32:22 -0700
   >   From: "H . J . Lu" <hjl@lucon.org>
   >
   >   If you can generalize it for ia32, I will implement it for Linux/ia32.

   The code on go32-nat.c manipulates an array which represents the ia32
   debug registers, including the status and control registers.  It
   leaves it to resume() and its subroutines on the target end to
   actually insert the watchpoints when the inferior is resumed and
   remove them when the inferior stops and control is passed to GDB.

   If this model suits most or all ia32 targets, pulling the code from
   go32-nat.c into a separate module (probably, as part of i386-nat.c)
   would be very easy for me.  If not, I'd ask the relevant maintainers
   to tell what provisions should I make for other platforms to fit in.

I looked into the the hardware watchpoint support some time ago, and
actually started coding, seperating the generic bits out of the go32
code.  I was thinking of having some sort of register cache for the
debugging registers but didn't immedeately see the right solution to
do that.  Perhaps they should simply be added to the register cache?
I also couldn't directly see how the hardware watchpoint supports
interacts with multiple threads, especially in presence of the nifty
way the go32 code maps multiple waitchpoints on a single debugging
register.

In principle I see no reason why x86 hardware waitchpoints couldn't be
implemented by i386-tdep.c though.

   >   If it won't be fixed in 5.1, I will follow your hints and implement a
   >   Linux only solution when it happens to me again.

A generic x86 solution would be preferable, but a clean, well
documented Linux-only solution is certainly welcome.

   I can do this Very Soon (tm) provided that I hear a GO from The Powers
   That Be.  Andrew?  Stan?  What say you?

   >   Hardware watchpoints
   >   have been known to be broken on Linux/ia32 for a long time and nothing
   >   has been done to it.

Apparently nobody cares enough.  It isn't at the top of my priority
list so if nobody else contributes the necessary code, it probably
won't happen in the near future.

Mark

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

> Date: Tue, 5 Sep 2000 15:33:54 +0200 (MET DST)
> From: Mark Kettenis <kettenis@wins.uva.nl>
> 
> I was thinking of having some sort of register cache for the
> debugging registers but didn't immedeately see the right solution to
> do that.  Perhaps they should simply be added to the register cache?

I think all we need is to store the debug registers somewhere.  They
need to be accessed by (1) the x86-dependent code which is called by
GDB's application level to insert and remove watchpoints; (2) the
target-specific code which actually calls ptrace or the equivalent
syscall to pass the values into the inferior's registers before
resuming it, and set bits after the inferior stops to indicate which
watchpoint(s) triggered; and (3) by stopped_by_watchpoint's target
end.

It's possible that the register cache is as good place as any to store
DRi, even though they slightly differ from the rest of the registers.

> I also couldn't directly see how the hardware watchpoint supports
> interacts with multiple threads

Sorry, I'm not sure I see the problem.  Could you please elaborate?
(I'm afraid I don't know much about Linux threads.)

> especially in presence of the nifty way the go32 code maps multiple
> waitchpoints on a single debugging register.

I don't think this should matter.  The reference counts are just a
means to know which register is used and which isn't.  As far as the
higher-level GDB code is concerned, what's important is (a) which
address, if any, triggered a watchpoint; and (b) which thread caused
the watchpoint to trigger.

However, if I'm missing the point, and there's some additional
infrastructure that is required to make this work with multiple
threads, please tell what is, or might be, missing.

> Apparently nobody cares enough.  It isn't at the top of my priority
> list so if nobody else contributes the necessary code, it probably
> won't happen in the near future.

If v5.1 freeze date is far away, and if the current code in go32-nat.c
is good enough to be used by other x86 platforms, I might take the
silence as a sign of agreement, given the fact that making it happen
is easy... ;-)

Re: gdb doesn't work very well with dynamic linked binaries

[Mark Kettenis]

   Date: Tue, 5 Sep 2000 11:46:48 -0400 (EDT)
   From: Eli Zaretskii <eliz@delorie.com>

   > Date: Tue, 5 Sep 2000 15:33:54 +0200 (MET DST)
   > From: Mark Kettenis <kettenis@wins.uva.nl>
   > 
   > I was thinking of having some sort of register cache for the
   > debugging registers but didn't immedeately see the right solution to
   > do that.  Perhaps they should simply be added to the register cache?

   I think all we need is to store the debug registers somewhere.  They
   need to be accessed by (1) the x86-dependent code which is called by
   GDB's application level to insert and remove watchpoints; (2) the
   target-specific code which actually calls ptrace or the equivalent
   syscall to pass the values into the inferior's registers before
   resuming it, and set bits after the inferior stops to indicate which
   watchpoint(s) triggered; and (3) by stopped_by_watchpoint's target
   end.

   It's possible that the register cache is as good place as any to store
   DRi, even though they slightly differ from the rest of the registers.

Yep, and that's where I got distracted by more urgent problems :-(.

   > I also couldn't directly see how the hardware watchpoint supports
   > interacts with multiple threads

   Sorry, I'm not sure I see the problem.  Could you please elaborate?
   (I'm afraid I don't know much about Linux threads.)

Me neither :-(.  I'm not sure whether the debug registers are
per-thread or per-VM-space in Linux.  I'll probably need to look into
the kernel source.

   > especially in presence of the nifty way the go32 code maps multiple
   > waitchpoints on a single debugging register.

   I don't think this should matter.  The reference counts are just a
   means to know which register is used and which isn't.  As far as the
   higher-level GDB code is concerned, what's important is (a) which
   address, if any, triggered a watchpoint; and (b) which thread caused
   the watchpoint to trigger.

If you can set the debug registers per-thread, we might need a
reference count for every thread.  If the debug registers are
per-VM-space there might be a potential problem with finding out the
right triggering address for the right thread.

   However, if I'm missing the point, and there's some additional
   infrastructure that is required to make this work with multiple
   threads, please tell what is, or might be, missing.

It's something that needs to be investigated.  But since DJGPP doesn't
seem to support multiple threads I certainly don't expect you to do
that :-).

Mark

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

> Date: Wed, 6 Sep 2000 02:06:50 +0200
> From: Mark Kettenis <kettenis@wins.uva.nl>
> 
> If you can set the debug registers per-thread, we might need a
> reference count for every thread.

Oh, I see.  I assumed that the debug registers are per VM; I should
have known better.

In that case, adding the debug registers to the register cache is
probably the way to go (since the other registers are per thread,
right?).  And the functions which manipulate the debug registers
should accept an array of registers as their argument--this should
allow any specific target to DTRT easily.

Does this sound as a good plan?

> But since DJGPP doesn't seem to support multiple threads I certainly
> don't expect you to do that :-).

Threading is possible with DJGPP, but the DJGPP port of GDB doesn't
have any special support for debugging such programs.

Re: gdb doesn't work very well with dynamic linked binaries

[H . J . Lu]

On Tue, Sep 05, 2000 at 03:33:54PM +0200, Mark Kettenis wrote:
> 
>    >   If it won't be fixed in 5.1, I will follow your hints and implement a
>    >   Linux only solution when it happens to me again.
> 
> A generic x86 solution would be preferable, but a clean, well
> documented Linux-only solution is certainly welcome.

Given do it clean, do it fast and make it to work, you can only
pick 2 if I am the only one to do it. Since I don't have much time
to do it, I have to pick do it fast.

> 
>    I can do this Very Soon (tm) provided that I hear a GO from The Powers
>    That Be.  Andrew?  Stan?  What say you?
> 
>    >   Hardware watchpoints
>    >   have been known to be broken on Linux/ia32 for a long time and nothing
>    >   has been done to it.
> 
> Apparently nobody cares enough.  It isn't at the top of my priority
> list so if nobody else contributes the necessary code, it probably
> won't happen in the near future.

It has to be fixed in 5.1, one way or the other. The worst case is
I will make my kludge available to the Linux community.


H.J.

Re: gdb doesn't work very well with dynamic linked binaries

[Stan Shebs]

"H . J . Lu" wrote:
> 
> On Tue, Sep 05, 2000 at 03:33:54PM +0200, Mark Kettenis wrote:
> >
> >    >   If it won't be fixed in 5.1, I will follow your hints and implement a
> >    >   Linux only solution when it happens to me again.
> >
> > A generic x86 solution would be preferable, but a clean, well
> > documented Linux-only solution is certainly welcome.
> 
> Given do it clean, do it fast and make it to work, you can only
> pick 2 if I am the only one to do it. Since I don't have much time
> to do it, I have to pick do it fast.

I feel like I should say something here - doing things fast for GNU/Linux
has been a continuing source of problems.  All this stuff is going
to take the same amount of time in the end, whether we kludge now and
rewrite over and over later, or do it right the first time.  GDB Linux
thread hacking has been going on for a couple years, and it's still not
quite done; if I had known it was going to take this long, I would
have taken a harder line about accepting the original expedient version.

If you don't think that you're getting enough time to do things right,
then you should take it up with your management.  Since VA Linux thinks
highly enough of you to feature your picture prominently in their Linux
World booth :-), you should have enough pull to say "this is how long
it needs to take".  If you supply me with names, I will be happy to
take it up with them myself too - VA Linux' business depends heavily on
its reputation for good Linux engineering, and I doubt they want to
become known as the company that is pushing hacky forked versions of GDB
out into the world.

Stan

Re: gdb doesn't work very well with dynamic linked binaries

[H . J . Lu]

On Tue, Sep 05, 2000 at 06:23:31PM -0700, Stan Shebs wrote:
> 
> If you don't think that you're getting enough time to do things right,
> then you should take it up with your management.  Since VA Linux thinks

The gdb work has nothing to do with my job at VA :-). I don't even
know they are aware of it. I only work on gdb because I don't want to
wait for a few hours when debugging the code nor start all over when
I use up all hardware registers. If it is a must-fix for gdb 5.1, I
am willing to pitch in. If noone else wants to spend time on it, I
will come up with some kludge for myself.

BTW, it has been a long standing problem for gdb. It seems that I
am the only one who thinks gdb sucks under Linux/ia32 and is willing
to do something about it.


H.J.

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

> Date: Tue, 5 Sep 2000 18:33:12 -0700
> From: "H . J . Lu" <hjl@lucon.org>
>
> I only work on gdb because I don't want to wait for a few hours when
> debugging the code nor start all over when I use up all hardware
> registers.

Same here.  I debug development versions of Emacs as a matter of
routine, and find that to be impossible in many ``interesting'' cases
without watchpoints.  That was my motivation to fix them, including
the ability to watch struct members and array elements.

> If it is a must-fix for gdb 5.1, I am willing to pitch in. If noone
> else wants to spend time on it, I will come up with some kludge for
> myself.

I will try to pull the relevant code out of go32-nat.c and put it on
i386-tdep.c in the next few weeks.

Re: gdb doesn't work very well with dynamic linked binaries

[H . J . Lu]

On Tue, Sep 05, 2000 at 06:35:52AM -0400, Eli Zaretskii wrote:
> >   Date: Mon, 4 Sep 2000 23:32:22 -0700
> >   From: "H . J . Lu" <hjl@lucon.org>
> >
> >   If you can generalize it for ia32, I will implement it for Linux/ia32.
> 
> The code on go32-nat.c manipulates an array which represents the ia32
> debug registers, including the status and control registers.  It
> leaves it to resume() and its subroutines on the target end to
> actually insert the watchpoints when the inferior is resumed and
> remove them when the inferior stops and control is passed to GDB.
> 
> If this model suits most or all ia32 targets, pulling the code from
> go32-nat.c into a separate module (probably, as part of i386-nat.c)
> would be very easy for me.  If not, I'd ask the relevant maintainers
> to tell what provisions should I make for other platforms to fit in.
> 
> >   If it won't be fixed in 5.1, I will follow your hints and implement a
> >   Linux only solution when it happens to me again.
> 
> I can do this Very Soon (tm) provided that I hear a GO from The Powers
> That Be.  Andrew?  Stan?  What say you?

I will vote for fixing it in 5.1. I will do the Linux work since I
brought it up. It has been broken for too long. There is no excuse
not to fix it given that at least 2 people are willing to do it.

> 
> >   Hardware watchpoints
> >   have been known to be broken on Linux/ia32 for a long time and nothing
> >   has been done to it.
> 
> That's not 100% true: a few important patches related to watchpoints
> went into mainstream sources (mainly in breakpoint.c) in preparation

Sorry for the misunderstanding. Thanks for your work on breakpoints.
It is just that hardware watchpoints under Linux/ia32 are as broken
as before :-).


H.J.

Re: gdb doesn't work very well with dynamic linked binaries

[Stan Shebs]

Eli Zaretskii wrote:
> 
> >   Date: Mon, 4 Sep 2000 23:32:22 -0700
> >   From: "H . J . Lu" <hjl@lucon.org>
> >
> >   If you can generalize it for ia32, I will implement it for Linux/ia32.
> 
> The code on go32-nat.c manipulates an array which represents the ia32
> debug registers, including the status and control registers.  It
> leaves it to resume() and its subroutines on the target end to
> actually insert the watchpoints when the inferior is resumed and
> remove them when the inferior stops and control is passed to GDB.
> 
> If this model suits most or all ia32 targets, pulling the code from
> go32-nat.c into a separate module (probably, as part of i386-nat.c)
> would be very easy for me.  If not, I'd ask the relevant maintainers
> to tell what provisions should I make for other platforms to fit in.
> 
> >   If it won't be fixed in 5.1, I will follow your hints and implement a
> >   Linux only solution when it happens to me again.
> 
> I can do this Very Soon (tm) provided that I hear a GO from The Powers
> That Be.  Andrew?  Stan?  What say you?

Uh, is there any reason not to?  We tell people that GDB can support
h/w watchpoints, seems like we ought to deliver it on our most popular
platforms.  Perhaps I could be evil and insist on adding a testsuite test
that would take 24 hours to run if h/w watchpoints don't work... think
that would help motivate anyone? :-)

Stan

Re: gdb doesn't work very well with dynamic linked binaries

[H . J . Lu]

On Tue, Sep 05, 2000 at 06:01:39PM -0700, Stan Shebs wrote:
> > 
> > I can do this Very Soon (tm) provided that I hear a GO from The Powers
> > That Be.  Andrew?  Stan?  What say you?
> 
> Uh, is there any reason not to?  We tell people that GDB can support
> h/w watchpoints, seems like we ought to deliver it on our most popular
> platforms.  Perhaps I could be evil and insist on adding a testsuite test
> that would take 24 hours to run if h/w watchpoints don't work... think
> that would help motivate anyone? :-)
> 

How about this patch? As I said, I will work on Linux since I use this
feature a lot.

Thanks.


H.J.
----
2000-09-05  H.J. Lu  <hjl@gnu.org>

	* TODO: Add hardware watchpint problems on x86 OSes for 5.1.

--- TODO	Thu Aug 10 15:23:13 2000
+++ /tmp/TODO	Tue Sep  5 20:41:08 2000
@@ -12,6 +12,26 @@ cycle.  People hope to have these proble
 
 --
 
+Hardware watchpint problems on x86 OSes, including Linux:
+
+1. Delete/disable hardware watchpoints should free hardware debug
+registers. 
+2. Watch for different values on a viariable with one hardware debug
+register.
+
+According to Eli Zaretskii <eliz@delorie.com>:
+
+These are not GDB/ia32 issues per se: the above features are all
+implemented in the DJGPP port of GDB and work in v5.0.  Every
+x86-based target should be able to lift the relevant parts of
+go32-nat.c and use them almost verbatim.  You get debug register
+sharing through reference counts, and the ability to watch large
+regions (up to 16 bytes) using multiple registers.  (The required
+infrastructure in high-level GDB application code, mostly in
+breakpoint.c, is also working since v5.0.)
+
+--
+
 RFD: infrun.c: No bpstat_stop_status call after proceed over break?
 http://sourceware.cygnus.com/ml/gdb-patches/2000-q1/msg00665.html
 

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

> Date: Tue, 5 Sep 2000 20:45:19 -0700
> From: "H . J . Lu" <hjl@lucon.org>
> 
> How about this patch? As I said, I will work on Linux since I use this
> feature a lot.
> 
> Thanks.
> 
> 
> H.J.
> ----
> 2000-09-05  H.J. Lu  <hjl@gnu.org>
> 
> 	* TODO: Add hardware watchpint problems on x86 OSes for 5.1.

Fine with me.

Re: gdb doesn't work very well with dynamic linked binaries

[Andrew Cagney]

FYI,

I'm going to apply this patch but as part of the 5.2 criteria.  If it is
fixed in 5.1 then it is a bonus.

	enjoy,
		Andrew


"H . J . Lu" wrote:
> 
> On Tue, Sep 05, 2000 at 06:01:39PM -0700, Stan Shebs wrote:
> > >
> > > I can do this Very Soon (tm) provided that I hear a GO from The Powers
> > > That Be.  Andrew?  Stan?  What say you?
> >
> > Uh, is there any reason not to?  We tell people that GDB can support
> > h/w watchpoints, seems like we ought to deliver it on our most popular
> > platforms.  Perhaps I could be evil and insist on adding a testsuite test
> > that would take 24 hours to run if h/w watchpoints don't work... think
> > that would help motivate anyone? :-)
> >
> 
> How about this patch? As I said, I will work on Linux since I use this
> feature a lot.
> 
> Thanks.
> 
> H.J.
> ----
> 2000-09-05  H.J. Lu  <hjl@gnu.org>
> 
>         * TODO: Add hardware watchpint problems on x86 OSes for 5.1.
> 
> --- TODO        Thu Aug 10 15:23:13 2000
> +++ /tmp/TODO   Tue Sep  5 20:41:08 2000
> @@ -12,6 +12,26 @@ cycle.  People hope to have these proble
> 
>  --
> 
> +Hardware watchpint problems on x86 OSes, including Linux:
> +
> +1. Delete/disable hardware watchpoints should free hardware debug
> +registers.
> +2. Watch for different values on a viariable with one hardware debug
> +register.
> +
> +According to Eli Zaretskii <eliz@delorie.com>:
> +
> +These are not GDB/ia32 issues per se: the above features are all
> +implemented in the DJGPP port of GDB and work in v5.0.  Every
> +x86-based target should be able to lift the relevant parts of
> +go32-nat.c and use them almost verbatim.  You get debug register
> +sharing through reference counts, and the ability to watch large
> +regions (up to 16 bytes) using multiple registers.  (The required
> +infrastructure in high-level GDB application code, mostly in
> +breakpoint.c, is also working since v5.0.)
> +
> +--
> +
>  RFD: infrun.c: No bpstat_stop_status call after proceed over break?
>  http://sourceware.cygnus.com/ml/gdb-patches/2000-q1/msg00665.html
>

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

> Date: Tue, 05 Sep 2000 18:01:39 -0700
> From: Stan Shebs <shebs@apple.com>
> > 
> > I can do this Very Soon (tm) provided that I hear a GO from The Powers
> > That Be.  Andrew?  Stan?  What say you?
> 
> Uh, is there any reason not to?  We tell people that GDB can support
> h/w watchpoints, seems like we ought to deliver it on our most popular
> platforms.

Agreed.  I think that having watchpoint support in good shape is
important, even if initially it won't work well with multithreaded
programs.

> Perhaps I could be evil and insist on adding a testsuite test
> that would take 24 hours to run if h/w watchpoints don't work... think
> that would help motivate anyone? :-)

I think I now have all the motivation I need ;-)

I'll be away for about two weeks starting this weekend, but I'll try
to do my part of this soon after I return.

Re: About unified debug register handling for i386 CPU.

[Eli Zaretskii]

> Date: Wed, 06 Sep 2000 00:13:39 +0200
> From: Pierre Muller <pierre@idefix.wisa.be>
> 
>   Just a short comment to remind everyone that 
> at least in DPMI you don't have direct access to the debug register.

Yes, there's a system call to set and reset them.

>   (Imagine for instance that you debug gdb with itself
> and place a hardware watchpoint on the top-gdb,
> only 3 hardware debug registers are available to
> the second gdb, but you will only notice that by a failure
> to set the fourth hardware register !)

This is a known problem with the DJGPP port.  It is one of a list of
problems that need special care because the debugger and the debuggee
are really one process.

In this case, I think the DJGPP debug support library functions should
allow to query about the number of free debug registers.  It is
somewhere on my TODO.

>   Thus handling them as normal registers,
> by changing individual values, can 
> lead to unconsitant states that can not be really passed
> to the DPMI provider.

If the access to debug registers is only through a well-defined API
(i.e., functions), then we can take care of this problem.

>   Moreover, I don't think that most people will care about 
> having debug registers listed on 'info all' command.
> Maybe 'info debug-register' would be a better solution here.

I'm not sure.  Most people don't care about FP registers, either, but
"info all" does print them.  We have "info registers" for those who
only want to see the ``normal'' registers.  IMHO, too many different
commands with overlapping functionality is not a Good Thing.

>   Remember also that the win32 API does not support 
> hardware debug registers

Really?  So you cannot have hardware-assisted debugging in Win32
applications?

> so adding them to the cygwin target would be very wrong

This could be solved in the same way as x86 targets which don't
support FP registers.

Re: About unified debug register handling for i386 CPU.

[Chris Faylor]

On Wed, Sep 06, 2000 at 02:08:10AM -0400, Eli Zaretskii wrote:
>>   Remember also that the win32 API does not support 
>> hardware debug registers
>
>Really?  So you cannot have hardware-assisted debugging in Win32
>applications?

I am not aware of any way to do this with the Win32 debugging API
but I'd be thrilled to be proved wrong.

cgf

Re: gdb doesn't work very well with dynamic linked binaries

[James Cownie]

This threads seems really to have morphed to be about using the
hardware watchpoint registers on IA32.

So :-

Note that there are a couple of linux kernel bugs in 2.2 (one of which
was fixed in 2.2.17) which affect the use of the debug registers.

The first bug (which was fixed in 2.2.17) was that the debug status
register wasn't being saved where it could be seen by ptrace, so if
you were relying on looking at the hardware status register you were
stuffed.

The second (which isn't yet fixed) I described with a test case in a
previous mail here. ( http://sources.redhat.com/ml/gdb/2000-08/msg00072.html )
Basically the kernel ignores all hardware
watchpoints from the point where one of them is triggered inside the
kernel until a reschedule. This means many watchpoints at user level
can be silently skipped :-(

AFAICS both bugs remain in 2.4.0-test7.

-- Jim 

James Cownie	<jcownie@etnus.com>
Etnus, LLC.     +44 117 9071438
http://www.etnus.com



-- 
-- Jim
James Cownie
jcownie@etnus.com
Etnus, Inc.
Phone +44 117 9071438

Re: gdb doesn't work very well with dynamic linked binaries

[Mark Kettenis]

James,

Thanks for reporting these issues to the GDB lists, and sorry for not
getting back to you earlier.

It's unfortunate that these bugs exist in the Linux kernel, especially
if Linux 2.2.17 contains a partial fix, but Linux 2.4 doesn't.
However, I don't think I can be much of a help here.  I'm not very
familiar with the relevant parts of the kernel so I don't feel I can
advocate your patch for you (which doesn't mean that it isn't right,
just that I don't know enough about it).

The only advice I can offer is: keep trying.  If you have a tested
patch for Linux 2.4, send it directly to Linus, and if the patch
doesn't appear in the next (test)-release, send it again.  Repeat this
until the patch has been integrated or has been rejected.

As for the unresolved issues: try sending a message to the
linux-kernel mailing list (linux-kernel@vger.kernel.org) again.  It
might be that the people interested in fixing the problems simply
missed it, or were too busy when you first sent it.

Mark

Re: gdb doesn't work very well with dynamic linked binaries

[James Cownie]

Mark Kettenis wrote :-
  I'm not sure whether the debug registers are
  per-thread or per-VM-space in Linux.  I'll probably need to look into
  the kernel source.

To save you the time, they are per-thread, just like all the 
other process' registers.

They are conceptually saved and restored on process scheduling
events (which for linuxthreads is the same thing as thread 
scheduling events, since linuxthreads _are_ processes as far as
the scheduler is concerned). 

The bug I mentioned previously is exactly that they're getting
cleared by the kernel and then not getting restored on return
to user space, leaving them wrong until the next reschedule :-(

-- 
-- Jim
James Cownie
jcownie@etnus.com
Etnus, Inc.
Phone +44 117 9071438

Re: gdb doesn't work very well with dynamic linked binaries

[Mark Kettenis]

   Date: Thu, 7 Sep 2000 04:55:38 -0400
   From: James Cownie <jcownie@etnus.com>

   Mark Kettenis wrote :-
     I'm not sure whether the debug registers are
     per-thread or per-VM-space in Linux.  I'll probably need to look into
     the kernel source.

   To save you the time, they are per-thread, just like all the 
   other process' registers.

Thanks, that was what I suspected from looking at the code.

   They are conceptually saved and restored on process scheduling
   events (which for linuxthreads is the same thing as thread 
   scheduling events, since linuxthreads _are_ processes as far as
   the scheduler is concerned).

Yep.  This means that getting HW watchpoints working for
multi-threaded processes is a bit difficult, since GDB expects them to
be process-wide.  So any HW watchpoints will have to be inserted in
*all* threads, not just one as we do now.

Eli, this probably means that adding the debugging registers to GDB's
register cache isn't a good idea.  Something more specialized is
needed, i.e. a native-dependent interface that updates the address and
control register in all threads.  This might implicate that keeping
the actual HW watchpoint support a native-only thing is a good idea.

   The bug I mentioned previously is exactly that they're getting
   cleared by the kernel and then not getting restored on return
   to user space, leaving them wrong until the next reschedule :-(

I think I understand the problems now.  It basically means that one
cannot reliably watch area's that are somehow used in system calls.

I suspect that Linux isn't the only kernel with this bug.  AFAICS
FreeBSD also simply disables any (user-space) watchpoints triggered
from within the kernel.  I don't know what the various x86 System V's
(Solaris, SCO Open Server 5) do, but I wouldn't be surprised if it is
broken there too.

Mark

Re: gdb doesn't work very well with dynamic linked binaries

[Eli Zaretskii]

>   Date: Thu, 7 Sep 2000 12:09:50 +0200 (MET DST)
>   From: Mark Kettenis <kettenis@wins.uva.nl>
>
>   Yep.  This means that getting HW watchpoints working for
>   multi-threaded processes is a bit difficult, since GDB expects them to
>   be process-wide.  So any HW watchpoints will have to be inserted in
>   *all* threads, not just one as we do now.
>
>   Eli, this probably means that adding the debugging registers to GDB's
>   register cache isn't a good idea.  Something more specialized is
>   needed, i.e. a native-dependent interface that updates the address and
>   control register in all threads.  This might implicate that keeping
>   the actual HW watchpoint support a native-only thing is a good idea.

Why ``native-dependent'' and not ``target-dependent''?  Won't the same
problem affect any multithreaded ia32 target?  Or am I missing
something?

In any case, would a special array of debug registers be an okay
solution?  The elements of that array will be set by
ia32_insert_watchpoint and ia32_remove_watchpoint (to be written), and
target-dependent subroutines which resume the inferior and get control
when the inferior is stopped will access that array to pass the
registers to the debuggee.

Re: gdb doesn't work very well with dynamic linked binaries

[Mark Kettenis]

   Date: Thu, 7 Sep 2000 11:00:32 -0400 (EDT)
   From: Eli Zaretskii <eliz@delorie.com>
   CC: jcownie@etnus.com, gdb@sources.redhat.com
   Content-Type: text
   Content-Length: 1217

   >   Date: Thu, 7 Sep 2000 12:09:50 +0200 (MET DST)
   >   From: Mark Kettenis <kettenis@wins.uva.nl>
   >
   >   Yep.  This means that getting HW watchpoints working for
   >   multi-threaded processes is a bit difficult, since GDB expects them to
   >   be process-wide.  So any HW watchpoints will have to be inserted in
   >   *all* threads, not just one as we do now.
   >
   >   Eli, this probably means that adding the debugging registers to GDB's
   >   register cache isn't a good idea.  Something more specialized is
   >   needed, i.e. a native-dependent interface that updates the address and
   >   control register in all threads.  This might implicate that keeping
   >   the actual HW watchpoint support a native-only thing is a good idea.

   Why ``native-dependent'' and not ``target-dependent''?  Won't the same
   problem affect any multithreaded ia32 target?  Or am I missing
   something?

Well, we already have a special packet in the remote protocol for
setting watchpoints, so in principle it isn't necessary to be
``target-dependant''.

The problems are likely to be different for the other multi-threaded
systems.  Linux is kinda weird in that respect.

   In any case, would a special array of debug registers be an okay
   solution?  The elements of that array will be set by
   ia32_insert_watchpoint and ia32_remove_watchpoint (to be written), and
   target-dependent subroutines which resume the inferior and get control
   when the inferior is stopped will access that array to pass the
   registers to the debuggee.

I think it is important to realize that the address and control
registers should be per-process, whereas the status register should be
per-thread.  On Linux, the code to actually pass the registers to the
debuggee should probably live in the (currently native-only)
threads-module.

Oh, and please use an i386_ prefix for the function names instead of
ia32_ for consistency :-).

Mark

Re: gdb doesn't work very well with dynamic linked binaries

[Peter.Schauer]

>    The bug I mentioned previously is exactly that they're getting
>    cleared by the kernel and then not getting restored on return
>    to user space, leaving them wrong until the next reschedule :-(
> 
> I think I understand the problems now.  It basically means that one
> cannot reliably watch area's that are somehow used in system calls.
> 
> I suspect that Linux isn't the only kernel with this bug.  AFAICS
> FreeBSD also simply disables any (user-space) watchpoints triggered
> from within the kernel.  I don't know what the various x86 System V's
> (Solaris, SCO Open Server 5) do, but I wouldn't be surprised if it is
> broken there too.

AFAIK Solaris never provided access to the x86 debug registers.
Starting with Solaris 2.6, support for hardware watchpoints is provided
via procfs (protect pages which contain watchpoint areas, examine address
on pagefault, cause breakpoint if address is within watchpoint area).

Perhaps this scheme (which allows for an arbitrary number of watchpoints)
could be adopted by the Linux kernel.

Debug register access might be valuable for embedded targets, but the
generic x86 target should not assume availability of debug register access.

-- 
Peter Schauer			pes@regent.e-technik.tu-muenchen.de

Re: gdb doesn't work very well with dynamic linked binaries

[James Cownie]

> I think I understand the problems now.  It basically means that one
> cannot reliably watch area's that are somehow used in system calls.

There are two slightly separate issues here 
1) If the kernel triggers a watchpoint do you get to see it
2) If the kernel triggers a watchpoint does that break 
   subsequent user level watchpoints.

The ideal behaviour, of course, would be that you saw the watchpoint
whether or not it was triggered by the kernel, and that didn't
break anything. (I believe this is what 2.0 kernels achieved).

The current (2.2.17) Linux behaviour is that 
1) The kernel triggered watchpoint is not reported to the debugger
*AND*
2) After the watchpoint is triggered by the kernel _all_ watchpoints
   in the thread are disabled until it gets rescheduled. (So some
   arbitrary number of user level watchpoint hits can be missed).

My patch would remove the second property, but would still leave
us ignorant about watchpoint hits from the kernel. 

So, with my kernel patch all watchpoint hits from user space are
reported, but watchpoint hits from kernel space are ignored. (Therefore
reading over a watched area doesn't get reported).

The ideal patch would also report kernel watchpoint hits back to the
process. Unfortunately I don't see any "free" way of doing that. 
(By which I mean a way which doesn't add code to the system call
return path whether or not debugging is enabled).

The current bug was introduced in the move from 2.0 to 2.2 because
the system call return path was optimised to remove the restoration
of the debug registers (which is fine as long as the kernel doesn't
change them, unfortunately it does).

> I suspect that Linux isn't the only kernel with this bug.  AFAICS
> FreeBSD also simply disables any (user-space) watchpoints triggered
> from within the kernel.  I don't know what the various x86 System V's
> (Solaris, SCO Open Server 5) do, but I wouldn't be surprised if it is
> broken there too.

It all depends on whether these OSes restore the debug registers in
the system call return path. If so changing them in the kernel is 
OK.

-- 
-- Jim
James Cownie
jcownie@etnus.com
Etnus, Inc.
Phone +44 117 9071438