GDB now takes 4 minutes to start up with remote gdbserver target

GDB now takes 4 minutes to start up with remote gdbserver target

[Sandra Loosemore]

I believe this is a consequence of these patch series:

https://sourceware.org/ml/gdb-patches/2015-03/msg00638.html
https://sourceware.org/ml/gdb-patches/2015-04/msg00015.html

$ nios2-linux-gnu-gdb a.out
[...]
(gdb) target remote altera-3c120-1:6789
Remote debugging using altera-3c120-1:6789
Reading symbols from 
target:/scratch/sandra/nios2-linux-trunk/install/opt/codesourcery/nios2-linux-gnu/libc/lib/ld-linux-nios2.so.1...done.
0x2aaa8cc4 in _start ()
    from 
target:/scratch/sandra/nios2-linux-trunk/install/opt/codesourcery/nios2-linux-gnu/libc/lib/ld-linux-nios2.so.1
(gdb) break main
Breakpoint 1 at 0x2760: file /home/sandra/examples/croak.c, line 10.
(gdb) shell date
Thu Jul 23 16:02:32 PDT 2015
(gdb) c
Continuing.
shell date

Breakpoint 1, main () at /home/sandra/examples/croak.c:10
10	  n = sizeof (s) / sizeof (const char *);
(gdb) shell date
Thu Jul 23 16:06:31 PDT 2015

Is this really appropriate and user-friendly default behavior?  Are 
users really expected to wait 4 minutes for GDB to start up whenever 
they debug a program on a remote target now?  :-(

It is definitely the vFile traffic that is slowing things down....  it 
comes back from the "c" command instantaneously if I disable those packets.

-Sandra the unhappy

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Sandra Loosemore]

On 07/23/2015 05:19 PM, Sandra Loosemore wrote:
>
> [snip]
>
> Is this really appropriate and user-friendly default behavior?  Are
> users really expected to wait 4 minutes for GDB to start up whenever
> they debug a program on a remote target now?  :-(
>
> It is definitely the vFile traffic that is slowing things down....  it
> comes back from the "c" command instantaneously if I disable those packets.

Some further observations:

(1) I don't see anything in the GDB manual to indicate that transferring 
files from the target back to the host is the default behavior for "set 
sysroot", nor do I see any kind of warning that this can be a very slow 
operation.

(2) I don't know how users are supposed to know that "set sysroot" is 
the source of this slowness, either.  There needs to be something in the 
section of the manual that discusses how to connect to a remote 
gdbserver target that tells you what to do if the default behavior is 
unacceptably slow.

(3) Once the "c" command is issued, there's nothing to inform the user 
exactly what GDB is doing or that this can be a very slow operation 
(e.g., with a progress bar).

(4) GDB doesn't respond to ^C during the 4 minutes it is doing whatever 
it's doing to transfer files.  It just sits there acting catatonic.

In absence of any other information, users are likely to come to the 
same conclusion I did -- that GDB and/or GDBserver are broken and just 
got wedged somewhere on startup.  I was knowledgeable enough about GDB 
internals to restart my session and do "set debug remote 1" prior to 
connecting so I could see the RSP traffic and get a clue where it was 
getting stuck; ordinary users would probably just give up in disgust, or 
complain to their toolchain vendor :-P that GDB is broken.

While I appreciate that this change may be useful in fixing a class of 
user problems, it is an incompatible change from past behavior and 
causes a whole different set of problems for users.  Can we please 
consider restoring the default for "set sysroot" to its previous behavior?

-Sandra

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Sandra Loosemore wrote:
> (1) I don't see anything in the GDB manual to indicate that
> transferring files from the target back to the host is the default
> behavior for "set sysroot", nor do I see any kind of warning that
> this can be a very slow operation.

The manual says this:

  If @var{path} starts with the sequence @file{target:} and the target
  system is remote then @value{GDBN} will retrieve the target binaries
  from the remote system.

It doesn't mention in the manual that this is the default, though it
is documented in NEWS:

  * Directory names supplied to the "set sysroot" commands may be
    prefixed with "target:" to tell GDB to access shared libraries from
    the target system, be it local or remote.  This replaces the prefix
    "remote:".  The default sysroot has been changed from "" to
    "target:".  "remote:" is automatically converted to "target:" for
    backward compatibility.

  * The system root specified by "set sysroot" will be prepended to the
    filename of the main executable (if reported to GDB as absolute by
    the operating system) when starting processes remotely, and when
    attaching to already-running local or remote processes.

  * GDB now supports automatic location and retrieval of executable
    files from remote targets.  Remote debugging can now be initiated
    using only a "target remote" or "target extended-remote" command
    (no "set sysroot" or "file" commands are required).  See "New remote
    packets" below.

> (2) I don't know how users are supposed to know that "set sysroot"
> is the source of this slowness, either.  There needs to be something
> in the section of the manual that discusses how to connect to a
> remote gdbserver target that tells you what to do if the default
> behavior is unacceptably slow.

Understood.

> (3) Once the "c" command is issued, there's nothing to inform the
> user exactly what GDB is doing or that this can be a very slow
> operation (e.g., with a progress bar).

This is kind of a shortcoming of GDB in general.  There was a similar
issue relating to tab-completion in programs with lots of symbols:

  https://sourceware.org/bugzilla/show_bug.cgi?id=11920

I don't have a good solution for this.

> (4) GDB doesn't respond to ^C during the 4 minutes it is doing
> whatever it's doing to transfer files.  It just sits there acting
> catatonic.

That's not great.

> In absence of any other information, users are likely to come to the
> same conclusion I did -- that GDB and/or GDBserver are broken and
> just got wedged somewhere on startup.  I was knowledgeable enough
> about GDB internals to restart my session and do "set debug remote
> 1" prior to connecting so I could see the RSP traffic and get a clue
> where it was getting stuck; ordinary users would probably just give
> up in disgust, or complain to their toolchain vendor :-P that GDB is
> broken.
> 
> While I appreciate that this change may be useful in fixing a class
> of user problems, it is an incompatible change from past behavior
> and causes a whole different set of problems for users.  Can we
> please consider restoring the default for "set sysroot" to its
> previous behavior?

A large part of the motivation for these patches was to automate as
much as possible so users did not have to tell GDB stuff it could
figure out itself.  Rather than reverting (the nuclear option!)
how about we see if we can make GDB handle this.

How were you debugging before these series went in?  Without symbols?
If you'd started GDB with "file" and "set sysroot" commands to set up
your environment the whole remote-fetch should not have happened.

I'll look into some combination of making the remote transfer
interruptable, and issuing a warning during slow transfers, to
see if something like that could work.  Could you update the
manual to add the information that you would have like to have
found there?

Thanks,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Sandra Loosemore]

On 07/24/2015 02:52 AM, Gary Benson wrote:
>
> A large part of the motivation for these patches was to automate as
> much as possible so users did not have to tell GDB stuff it could
> figure out itself.  Rather than reverting (the nuclear option!)
> how about we see if we can make GDB handle this.
>
> How were you debugging before these series went in?  Without symbols?
> If you'd started GDB with "file" and "set sysroot" commands to set up
> your environment the whole remote-fetch should not have happened.

I am passing the executable to GDB on the command line.  The executable 
is linked with explicit -Wl,-dynamic-linker= and -Wl,-rpath options 
pointing to a sysroot that is NFS-mounted on the remote target at the 
same pathname as on the host system.  (This is our normal automated test 
configuration here, BTW....  I was trying to investigate why mainline 
GDB testing now takes 12 hours versus 38 minutes for our last stable 
release, and quickly realized that GDB's current behavior is going to be 
totally unacceptable for interactive use as well.)

The user documentation we at Mentor distribute with our toolchains that 
explains sysroots says that you only need to do "set sysroot" in the 
debugger if you are interested in debugging shared libraries or 
multi-threaded applications, *and* the remote sysroot pathname is not 
valid on the host system.  Neither of these things applies to what I was 
trying to do.  Plus, generally speaking, if you fail to do "set sysroot" 
in older GDB versions, application debugging still works even if there 
is a mismatch in the sysroot pathnames between host and target....  you 
just get some messages now and then about how GDB couldn't find some 
shared library symbols, and suggesting that you try "set sysroot".

What I find particularly confusing is that continuing to main doesn't 
seem like a user action that requires information from the sysroot at 
all.  I have not set any breakpoints in shared libraries, I'm not trying 
to backtrace through a library call, and I haven't requested that GDB 
stop on solib events.  Why does GDB need to transfer these files from 
the target at all if the user is not interested in them?

> I'll look into some combination of making the remote transfer
> interruptable, and issuing a warning during slow transfers, to
> see if something like that could work.  Could you update the
> manual to add the information that you would have like to have
> found there?

I'd rather that we fix GDB to "just work", as it used to do, rather than 
have to document workarounds for this breakage.

-Sandra

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Paul_Koning]

> On Jul 24, 2015, at 9:57 AM, Sandra Loosemore <sandra@codesourcery.com> wrote:
> 
> On 07/24/2015 02:52 AM, Gary Benson wrote:
>> ...
> 
> I'd rather that we fix GDB to "just work", as it used to do, rather than have to document workarounds for this breakage.

I agree.  Taking multiple minutes to start up, by defaut, is a bug.

	paul

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Paul_Koning@dell.com wrote:
> Sandra Loosemore wrote:
> > I'd rather that we fix GDB to "just work", as it used to do,
> > rather than have to document workarounds for this breakage.
> 
> I agree.  Taking multiple minutes to start up, by defaut, is a bug.

I also agree that GDB suddenly taking very much longer to start up
is a bug.  I don't need convincing, we're all on the same page here.

I don't agree that the fix is reverting the "target:" sysroot
changes.  While that would make GDB "just work" for Sandra's
scenario, it would make GDB not "just work" for other scenarios.
Not everybody using remote debugging has a slow link.
Not everybody using "target:" sysroots is debugging remote targets.

The proper fix is making GDB recognise what's happening and adjust.
The information Sandra provided should allow me to put together
some kind of test environment.

In the meantime, Sandra, could you please check that the patch I
mailed makes the long file transfer operations interruptible.
Irrespective of how your main issue is solved, uninterruptible
long operations are a bug, and I'd like to fix this one.

Thanks,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Paul_Koning]

> On Jul 24, 2015, at 11:11 AM, Gary Benson <gbenson@redhat.com> wrote:
> 
> Paul_Koning@dell.com wrote:
>> Sandra Loosemore wrote:
>>> I'd rather that we fix GDB to "just work", as it used to do,
>>> rather than have to document workarounds for this breakage.
>> 
>> I agree.  Taking multiple minutes to start up, by defaut, is a bug.
> 
> I also agree that GDB suddenly taking very much longer to start up
> is a bug.  I don't need convincing, we're all on the same page here.
> 
> I don't agree that the fix is reverting the "target:" sysroot
> changes.  While that would make GDB "just work" for Sandra's
> scenario, it would make GDB not "just work" for other scenarios.

But having sysroot default to target is also a bad idea for lots of other people.  Consider embedded systems: you presumably have stripped images there, but unstripped ones on your build host.  Flipping the default sysroot to target means that suddenly you don’t see debug symbols any more.  I don’t see how that can be a good thing.

Also, remote file access is always slower.  Especially when using a protocol such as the GDB remote protocol that is a single operation request/response protocol designed only for simplicity, not at all for performance.

	paul

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Pedro Alves]

On 07/24/2015 04:27 PM, Paul_Koning@Dell.com wrote:

> But having sysroot default to target is also a bad idea for lots of other people.  Consider embedded systems: you presumably have stripped images there, but unstripped ones on your build host.

But in that scenario, with the old default sysroot, how was gdb finding
the binaries on the build host?  The binaries on the equilalent locations
on the host's root will certainly not match the embedded/target system's.
In that scenario, you must have been pointing the "set sysroot" somewhere
local?  And if you do that, nothing changes in 7.10, gdb will still access
the files on the local filesystem.

From the discussion so far, it seems that the only case that ends up
regressing is the case where the host and target share both the
filesystem, and the host/target paths match.  I don't know off hand how to
make gdb aware of that automatically.

That seems like enough of a special case that could well be handled
by an explicit "set sysroot /" in e.g., the toolchain's system-gdbinit, or
by building gdb with "--with-sysroot=/".

Thanks,
Pedro Alves

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Paul_Koning]

> On Jul 24, 2015, at 12:05 PM, Pedro Alves <palves@redhat.com> wrote:
> 
> On 07/24/2015 04:27 PM, Paul_Koning@Dell.com wrote:
> 
>> But having sysroot default to target is also a bad idea for lots of other people.  Consider embedded systems: you presumably have stripped images there, but unstripped ones on your build host.
> 
> But in that scenario, with the old default sysroot, how was gdb finding
> the binaries on the build host?  The binaries on the equilalent locations
> on the host's root will certainly not match the embedded/target system's.
> In that scenario, you must have been pointing the "set sysroot" somewhere
> local?  And if you do that, nothing changes in 7.10, gdb will still access
> the files on the local filesystem.
> 
> From the discussion so far, it seems that the only case that ends up
> regressing is the case where the host and target share both the
> filesystem, and the host/target paths match.  I don't know off hand how to
> make gdb aware of that automatically.
> 
> That seems like enough of a special case that could well be handled
> by an explicit "set sysroot /" in e.g., the toolchain's system-gdbinit, or
> by building gdb with "--with-sysroot=/“.

If you’re doing cross-builds, then yes, you’d have a non-default sysroot.  But if the host and target are the same OS, but the target has a small local file system with stripped images on it, then the default sysroot was valid in the past, but the new default is not.

	paul

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Pedro Alves]

On 07/24/2015 05:58 PM, Paul_Koning@Dell.com wrote:
> 
>> On Jul 24, 2015, at 12:05 PM, Pedro Alves <palves@redhat.com> wrote:
>>
>> On 07/24/2015 04:27 PM, Paul_Koning@Dell.com wrote:
>>
>>> But having sysroot default to target is also a bad idea for lots of other people.  Consider embedded systems: you presumably have stripped images there, but unstripped ones on your build host.
>>
>> But in that scenario, with the old default sysroot, how was gdb finding
>> the binaries on the build host?  The binaries on the equilalent locations
>> on the host's root will certainly not match the embedded/target system's.
>> In that scenario, you must have been pointing the "set sysroot" somewhere
>> local?  And if you do that, nothing changes in 7.10, gdb will still access
>> the files on the local filesystem.
>>
>> From the discussion so far, it seems that the only case that ends up
>> regressing is the case where the host and target share both the
>> filesystem, and the host/target paths match.  I don't know off hand how to
>> make gdb aware of that automatically.
>>
>> That seems like enough of a special case that could well be handled
>> by an explicit "set sysroot /" in e.g., the toolchain's system-gdbinit, or
>> by building gdb with "--with-sysroot=/“.
> 
> If you’re doing cross-builds, then yes, you’d have a non-default sysroot.  But if the host and target are the same OS, but the target has a small local file system with stripped images on it, then the default sysroot was valid in the past, but the new default is not.

And in case the host and target are the same OS, but the binaries are
of different versions (which happens frequently, machines
don't usually update their packages at exactly the same time), then
the default is almost right, but then the user gets misleading backtraces.

It seems to me like if you take the trouble to build such a setup
as you suggest, putting a "set sysroot /" somewhere is just another
small setup step.

The idea of the new default was both what's more convenient in the wider
set of use cases and also picking correctness over efficiency, while still
letting the user point at local copies of files (or set sysroot /dev/null
to point nowhere, for e.g.) to get efficiency back, when possible.

The discoverability of the need to do the latter I believe should be
treated as a separate matter, which seems could be resolved with a warning
around "warning: fetching files from the target.  Use "set sysroot" to
point at local copies for faster access." or some such, as has already
been suggested elsewhere.  IOW, if we had that already, how bad would it
really be if target: remains the default?

Thanks,
Pedro Alves

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Sandra Loosemore]

On 07/24/2015 10:05 AM, Pedro Alves wrote:
> On 07/24/2015 04:27 PM, Paul_Koning@Dell.com wrote:
>
>> But having sysroot default to target is also a bad idea for lots of other people.  Consider embedded systems: you presumably have stripped images there, but unstripped ones on your build host.
>
> But in that scenario, with the old default sysroot, how was gdb finding
> the binaries on the build host?  The binaries on the equilalent locations
> on the host's root will certainly not match the embedded/target system's.
> In that scenario, you must have been pointing the "set sysroot" somewhere
> local?  And if you do that, nothing changes in 7.10, gdb will still access
> the files on the local filesystem.
>
>  From the discussion so far, it seems that the only case that ends up
> regressing is the case where the host and target share both the
> filesystem, and the host/target paths match.  I don't know off hand how to
> make gdb aware of that automatically.

There's also the case where the host and target sysroot locations do not 
match at all.  As I said, this used to work reasonably well for 
application debugging, where the user isn't interested in debugging 
shared libraries and doesn't care if the shared library symbol 
information isn't available to GDB.  It used to print a helpful message 
suggesting using "set sysroot" if the user wants the shared library 
information, instead of hanging on startup with no indication of what 
the trouble is or how to fix it.  I can't see the new default behavior 
as an improvement over the old.

> That seems like enough of a special case that could well be handled
> by an explicit "set sysroot /" in e.g., the toolchain's system-gdbinit, or
> by building gdb with "--with-sysroot=/".

There are a bunch of possible workarounds for this, but why can't we 
make GDB "just work" by default, as it used to, instead of requiring 
users to build GDB differently or install a workaround or issue extra 
commands manually that they didn't used to need at all?

-Sandra

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Sandra Loosemore wrote:
> On 07/24/2015 10:05 AM, Pedro Alves wrote:
> > On 07/24/2015 04:27 PM, Paul_Koning@Dell.com wrote:
> > > But having sysroot default to target is also a bad idea for lots
> > > of other people.  Consider embedded systems: you presumably have
> > > stripped images there, but unstripped ones on your build host.
> > 
> > But in that scenario, with the old default sysroot, how was gdb
> > finding the binaries on the build host?  The binaries on the
> > equilalent locations on the host's root will certainly not match
> > the embedded/target system's.  In that scenario, you must have
> > been pointing the "set sysroot" somewhere local?  And if you do
> > that, nothing changes in 7.10, gdb will still access the files on
> > the local filesystem.
> > 
> > From the discussion so far, it seems that the only case that ends
> > up regressing is the case where the host and target share both the
> > filesystem, and the host/target paths match.  I don't know off
> > hand how to make gdb aware of that automatically.
> 
> There's also the case where the host and target sysroot locations do
> not match at all.  As I said, this used to work reasonably well for
> application debugging, where the user isn't interested in debugging
> shared libraries and doesn't care if the shared library symbol
> information isn't available to GDB.  It used to print a helpful
> message suggesting using "set sysroot" if the user wants the shared
> library information, instead of hanging on startup with no
> indication of what the trouble is or how to fix it.  I can't see the
> new default behavior as an improvement over the old.
> 
> > That seems like enough of a special case that could well be
> > handled by an explicit "set sysroot /" in e.g., the toolchain's
> > system-gdbinit, or by building gdb with "--with-sysroot=/".
> 
> There are a bunch of possible workarounds for this, but why can't we
> make GDB "just work" by default, as it used to, instead of requiring
> users to build GDB differently or install a workaround or issue
> extra commands manually that they didn't used to need at all?

I have an idea for a solution to this.  I should know in a few hours
if it can work.  Tomorrow morning at the very latest.

Thanks,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Gary Benson wrote:
> Sandra Loosemore wrote:
> > On 07/24/2015 10:05 AM, Pedro Alves wrote:
> > > On 07/24/2015 04:27 PM, Paul_Koning@Dell.com wrote:
> > > > But having sysroot default to target is also a bad idea for lots
> > > > of other people.  Consider embedded systems: you presumably have
> > > > stripped images there, but unstripped ones on your build host.
> > > 
> > > But in that scenario, with the old default sysroot, how was gdb
> > > finding the binaries on the build host?  The binaries on the
> > > equilalent locations on the host's root will certainly not match
> > > the embedded/target system's.  In that scenario, you must have
> > > been pointing the "set sysroot" somewhere local?  And if you do
> > > that, nothing changes in 7.10, gdb will still access the files on
> > > the local filesystem.
> > > 
> > > From the discussion so far, it seems that the only case that ends
> > > up regressing is the case where the host and target share both the
> > > filesystem, and the host/target paths match.  I don't know off
> > > hand how to make gdb aware of that automatically.
> > 
> > There's also the case where the host and target sysroot locations do
> > not match at all.  As I said, this used to work reasonably well for
> > application debugging, where the user isn't interested in debugging
> > shared libraries and doesn't care if the shared library symbol
> > information isn't available to GDB.  It used to print a helpful
> > message suggesting using "set sysroot" if the user wants the shared
> > library information, instead of hanging on startup with no
> > indication of what the trouble is or how to fix it.  I can't see the
> > new default behavior as an improvement over the old.
> > 
> > > That seems like enough of a special case that could well be
> > > handled by an explicit "set sysroot /" in e.g., the toolchain's
> > > system-gdbinit, or by building gdb with "--with-sysroot=/".
> > 
> > There are a bunch of possible workarounds for this, but why can't we
> > make GDB "just work" by default, as it used to, instead of requiring
> > users to build GDB differently or install a workaround or issue
> > extra commands manually that they didn't used to need at all?
> 
> I have an idea for a solution to this.  I should know in a few hours
> if it can work.  Tomorrow morning at the very latest.

Ok, here goes...

 * From a user's perspective GDB is magically prefixing *some*
   executable and shared library filenames with "target:".

 * From a developer's perspective this magic prefixing is implemented
   by having the string "target:" as the default sysroot.

My proposal is to make the default sysroot be "" again, and add the
prefix in solib_find_1 if certain conditions are met, specifically:

 * Executable filenames get prefixed with "target:" iff:
     Automatic "target:" prefixing is enabled
     AND gdb_sysroot is ""
     AND the filesystem is nonlocal

 * Shared library filenames get prefixed with "target:" iff:
     Automatic "target:" prefixing is enabled
     AND gdb_sysroot is ""
     AND the filesystem is nonlocal
     AND exec_filename starts with "target:"

There's a new boolean here, "set auto-target-prefix on|off", which
is enabled by default.

So:

 gdb; target remote :9999 -> "target:" prefix applied
 gdb FILE; target remote :9999 -> no "target:" prefix

The latter is how Sandra is invoking GDB.  Also:
 
 gdb -n PID, and
 gdb; attach PID -> "target:" prefix applied iff FS is nonlocal

Aside from the fact that this should fix Sandra's use case without
breaking any I care about, I like that users doing non-remote, non-
container debugging will not see "target:" prefixes onscreen unless
they're actually necessary.

Comments?

Cheers,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Sandra Loosemore]

On 07/28/2015 03:25 AM, Gary Benson wrote:
>
> Ok, here goes...
>
>   * From a user's perspective GDB is magically prefixing *some*
>     executable and shared library filenames with "target:".
>
>   * From a developer's perspective this magic prefixing is implemented
>     by having the string "target:" as the default sysroot.
>
> My proposal is to make the default sysroot be "" again, and add the
> prefix in solib_find_1 if certain conditions are met, specifically:
>
>   * Executable filenames get prefixed with "target:" iff:
>       Automatic "target:" prefixing is enabled
>       AND gdb_sysroot is ""
>       AND the filesystem is nonlocal
>
>   * Shared library filenames get prefixed with "target:" iff:
>       Automatic "target:" prefixing is enabled
>       AND gdb_sysroot is ""
>       AND the filesystem is nonlocal
>       AND exec_filename starts with "target:"

Can you explain what "the filesystem is nonlocal" means?  Which 
filesystem?  Nonlocal relative to the host or the target?  And what 
qualifies as nonlocal -- e.g. are NFS or SSHFS mounts nonlocal?

-Sandra the confused

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Sandra Loosemore wrote:
> On 07/28/2015 03:25 AM, Gary Benson wrote:
> > Ok, here goes...
> > 
> >   * From a user's perspective GDB is magically prefixing *some*
> >     executable and shared library filenames with "target:".
> > 
> >   * From a developer's perspective this magic prefixing is implemented
> >     by having the string "target:" as the default sysroot.
> > 
> > My proposal is to make the default sysroot be "" again, and add the
> > prefix in solib_find_1 if certain conditions are met, specifically:
> > 
> >   * Executable filenames get prefixed with "target:" iff:
> >       Automatic "target:" prefixing is enabled
> >       AND gdb_sysroot is ""
> >       AND the filesystem is nonlocal
> > 
> >   * Shared library filenames get prefixed with "target:" iff:
> >       Automatic "target:" prefixing is enabled
> >       AND gdb_sysroot is ""
> >       AND the filesystem is nonlocal
> >       AND exec_filename starts with "target:"
> 
> Can you explain what "the filesystem is nonlocal" means?  Which
> filesystem?  Nonlocal relative to the host or the target?  And what
> qualifies as nonlocal -- e.g. are NFS or SSHFS mounts nonlocal?

It refers to the method GDB must use to access inferior files.
If GDB can access them using standard open/read/pread/write/unlink
calls then the (inferior's) filesystem is local (to GDB).
So NFS or SSHFS mounts are local in this sense.

Remote targets have nonlocal filesystems because GDB must access
them using special calls.

Inferiors running in containers also have nonlocal filesystems
because, even though the code is running on the same system, the
inferior has a different filesystem.  open("/usr/bin/python") in
the inferior will get a different file to open("/usr/bin/python")
in GDB.

Cheers,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Gary Benson wrote:
> Ok, here goes...
> 
>  * From a user's perspective GDB is magically prefixing *some*
>    executable and shared library filenames with "target:".
> 
>  * From a developer's perspective this magic prefixing is implemented
>    by having the string "target:" as the default sysroot.
> 
> My proposal is to make the default sysroot be "" again, and add the
> prefix in solib_find_1 if certain conditions are met, specifically:
> 
>  * Executable filenames get prefixed with "target:" iff:
>      Automatic "target:" prefixing is enabled
>      AND gdb_sysroot is ""
>      AND the filesystem is nonlocal
> 
>  * Shared library filenames get prefixed with "target:" iff:
>      Automatic "target:" prefixing is enabled
>      AND gdb_sysroot is ""
>      AND the filesystem is nonlocal
>      AND exec_filename starts with "target:"
> 
> There's a new boolean here, "set auto-target-prefix on|off", which
> is enabled by default.
> 
> So:
> 
>  gdb; target remote :9999 -> "target:" prefix applied
>  gdb FILE; target remote :9999 -> no "target:" prefix
> 
> The latter is how Sandra is invoking GDB.  Also:
>  
>  gdb -n PID, and
>  gdb; attach PID -> "target:" prefix applied iff FS is nonlocal
> 
> Aside from the fact that this should fix Sandra's use case without
> breaking any I care about, I like that users doing non-remote, non-
> container debugging will not see "target:" prefixes onscreen unless
> they're actually necessary.

I've submitted a series of the work I've done on this:
https://sourceware.org/ml/gdb-patches/2015-07/msg00828.html

Please reply to that thread instead of this.

Thanks,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Doug Evans]

On Tue, Jul 28, 2015 at 2:25 AM, Gary Benson <gbenson@redhat.com> wrote:
> So:
>
>  gdb; target remote :9999 -> "target:" prefix applied
>  gdb FILE; target remote :9999 -> no "target:" prefix

That's seems a bit subtle.
Have to check the patches.

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Pedro Alves]

On 07/24/2015 06:18 PM, Sandra Loosemore wrote:
> On 07/24/2015 10:05 AM, Pedro Alves wrote:
>> On 07/24/2015 04:27 PM, Paul_Koning@Dell.com wrote:
>>
>>> But having sysroot default to target is also a bad idea for lots of other people.  Consider embedded systems: you presumably have stripped images there, but unstripped ones on your build host.
>>
>> But in that scenario, with the old default sysroot, how was gdb finding
>> the binaries on the build host?  The binaries on the equilalent locations
>> on the host's root will certainly not match the embedded/target system's.
>> In that scenario, you must have been pointing the "set sysroot" somewhere
>> local?  And if you do that, nothing changes in 7.10, gdb will still access
>> the files on the local filesystem.
>>
>>  From the discussion so far, it seems that the only case that ends up
>> regressing is the case where the host and target share both the
>> filesystem, and the host/target paths match.  I don't know off hand how to
>> make gdb aware of that automatically.
> 
> There's also the case where the host and target sysroot locations do not 
> match at all.  As I said, this used to work reasonably well for 
> application debugging, where the user isn't interested in debugging 
> shared libraries and doesn't care if the shared library symbol 
> information isn't available to GDB.  

Doesn't care, until she wants to backtrace across code in the shared
library, e.g., across qsort, or until she starts wondering what are
all those ???'s in the backtrace.

> It used to print a helpful message 
> suggesting using "set sysroot" if the user wants the shared library 
> information, instead of hanging on startup with no indication of what 
> the trouble is or how to fix it.  I can't see the new default behavior 
> as an improvement over the old.

I fully agree that those should be fixed.  If GDB had warned giving a
"set sysroot" suggestion, and the slow retrieval was cancelable, would we
be discussing the new default?

> 
>> That seems like enough of a special case that could well be handled
>> by an explicit "set sysroot /" in e.g., the toolchain's system-gdbinit, or
>> by building gdb with "--with-sysroot=/".
> 
> There are a bunch of possible workarounds for this, but why can't we 
> make GDB "just work" by default, as it used to, instead of requiring 
> users to build GDB differently or install a workaround or issue extra 
> commands manually that they didn't used to need at all?

Over the years we kept hearing the same in the other direction -- 'why
do we have to point at copies of local files, or explicitly
do "set sysroot remote:"?  Why isn't "remote:" the default sysroot?
Why doesn't GDB "just work" by default?'

The definition of "just work" here depends on perspective.  We're
just looking at this from different angles.

Thanks,
Pedro Alves

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Sandra Loosemore]

On 07/28/2015 04:13 PM, Pedro Alves wrote:
> On 07/24/2015 06:18 PM, Sandra Loosemore wrote:
>
>> It used to print a helpful message
>> suggesting using "set sysroot" if the user wants the shared library
>> information, instead of hanging on startup with no indication of what
>> the trouble is or how to fix it.  I can't see the new default behavior
>> as an improvement over the old.
>
> I fully agree that those should be fixed.  If GDB had warned giving a
> "set sysroot" suggestion, and the slow retrieval was cancelable, would we
> be discussing the new default?
>

I think that would be about 1000 times better than the current 
situation.  One mild concern I have is that if "set sysroot" with no 
arguments is the recommended way to restore the old behavior, IIUC old 
versions of GDB give an error on that form of the command.  So, if you 
have startup scripts, IDE-based launchers, documentation, etc that need 
to support multiple versions of GDB, is there a backward-compatible 
command that works everywhere to get the old behavior, if that's what 
you want?  Perhaps the GDB manual should explicitly document that 
instead of/in addition to plain "set sysroot".

I'd also like to see the manual be more explicit that transferring files 
from the target can be slow and it is recommended to make a copy 
available on the host.  And, I think the language currently in the 
manual that libraries on the target can be stripped as long as libraries 
on the host aren't needs to be tweaked, since this makes no sense if the 
default is to get the potentially-stripped libraries from the target anyway.

-Sandra

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Frank Ch. Eigler]

palves wrote:

> [...]
> But in that scenario, with the old default sysroot, how was gdb finding
> the binaries on the build host?  The binaries on the equilalent locations
> on the host's root will certainly not match the embedded/target system's.
> [...]
> regressing is the case where the host and target share both the
> filesystem, and the host/target paths match.  I don't know off hand how to
> make gdb aware of that automatically.
> [...]

GDB could grow a qCRC type packet for the remote file server, so as to let
it compare the remote files to the local ones.  (Or even qRSYNC :-)

- FChE

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Jan Kratochvil]

On Sun, 26 Jul 2015 22:05:00 +0200, Frank Ch. Eigler wrote:
> GDB could grow a qCRC type packet for the remote file server, so as to let
> it compare the remote files to the local ones.  (Or even qRSYNC :-)

build-id is there for that.


Jan

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Frank Ch. Eigler]

Hi -

> > GDB could grow a qCRC type packet for the remote file server, so as to let
> > it compare the remote files to the local ones.  (Or even qRSYNC :-)
> 
> build-id is there for that.

Yes, esp. if it easily extracted from the remote-target filesystem.

- FChE

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Doug Evans]

On Fri, Jul 24, 2015 at 1:52 AM, Gary Benson <gbenson@redhat.com> wrote:
>> (3) Once the "c" command is issued, there's nothing to inform the
>> user exactly what GDB is doing or that this can be a very slow
>> operation (e.g., with a progress bar).
>
> This is kind of a shortcoming of GDB in general.  There was a similar
> issue relating to tab-completion in programs with lots of symbols:
>
>   https://sourceware.org/bugzilla/show_bug.cgi?id=11920
>
> I don't have a good solution for this.

I'm sure there are fine solutions.
The problem is getting gdb to a point where
good solutions fit in easily, without having to
do something specific for each case.

>> While I appreciate that this change may be useful in fixing a class
>> of user problems, it is an incompatible change from past behavior
>> and causes a whole different set of problems for users.  Can we
>> please consider restoring the default for "set sysroot" to its
>> previous behavior?
>
> A large part of the motivation for these patches was to automate as
> much as possible so users did not have to tell GDB stuff it could
> figure out itself.  Rather than reverting (the nuclear option!)
> how about we see if we can make GDB handle this.

Being one of my pet peeves, I'm always on the lookout for examples,
hoping to raise awareness.
Is this another case of gdb trying to be "clever"
with no workaround when it's not what one wants?

ref: https://sourceware.org/ml/gdb-patches/2015-07/msg00767.html

>
> How were you debugging before these series went in?  Without symbols?
> If you'd started GDB with "file" and "set sysroot" commands to set up
> your environment the whole remote-fetch should not have happened.
>
> I'll look into some combination of making the remote transfer
> interruptable, and issuing a warning during slow transfers, to
> see if something like that could work.  Could you update the
> manual to add the information that you would have like to have
> found there?

I think just making things interruptable is insufficient.
We need to make it easy and obvious how to just not start these
transfers at all.

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Pedro Alves]

On 07/28/2015 05:54 PM, Doug Evans wrote:
> On Fri, Jul 24, 2015 at 1:52 AM, Gary Benson <gbenson@redhat.com> wrote:
>>> (3) Once the "c" command is issued, there's nothing to inform the
>>> user exactly what GDB is doing or that this can be a very slow
>>> operation (e.g., with a progress bar).
>>
>> This is kind of a shortcoming of GDB in general.  There was a similar
>> issue relating to tab-completion in programs with lots of symbols:
>>
>>   https://sourceware.org/bugzilla/show_bug.cgi?id=11920
>>
>> I don't have a good solution for this.
> 
> I'm sure there are fine solutions.
> The problem is getting gdb to a point where
> good solutions fit in easily, without having to
> do something specific for each case.

I agree.  I worry much about lots of "smartness" at
the last minute, and then be stuck with it.  The "target:" sysroot is
simple to explain and reason about.  The new proposal, not so much.
Also, with Aleksandar/Jan's gdbserver build-id validation series
in place, we may be able to come up with a different/better solution.

If resolving the interruptability and adding a suggestive warning
is deemed insufficient resolution (though I think we should try it
first), then I think it's too late to add too much magic, and we should
change the default sysroot back to "" by default.  Users can still then
put "set sysroot target:" in .gdbinit with 7.10, and we can continue
addressing identified issues until "target:" (or something around it)
can be made the default, on master.

(Another idea to build on top of this all to minimize downloads is
to have a gdb-side cache of remote file chunks, that persists
across gdb invocations.)

> 
>>> While I appreciate that this change may be useful in fixing a class
>>> of user problems, it is an incompatible change from past behavior
>>> and causes a whole different set of problems for users.  Can we
>>> please consider restoring the default for "set sysroot" to its
>>> previous behavior?
>>
>> A large part of the motivation for these patches was to automate as
>> much as possible so users did not have to tell GDB stuff it could
>> figure out itself.  Rather than reverting (the nuclear option!)
>> how about we see if we can make GDB handle this.
> 
> Being one of my pet peeves, I'm always on the lookout for examples,
> hoping to raise awareness.
> Is this another case of gdb trying to be "clever"
> with no workaround when it's not what one wants?
> 
> ref: https://sourceware.org/ml/gdb-patches/2015-07/msg00767.html

I agree.

> 
>>
>> How were you debugging before these series went in?  Without symbols?
>> If you'd started GDB with "file" and "set sysroot" commands to set up
>> your environment the whole remote-fetch should not have happened.
>>
>> I'll look into some combination of making the remote transfer
>> interruptable, and issuing a warning during slow transfers, to
>> see if something like that could work.  Could you update the
>> manual to add the information that you would have like to have
>> found there?
> 
> I think just making things interruptable is insufficient.
> We need to make it easy and obvious how to just not start these
> transfers at all.

Agreed.

Thanks,
Pedro Alves

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Pedro Alves wrote:
> On 07/28/2015 05:54 PM, Doug Evans wrote:
> > On Fri, Jul 24, 2015 at 1:52 AM, Gary Benson <gbenson@redhat.com> wrote:
> >>> (3) Once the "c" command is issued, there's nothing to inform the
> >>> user exactly what GDB is doing or that this can be a very slow
> >>> operation (e.g., with a progress bar).
> >>
> >> This is kind of a shortcoming of GDB in general.  There was a similar
> >> issue relating to tab-completion in programs with lots of symbols:
> >>
> >>   https://sourceware.org/bugzilla/show_bug.cgi?id=11920
> >>
> >> I don't have a good solution for this.
> > 
> > I'm sure there are fine solutions.
> > The problem is getting gdb to a point where
> > good solutions fit in easily, without having to
> > do something specific for each case.
> 

> I agree.  I worry much about lots of "smartness" at the last minute,
> and then be stuck with it.  The "target:" sysroot is simple to
> explain and reason about.  The new proposal, not so much.

Agreed.

I'm in no way pushing the series I posted, I made it more to turn
what was previously a thought experiment into something more concrete.
I should maybe have posted it as an RFC...

> Also, with Aleksandar/Jan's gdbserver build-id validation series in
> place, we may be able to come up with a different/better solution.

Yes.  Jan told me he was considering changing sysroot to a multiple-
component path, with the default being something like "/:target:/".
This should work really nicely, but it only makes sense with build-id
validation (it's just too easy to get the wrong files otherwise).
I'm assuming we can cope with the fact that the separator ":" appears
in "target:" somehow :)

> If resolving the interruptability and adding a suggestive warning
> is deemed insufficient resolution (though I think we should try it
> first), then I think it's too late to add too much magic, and we should
> change the default sysroot back to "" by default.  Users can still then
> put "set sysroot target:" in .gdbinit with 7.10, and we can continue
> addressing identified issues until "target:" (or something around it)
> can be made the default, on master.

I'm going to look at adding a warning.

Note that reverting the sysroot back to "" is not exactly zero-risk
given that GDB is now more aggressive in discovering executable
filenames from remote targets.

Cheers,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Jan Kratochvil]

On Wed, 29 Jul 2015 12:39:48 +0200, Gary Benson wrote:
> Yes.  Jan told me he was considering changing sysroot to a multiple-
> component path, with the default being something like "/:target:/".
> This should work really nicely, but it only makes sense with build-id
> validation (it's just too easy to get the wrong files otherwise).

> I'm assuming we can cope with the fact that the separator ":" appears
> in "target:" somehow :)

Yes, if a parsed component is "target" it is merged back with the next one.
	https://sourceware.org/git/?p=archer.git;a=commitdiff;h=972af46ddba59d32ba903da5bd736bc6239a1958#patch2


> > If resolving the interruptability and adding a suggestive warning
> > is deemed insufficient resolution (though I think we should try it
> > first), then I think it's too late to add too much magic, and we should
> > change the default sysroot back to "" by default.  Users can still then
> > put "set sysroot target:" in .gdbinit with 7.10, and we can continue
> > addressing identified issues until "target:" (or something around it)
> > can be made the default, on master.
> 
> I'm going to look at adding a warning.

There should be also some 'set' possibility to suppress that warning if one
knows what s/he is doing.


Jan

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Doug Evans wrote:
> On Fri, Jul 24, 2015 at 1:52 AM, Gary Benson <gbenson@redhat.com> wrote:
> > A large part of the motivation for these patches was to automate
> > as much as possible so users did not have to tell GDB stuff it
> > could figure out itself.  Rather than reverting (the nuclear
> > option!)  how about we see if we can make GDB handle this.
> 
> Being one of my pet peeves, I'm always on the lookout for examples,
> hoping to raise awareness.  Is this another case of gdb trying to be
> "clever" with no workaround when it's not what one wants?

No.  There is a workaround, you issue the command "set sysroot"
(with no arguments) and the sysroot gets set to the empty string,
the previous default.

The very first thing I did when considering changing the default
sysroot was to check that the previous default could be restored
by the user.  As it happened it wasn't possible, so I fixed it so
it was (see commit 811a659a779fdf93293fe1105d99e9db171a8b68).

Thanks,
Gary

-- 
http://gbenson.net/

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Gary Benson]

Sandra Loosemore wrote:
> (4) GDB doesn't respond to ^C during the 4 minutes it is doing
> whatever it's doing to transfer files.  It just sits there acting
> catatonic.

Sandra, could you tell me if this patch makes GDB interruptible for
you?

Thanks,
Gary

-- 
diff --git a/gdb/gdb_bfd.c b/gdb/gdb_bfd.c
index 1781d80..15ea03b 100644
--- a/gdb/gdb_bfd.c
+++ b/gdb/gdb_bfd.c
@@ -252,6 +252,8 @@ gdb_bfd_iovec_fileio_pread (struct bfd *abfd, void *stream, void *buf,
   pos = 0;
   while (nbytes > pos)
     {
+      QUIT;
+
       bytes = target_fileio_pread (fd, (gdb_byte *) buf + pos,
 				   nbytes - pos, offset + pos,
 				   &target_errno);

Re: GDB now takes 4 minutes to start up with remote gdbserver target

[Sandra Loosemore]

On 07/24/2015 04:34 AM, Gary Benson wrote:
> Sandra Loosemore wrote:
>> (4) GDB doesn't respond to ^C during the 4 minutes it is doing
>> whatever it's doing to transfer files.  It just sits there acting
>> catatonic.
>
> Sandra, could you tell me if this patch makes GDB interruptible for
> you?

No, it is not doing anything to help.  It only notices the ^C when it 
finishes whatever it is doing and reaches the breakpoint at main, same 
as before.

Also, what kind of state would you expect GDB to be left in, if it could 
be interrupted?  It might be somewhat reassuring to users to get back to 
a gdb prompt so they know it's still alive, but if there's nothing to 
indicate what was taking so long and any attempt to continue debugging 
the program results in another long delay while the file transfer is 
retried, that wouldn't be helpful.

-Sandra