[RFC/PATCH] Testsuite: set sysroot when using gdbserver

[RFC/PATCH] Testsuite: set sysroot when using gdbserver

[Alan Hayward]

When testing using native-gdbserver and native-extended-gdbserver, the sysroot
is not set.  This results in a warning from GDB and files are sent via the
remote protocol, which can be slow.

On Ubuntu 18.04 (unlike most distros) the debug versions of the standard
libraries are included by default in /usr/lib/debug/

Example log of a test on Ubuntu 18.04 AArch64:

Listening on port 2346
target remote localhost:2346
Remote debugging using localhost:2346
Reading /lib/ld-linux-aarch64.so.1 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /lib/ld-linux-aarch64.so.1 from remote target...
Reading symbols from target:/lib/ld-linux-aarch64.so.1...
Reading /lib/ld-2.27.so from remote target...
Reading /lib/.debug/ld-2.27.so from remote target...
Reading /usr/lib/debug//lib/ld-2.27.so from remote target...
Reading /usr/lib/debug/lib//ld-2.27.so from remote target...
Reading target:/usr/lib/debug/lib//ld-2.27.so from remote target...
(No debugging symbols found in target:/lib/ld-linux-aarch64.so.1)
0x0000ffffbf6d31c0 in ?? () from target:/lib/ld-linux-aarch64.so.1
(gdb) continue
Continuing.
Reading /lib/aarch64-linux-gnu/libc.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libc-2.27.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libc-2.27.so from remote target...
Reading /usr/lib/debug//lib/aarch64-linux-gnu/libc-2.27.so from remote target...
Reading /usr/lib/debug//lib/aarch64-linux-gnu/libc-2.27.so from remote target...

These file reads are causing a complete native-gdbserver run on the AArch64
buildbot slave to timeout after 2.5 hours.  This is also causing the builds
to back up on the slave.

The solution is to ensure the sysroot is set to / immediately after connecting.

This drastically reduces the time of a test. For example, gdb.base/sigall.exp
drops from 23 seconds to 4 seconds.
A full native-gdbserver run on the AArch64 slave now takes 8 minutes.

However, I'm not sure if putting the sysroot into gdbserver_start will break
other remote setups.  In addition, I don't think gdb_test is the correct
function to call within the library - for the mi tests this causes a timeout
as it expect to use mi_gdb_test.

gdb/testsuite/ChangeLog:

2019-03-27  Alan Hayward  <alan.hayward@arm.com>

	* lib/gdbserver-support.exp (gdbserver_start): Set sysroot.
---
 gdb/testsuite/lib/gdbserver-support.exp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/gdb/testsuite/lib/gdbserver-support.exp b/gdb/testsuite/lib/gdbserver-support.exp
index dbd885aa22..065946eb8d 100644
--- a/gdb/testsuite/lib/gdbserver-support.exp
+++ b/gdb/testsuite/lib/gdbserver-support.exp
@@ -336,6 +336,9 @@ proc gdbserver_start { options arguments } {
 	break
     }
 
+    # Set sysroot to ensure files are read locally instead of via the remote protocol
+    gdb_test "set sysroot /" ""
+
     return [list $protocol [$get_remote_address $debughost $portnum]]
 }
 
-- 
2.20.1 (Apple Git-117)

Re: [RFC/PATCH] Testsuite: set sysroot when using gdbserver

[Kevin Buettner]

On Wed, 27 Mar 2019 16:40:30 +0000
Alan Hayward <Alan.Hayward@arm.com> wrote:

> However, I'm not sure if putting the sysroot into gdbserver_start will break
> other remote setups.  In addition, I don't think gdb_test is the correct
> function to call within the library - for the mi tests this causes a timeout
> as it expect to use mi_gdb_test.

Can sysroot be set in the board file(s)?

If so, I think that would be preferable to your proposed patch.

Kevin

Re: [RFC/PATCH] Testsuite: set sysroot when using gdbserver

[Kevin Buettner]

Hi Alan,

I want to apologize in case my latter comment came across as harsh.  I
think I could have phrased it more diplomatically.

I'll note that I also have an interest in setting up sysroot for the
OpenMP tests that I've been working on.  So I am genuinely interested
in whether or how sysroot can be set from a board file.

Also, making the tests run faster is definitely a good thing, so
thanks for looking into this.

Kevin

On Wed, 27 Mar 2019 13:49:15 -0700
Kevin Buettner <kevinb@redhat.com> wrote:

> On Wed, 27 Mar 2019 16:40:30 +0000
> Alan Hayward <Alan.Hayward@arm.com> wrote:
> 
> > However, I'm not sure if putting the sysroot into gdbserver_start will break
> > other remote setups.  In addition, I don't think gdb_test is the correct
> > function to call within the library - for the mi tests this causes a timeout
> > as it expect to use mi_gdb_test.  
> 
> Can sysroot be set in the board file(s)?
> 
> If so, I think that would be preferable to your proposed patch.
> 
> Kevin

Re: [RFC/PATCH] Testsuite: set sysroot when using gdbserver

[Alan Hayward]

> On 28 Mar 2019, at 05:14, Kevin Buettner <kevinb@redhat.com> wrote:
> 
> Hi Alan,
> 
> I want to apologize in case my latter comment came across as harsh.  I
> think I could have phrased it more diplomatically.

Oh, no, that’s fine :)

> 
> I'll note that I also have an interest in setting up sysroot for the
> OpenMP tests that I've been working on.  So I am genuinely interested
> in whether or how sysroot can be set from a board file.
> 
> Also, making the tests run faster is definitely a good thing, so
> thanks for looking into this.


I don’t know much about how the board files work, but, looks like I can
just add a flag via “set_board_info” in the relevant board files.

I can then check for it at the end of gdbserver_start. Or instead check
at the same place “target remote” is run (although I couldn’t find where
that was yesterday!).

This looks like a much better way of doing it.

I’m going to we on holiday for the next week, so I doubt I’ll be able to
get a new patch together before then.  I will look at it as soon as I get
back, as I’d like to get rid of the buildbot timeouts.

Alan.

> 
> Kevin
> 
> On Wed, 27 Mar 2019 13:49:15 -0700
> Kevin Buettner <kevinb@redhat.com> wrote:
> 
>> On Wed, 27 Mar 2019 16:40:30 +0000
>> Alan Hayward <Alan.Hayward@arm.com> wrote:
>> 
>>> However, I'm not sure if putting the sysroot into gdbserver_start will break
>>> other remote setups.  In addition, I don't think gdb_test is the correct
>>> function to call within the library - for the mi tests this causes a timeout
>>> as it expect to use mi_gdb_test.  
>> 
>> Can sysroot be set in the board file(s)?
>> 
>> If so, I think that would be preferable to your proposed patch.
>> 
>> Kevin

Re: [RFC/PATCH] Testsuite: set sysroot when using gdbserver

[Pedro Alves]

On 03/28/2019 11:02 AM, Alan Hayward wrote:
> 
> 
>> On 28 Mar 2019, at 05:14, Kevin Buettner <kevinb@redhat.com> wrote:
>>
>> Hi Alan,
>>
>> I want to apologize in case my latter comment came across as harsh.  I
>> think I could have phrased it more diplomatically.
> 
> Oh, no, that’s fine :)
> 
>>
>> I'll note that I also have an interest in setting up sysroot for the
>> OpenMP tests that I've been working on.  So I am genuinely interested
>> in whether or how sysroot can be set from a board file.
>>
>> Also, making the tests run faster is definitely a good thing, so
>> thanks for looking into this.
> 

IMO we could also look at this from the perspective that the slowness is
something that we should tackle, improve in gdb somehow.  For example, we
could improve caching a lot.  We have a readahead cache for vFile:pread, but
we could also have a persistent cache layer, so that we wouldn't be
downloading the same shared library files over and over again.

> 
> I don’t know much about how the board files work, but, looks like I can
> just add a flag via “set_board_info” in the relevant board files.
> 
> I can then check for it at the end of gdbserver_start. Or instead check
> at the same place “target remote” is run (although I couldn’t find where
> that was yesterday!).

Probably just adding

 set GDBFLAGS "${GDBFLAGS} -ex \"set sysroot /\""

to testsuite/boards/local-board.exp is all you'd need.

I think it'd be good to have a testcase in gdb.server/ that explicitly
tests debugging a bit with "set sysroot target:" enabled, so that we
don't lose coverage of that.

> This looks like a much better way of doing it.
> 
> I’m going to we on holiday for the next week, so I doubt I’ll be able to
> get a new patch together before then.  I will look at it as soon as I get
> back, as I’d like to get rid of the buildbot timeouts.

Thanks,
Pedro Alves

Re: [RFC/PATCH] Testsuite: set sysroot when using gdbserver

[Alan Hayward]

> On 28 Mar 2019, at 11:32, Pedro Alves <palves@redhat.com> wrote:
> 
> On 03/28/2019 11:02 AM, Alan Hayward wrote:
>> 
>> 
>>> On 28 Mar 2019, at 05:14, Kevin Buettner <kevinb@redhat.com> wrote:
>>> 
>>> Hi Alan,
>>> 
>>> I want to apologize in case my latter comment came across as harsh.  I
>>> think I could have phrased it more diplomatically.
>> 
>> Oh, no, that’s fine :)
>> 
>>> 
>>> I'll note that I also have an interest in setting up sysroot for the
>>> OpenMP tests that I've been working on.  So I am genuinely interested
>>> in whether or how sysroot can be set from a board file.
>>> 
>>> Also, making the tests run faster is definitely a good thing, so
>>> thanks for looking into this.
>> 
> 
> IMO we could also look at this from the perspective that the slowness is
> something that we should tackle, improve in gdb somehow.  For example, we
> could improve caching a lot.  We have a readahead cache for vFile:pread, but
> we could also have a persistent cache layer, so that we wouldn't be
> downloading the same shared library files over and over again.

Will caching only effect extended gdbserver (which AIUI, gdbserver stays alive
between tests)? It’s not an area I’ve (yet) looked at.

> 
>> 
>> I don’t know much about how the board files work, but, looks like I can
>> just add a flag via “set_board_info” in the relevant board files.
>> 
>> I can then check for it at the end of gdbserver_start. Or instead check
>> at the same place “target remote” is run (although I couldn’t find where
>> that was yesterday!).
> 
> Probably just adding
> 
> set GDBFLAGS "${GDBFLAGS} -ex \"set sysroot /\""
> 
> to testsuite/boards/local-board.exp is all you'd need.


That fixes it! And logically makes sense too.
Works fine on the buildbot slave.

> 
> I think it'd be good to have a testcase in gdb.server/ that explicitly
> tests debugging a bit with "set sysroot target:" enabled, so that we
> don't lose coverage of that.

Agreed. Happy to raise that as a new patch next week - trying to avoid
writing anything new as I’m not really here today.

Is the following patch ok for pushing?

Given you told me the line to add, I wasn’t sure if I should be adding you
to the changelog line too :)



    Testsuite: set sysroot when using gdbserver

    When testing using native-gdbserver and native-extended-gdbserver, the sysroot
    is not set.  This results in a warning from GDB and files are sent via the
    remote protocol, which can be slow.

    On Ubuntu 18.04 (unlike most distros) the debug versions of the standard
    libraries are included by default in /usr/lib/debug/.

    These file reads are causing a complete native-gdbserver run on the AArch64
    buildbot slave to timeout after 2.5 hours.  This is also causing the builds
    to back up on the slave.

    The solution is to ensure the sysroot is set to / for all local boards.

    This drastically reduces the time of a test. For example, gdb.base/sigall.exp
    drops from 23 seconds to 4 seconds.
    A full native-gdbserver run on the AArch64 slave now takes 8 minutes.

    gdb/testsuite/ChangeLog:

    2019-03-28  Alan Hayward  <alan.hayward@arm.com>
                Pedro Alves  <palves@redhat.com>

            * boards/local-board.exp: set sysroot to /.

diff --git a/gdb/testsuite/boards/local-board.exp b/gdb/testsuite/boards/local-board.exp
index 4150f1d1d6..bf710f4632 100644
--- a/gdb/testsuite/boards/local-board.exp
+++ b/gdb/testsuite/boards/local-board.exp
@@ -22,3 +22,6 @@ global board_info
 # Remove any target variant specifications from the name.
 set baseboard [lindex [split $board "/"] 0]
 set board_info($baseboard,isremote) 0
+
+# Set sysroot to avoid sending files via the remote protocol.
+set GDBFLAGS "${GDBFLAGS} -ex \"set sysroot /\""

Re: [RFC/PATCH] Testsuite: set sysroot when using gdbserver

[Pedro Alves]

On 03/28/2019 12:50 PM, Alan Hayward wrote:
> 

>>
>> IMO we could also look at this from the perspective that the slowness is
>> something that we should tackle, improve in gdb somehow.  For example, we
>> could improve caching a lot.  We have a readahead cache for vFile:pread, but
>> we could also have a persistent cache layer, so that we wouldn't be
>> downloading the same shared library files over and over again.
> 
> Will caching only effect extended gdbserver (which AIUI, gdbserver stays alive
> between tests)? It’s not an area I’ve (yet) looked at.

It'd work for plain target remote too.  What I meant by persistent is that it
would persist across gdb invocations, saved in a cache directory, similarly
to where we put the new index cache.  We could e.g., key that on build id.

> 
>>
>>>
>>> I don’t know much about how the board files work, but, looks like I can
>>> just add a flag via “set_board_info” in the relevant board files.
>>>
>>> I can then check for it at the end of gdbserver_start. Or instead check
>>> at the same place “target remote” is run (although I couldn’t find where
>>> that was yesterday!).
>>
>> Probably just adding
>>
>> set GDBFLAGS "${GDBFLAGS} -ex \"set sysroot /\""
>>
>> to testsuite/boards/local-board.exp is all you'd need.
> 
> 
> That fixes it! And logically makes sense too.
> Works fine on the buildbot slave.
> 
>>
>> I think it'd be good to have a testcase in gdb.server/ that explicitly
>> tests debugging a bit with "set sysroot target:" enabled, so that we
>> don't lose coverage of that.
> 
> Agreed. Happy to raise that as a new patch next week - trying to avoid
> writing anything new as I’m not really here today.

That'd be great, thanks.

> 
> Is the following patch ok for pushing?
> 
> Given you told me the line to add, I wasn’t sure if I should be adding you
> to the changelog line too :)
> 

Fine with me either way.

Patch is OK.

Thanks,
Pedro Alves

Regression on gdb.base/break-probes.exp - native-{,extended-}gdbserver (was: Re: [RFC/PATCH] Testsuite: set sysroot when using gdbserver)

[Sergio Durigan Junior]

On Wednesday, March 27 2019, Alan Hayward wrote:

> When testing using native-gdbserver and native-extended-gdbserver, the sysroot
> is not set.  This results in a warning from GDB and files are sent via the
> remote protocol, which can be slow.
>
> On Ubuntu 18.04 (unlike most distros) the debug versions of the standard
> libraries are included by default in /usr/lib/debug/
>
> Example log of a test on Ubuntu 18.04 AArch64:
>
> Listening on port 2346
> target remote localhost:2346
> Remote debugging using localhost:2346
> Reading /lib/ld-linux-aarch64.so.1 from remote target...
> warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
> Reading /lib/ld-linux-aarch64.so.1 from remote target...
> Reading symbols from target:/lib/ld-linux-aarch64.so.1...
> Reading /lib/ld-2.27.so from remote target...
> Reading /lib/.debug/ld-2.27.so from remote target...
> Reading /usr/lib/debug//lib/ld-2.27.so from remote target...
> Reading /usr/lib/debug/lib//ld-2.27.so from remote target...
> Reading target:/usr/lib/debug/lib//ld-2.27.so from remote target...
> (No debugging symbols found in target:/lib/ld-linux-aarch64.so.1)
> 0x0000ffffbf6d31c0 in ?? () from target:/lib/ld-linux-aarch64.so.1
> (gdb) continue
> Continuing.
> Reading /lib/aarch64-linux-gnu/libc.so.6 from remote target...
> Reading /lib/aarch64-linux-gnu/libc-2.27.so from remote target...
> Reading /lib/aarch64-linux-gnu/.debug/libc-2.27.so from remote target...
> Reading /usr/lib/debug//lib/aarch64-linux-gnu/libc-2.27.so from remote target...
> Reading /usr/lib/debug//lib/aarch64-linux-gnu/libc-2.27.so from remote target...
>
> These file reads are causing a complete native-gdbserver run on the AArch64
> buildbot slave to timeout after 2.5 hours.  This is also causing the builds
> to back up on the slave.
>
> The solution is to ensure the sysroot is set to / immediately after connecting.
>
> This drastically reduces the time of a test. For example, gdb.base/sigall.exp
> drops from 23 seconds to 4 seconds.
> A full native-gdbserver run on the AArch64 slave now takes 8 minutes.
>
> However, I'm not sure if putting the sysroot into gdbserver_start will break
> other remote setups.  In addition, I don't think gdb_test is the correct
> function to call within the library - for the mi tests this causes a timeout
> as it expect to use mi_gdb_test.
>
> gdb/testsuite/ChangeLog:
>
> 2019-03-27  Alan Hayward  <alan.hayward@arm.com>
>
> 	* lib/gdbserver-support.exp (gdbserver_start): Set sysroot.
> ---
>  gdb/testsuite/lib/gdbserver-support.exp | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/gdb/testsuite/lib/gdbserver-support.exp b/gdb/testsuite/lib/gdbserver-support.exp
> index dbd885aa22..065946eb8d 100644
> --- a/gdb/testsuite/lib/gdbserver-support.exp
> +++ b/gdb/testsuite/lib/gdbserver-support.exp
> @@ -336,6 +336,9 @@ proc gdbserver_start { options arguments } {
>  	break
>      }
>  
> +    # Set sysroot to ensure files are read locally instead of via the remote protocol
> +    gdb_test "set sysroot /" ""
> +
>      return [list $protocol [$get_remote_address $debughost $portnum]]
>  }

Hi Alan,

I have just noticed that two tests of gdb.base/break-probes.exp are
failing, most likely because of this patch.

These are the failures:

  (gdb) PASS: gdb.base/break-probes.exp: ensure using probes
  show sysroot
  The current system root is "/".
  (gdb) c
  Continuing.
  Stopped due to shared library event:
    Inferior loaded /lib64/libdl.so.2
      /lib64/libm.so.6
      /lib64/libc.so.6
  (gdb) c
  Continuing.
  Stopped due to shared library event (no libraries added or removed)
  (gdb) c
  Continuing.
  Stopped due to shared library event:
    Inferior loaded /build/gdb/testsuite/outputs/gdb.base/break-probes/break-probes-solib.so
  (gdb) c
  Continuing.
  Stopped due to shared library event (no libraries added or removed)
  (gdb) c
  Continuing.
  Stopped due to shared library event:
    Inferior unloaded /build/gdb/testsuite/outputs/gdb.base/break-probes/break-probes-solib.so
  (gdb) c
  Continuing.
  [Inferior 1 (process 20591) exited normally]
  (gdb) FAIL: gdb.base/break-probes.exp: run til our library loads (the program exited)
  call (int) foo(23)
  No symbol "foo" in current context.
  (gdb) FAIL: gdb.base/break-probes.exp: call (int) foo(23)
  Remote debugging from host ::1, port 33292
  Process /build/gdb/testsuite/outputs/gdb.base/break-probes/break-probes created; pid = 20591
  Child exited with status 0
  monitor exit

You can see the BuildBot build here:

  https://gdb-build.sergiodj.net/builders/Fedora-x86_64-native-gdbserver-m64/builds/12335

The logs:

  https://gdb-build.sergiodj.net/results/Fedora-x86_64-native-gdbserver-m64/c9/c92df149c29518f6e1d4a3174b3e29162fcd3ad6/

I'm in the middle of something else now, but let me know if you need
help with the investigation.

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/