[Bug symtab/29942] New: Read some DWARF purely in background

[Bug symtab/29942] New: Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

            Bug ID: 29942
           Summary: Read some DWARF purely in background
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: symtab
          Assignee: unassigned at sourceware dot org
          Reporter: tromey at sourceware dot org
  Target Milestone: ---

Currently, dwarf2_build_psymtabs_hard works synchronously: it sends
some work to worker threads, but it waits for all work to be
completed before proceeding.

However, it doesn't have to work this way.  Instead, it could start
reading immediately; and then after starting the background tasks,
it could store a future that would then be waited on by
cooked_index_functions::read_partial_symbols.

I think the main benefit to doing this would be that "attach"
might appear to be faster.  In particular work like downloading
from debuginfod could be done while other threads are scanning
the DWARF.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |29366


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=29366
[Bug 29366] [meta] New DWARF indexer meta bug
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #1 from Tom Tromey <tromey at sourceware dot org> ---
FWIW I have a WIP patch for this.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #2 from Tom Tromey <tromey at sourceware dot org> ---
It occurs to me that maybe everything in dwarf2_build_psymtabs_hard
after the sections are read could be done in a worker thread.
(This kind of thing would maybe be easier with C++20 coroutines.)

> I think the main benefit to doing this would be that "attach"
> might appear to be faster.

The theory behind this is that with "attach", breakpoint re-sets
are deferred until all the shared libraries are read, providing
a little extra time for the background threads to work.

It might be interesting to see if minsym reading can also be
done in a worker.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #3 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Tom Tromey from comment #2)
> It occurs to me that maybe everything in dwarf2_build_psymtabs_hard
> after the sections are read could be done in a worker thread.

I haven't tested 'attach' yet, but this doesn't improve things
in the ordinary 'file' case, mainly I think due to the handling
of 'main'.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #4 from Tom Tromey <tromey at sourceware dot org> ---
I had a patch to add some thread-safety to BFD on this branch,
but I belatedly realized it can't really work, due to the way
the BFD fd cache works -- we'd have to introduce some way to
lock an fd while a function is using it.

Instead I'm going to try external locking and avoid modifying BFD at all.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #5 from Tom Tromey <tromey at sourceware dot org> ---
I also looked at wrapping the cache iovec in a locked variant.
However, this code in bfd_cache_close prevents that approach:

  /* Don't remove this test.  bfd_reinit depends on it.  */
  if (abfd->iovec != &cache_iovec)
    return true;

I think I'll just end up wrapping all "suspect" BFD calls in
a global lock.  This may not be too bad, especially if we switch
back to eager mapping of DWARF sections.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #6 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Tom Tromey from comment #5)
> I also looked at wrapping the cache iovec in a locked variant.
> However, this code in bfd_cache_close prevents that approach:
> 
>   /* Don't remove this test.  bfd_reinit depends on it.  */
>   if (abfd->iovec != &cache_iovec)
>     return true;
> 
> I think I'll just end up wrapping all "suspect" BFD calls in
> a global lock.  This may not be too bad, especially if we switch
> back to eager mapping of DWARF sections.

There are 135 separate bfd_* functions called by gdb.
So, auditing this is a pain and probably error-prone.
And then some subset have to be wrapped in a lock, including
any that set the BFD error.
Coming back around to maybe trying to make the BFD error be thread-local.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at sourceware dot org   |tromey at sourceware dot org

--- Comment #7 from Tom Tromey <tromey at sourceware dot org> ---
I've been working on this.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #8 from Tom Tromey <tromey at sourceware dot org> ---
https://sourceware.org/pipermail/gdb-patches/2023-November/203984.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

--- Comment #9 from Sourceware Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Tom Tromey <tromey@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=33c6eaaefcedd45e86d564d014f14cce2620f933

commit 33c6eaaefcedd45e86d564d014f14cce2620f933
Author: Tom Tromey <tom@tromey.com>
Date:   Fri Mar 24 23:35:02 2023 -0600

    Do more DWARF reading in the background

    This patch rearranges the DWARF reader so that more work is done in
    the background.  This is PR symtab/29942.

    The idea here is that there is only a small amount of work that must
    be done on the main thread when scanning DWARF -- before the main
    scan, the only part is mapping the section data.

    Currently, the DWARF reader uses the quick_symbol_functions "lazy"
    functionality to defer even starting to read.  This patch instead
    changes the reader to start reading immediately, but doing more in
    worker tasks.

    Before this patch, "file" on my machine:

        (gdb) file /tmp/gdb
        2023-10-23 12:29:56.885 - command started
        Reading symbols from /tmp/gdb...
        2023-10-23 12:29:58.047 - command finished
        Command execution time: 5.867228 (cpu), 1.162444 (wall)

    After the patch, more work is done in the background and so this takes
    a bit less time:

        (gdb) file /tmp/gdb
        2023-10-23 13:25:51.391 - command started
        Reading symbols from /tmp/gdb...
        2023-10-23 13:25:51.712 - command finished
        Command execution time: 1.894500 (cpu), 0.320306 (wall)

    I think this could be further sped up by using the shared library load
    map to avoid objfile loops like the one in expand_symtab_containing_pc
    -- it seems like the correct objfile could be chosen more directly.

    Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=29942
    Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30174

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29942] Read some DWARF purely in background

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29942

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED
   Target Milestone|---                         |15.1

--- Comment #10 from Tom Tromey <tromey at sourceware dot org> ---
Fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.