[Bug symtab/29398] New: [gdb/symtab] lazy CU expansion

[Bug symtab/29398] New: [gdb/symtab] lazy CU expansion

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

            Bug ID: 29398
           Summary: [gdb/symtab] lazy CU expansion
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: symtab
          Assignee: unassigned at sourceware dot org
          Reporter: vries at gcc dot gnu.org
  Target Milestone: ---

>From PR29391 comment 1:
...
The second problem is that CU expansion can be slow.  Here I think
gdb could do a lot better, the basic idea being lazy CU expansion.
In response to a CU expansion request, the DWARF reader would
create the symtab / compunit_symtab structures and it would also
create some "outline" struct symbols -- one for each cooked_index_entry.
Then when some attribute of a symbol is needed (say, the type),
the DWARF reader would read the rest of the symbol that that moment.

The major advantage of this approach is that most data in a CU
is not needed at all.  So, much less work would need to be done in
general.  I think it would be possible to avoid reading every DIE.

A secondary advantage is that, because the symbols are created directly
from the cooked index, we would avoid the situation where the
two readers could diverge.  That would no longer be possible at all.

There are some downsides.  It's more complex, and it is complicated to
implement and test.  Also I think it would require fixing the .debug_names
bug, and also probably removing .gdb_index support.  Finally, we'd have
to change the blockvector to be expandable.
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tromey at sourceware dot org

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Tom de Vries from comment #0)
> Also I think it would require fixing the .debug_names
> bug, 

Can you point out which one? Is it perhaps PR24820?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #2 from Tom de Vries <vries at gcc dot gnu.org> ---
FTR, this (
https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/vries/lazy-symtab-expansion
) is a WIP branch from before the cooked index rewrite.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #3 from Tom de Vries <vries at gcc dot gnu.org> ---
(In reply to Tom de Vries from comment #2)
> FTR, this (
> https://sourceware.org/git/?p=binutils-gdb.git;a=shortlog;h=refs/heads/users/
> vries/lazy-symtab-expansion ) is a WIP branch from before the cooked index
> rewrite.

Also sent out an RFC:
https://sourceware.org/pipermail/gdb-patches/2021-June/179886.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #4 from Tom Tromey <tromey at sourceware dot org> ---
Bug #17762 is an example of a name divergence that
would be impossible by unifying the readers.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #5 from Tom Tromey <tromey at sourceware dot org> ---
This came up a little on the list:
https://sourceware.org/pipermail/gdb-patches/2022-December/194993.html
so I thought I'd write up my tentative notes/plan.

The basic idea is that the cooked index can be used to directly
construct the symtab.  The names can easily be computed, and
some (but not all) of each symbol's contents can be filled in.

Then, we'd set a flag on the symbol and have the various accessor
methods in 'symbol' call the DWARF reader to fill in the details.

Ideally the DWARF reader would do the minimum possible to satisfy
the request.  This would require making the blockvector
expandable, if function bodies are read lazily.  Types could
be read lazily by hooking into check_typedef.

It would also be good to avoid reading every DIE in the CU.
Really excellent might be to do what the indexer does, and
simply not read DIEs at all -- just walk attributes and
process them.  However, this would require a substantial rewrite.

If just the DIEs rooted at the offset mentioned in the cooked entry
are read, it seems straightforward enough to read other DIEs
on demand.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |29366


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=29366
[Bug 29366] [meta] New DWARF indexer meta bug
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #6 from Tom Tromey <tromey at sourceware dot org> ---
Bug #30271 is another one that would be impossible if
the symtabs were unified -- it would have been caught
in testing, I think, because expanding the symtab would
still provoke the bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[sam at gentoo dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #7 from Tom Tromey <tromey at sourceware dot org> ---
I was thinking about this a little and realized it may be harder
than I thought :-(

The main issue is that full symbols for functions include 
argument types -- but this information is omitted from the index.

It's tempting to try to change this, but I suspect it would
be fairly difficult.

But then this means that simply constructing the symtab would
require reading DIEs and constructing type names.

I wonder if a different approach can be found.
We've sometimes discussed the idea of having the "quick"
functions also do the symtab searching -- that is, instead
of pre-expanding symtabs and then separately searching,
combine the two phases.  Maybe this idea could be extended
so that the DWARF reader doesn't even need to build a full
symtab.

Alternatively maybe we should reconsider the idea of expanding
CUs in the background.  The idea here would be to have the
pre-expansion steps push work to background threads, then
have the later phase wait for the work.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #8 from Tom Tromey <tromey at sourceware dot org> ---
Maybe one idea would be to have symbol creation itself be
lazy.  Then, lookups that end up in an expansion would also
have to read enough type info to construct the function's
full name.

A more radical option would be to drop the function parameter
info from full symbols.  I don't know what the drawbacks of
this would be.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #9 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Tom Tromey from comment #8)

> A more radical option would be to drop the function parameter
> info from full symbols.  I don't know what the drawbacks of
> this would be.

Maybe it would make completion more difficult.
That might be the worst / hardest part.

It might also affect what's seen in backtraces, but that doesn't
seem so bad; and anyway is fixable by examining the function's
type.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug symtab/29398] [gdb/symtab] lazy CU expansion

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29398

--- Comment #10 from Tom Tromey <tromey at sourceware dot org> ---
I've been thinking I'd really like to remove the
full parameters from symbol names.  This would allow
the removal of a bunch of hacks.

It's difficult though.  And there are things like stabs
to maybe deal with (not sure).

I'd also like to remove the C parser extension that
lets you put full parameter types into a call, but
I don't know if that's important for this project.

-- 
You are receiving this mail because:
You are on the CC list for the bug.