.. in the global namespace
The problem was the interaction of #116989 with an optimization in
GetTypesWithQuery. The optimization was only correct for non-exact
matches, but that didn't matter before this PR due to the "second layer
of defense". After that was removed, the query started returning more
types than it should.
## Summary
This PR is a continuation of
https://github.com/llvm/llvm-project/pull/108907 by using `.debug_names`
parent chain faster lookup for namespaces.
## Implementation
Similar to https://github.com/llvm/llvm-project/pull/108907. This PR
adds a new API: `GetNamespacesWithParents` in `DWARFIndex` base class.
The API performs the same function as `GetNamespaces()` with additional
filtering using parents `CompilerDeclContext`. A default implementation
is given in `DWARFIndex` class which parses debug info and performs the
matching. In the `DebugNameDWARFIndex` override, parents
`CompilerDeclContext` is cross checked with parent chain in
`.debug_names` for much faster filtering before fallback to base
implementation for final filtering.
## Performance Results
For the same benchmark used in
https://github.com/llvm/llvm-project/pull/108907, this PR improves: 48s
=> 28s
---------
Co-authored-by: jeffreytan81 <jeffreytan@fb.com>
## Summary
This PR improves `SymbolFileDWARF::FindTypes()` by utilizing the newly
added parent chain `DW_IDX_parent` in `.debug_names`. The proposal was
originally discussed in [this
RFC](https://discourse.llvm.org/t/rfc-improve-dwarf-5-debug-names-type-lookup-parsing-speed/74151).
## Implementation
To leverage the parent chain for `SymbolFileDWARF::FindTypes()`, this PR
adds a new API: `GetTypesWithQuery` in `DWARFIndex` base class. The API
performs the same function as `GetTypes` with additional filtering using
`TypeQuery`. Since this only introduces filtering, the callback
mechanisms at all call sites remain unchanged. A default implementation
is given in `DWARFIndex` class which parses debug info and performs the
matching. In the `DebugNameDWARFIndex` override, the parent_contexts in
the `TypeQuery` is cross checked with parent chain in `.debug_names` for
for much faster filtering before fallback to base implementation for
final filtering.
Unlike the `GetFullyQualifiedType` API, which fully consumes the
`DW_IDX_parent` parent chain for exact matching, these new APIs perform
partial subset matching for type/namespace queries. This is necessary to
support queries involving anonymous or inline namespaces. For instance,
a user might request `NS1::NS2::NS3::Foo`, while the index table's
parent chain might contain `NS1::inline_NS2::NS3::Foo`, which would fail
exact matching.
## Performance Results
In one of our internal target using `.debug_names` + split dwarf.
Expanding a "this" pointer in locals view in VSCode:
94s => 48s. (Not sure why I got 94s this time instead of 70s last week).
---------
Co-authored-by: jeffreytan81 <jeffreytan@fb.com>
As specified in the docs,
1) raw_string_ostream is always unbuffered and
2) the underlying buffer may be used directly
( 65b13610a5226b84889b923bae884ba395ad084d for further reference )
* Don't call raw_string_ostream::flush(), which is essentially a no-op.
* Avoid unneeded calls to raw_string_ostream::str(), to avoid excess
indirection.
The test appears to be flaky. Revert it while I investigate.
This reverts commits 7027cc6a073cb5ae7a0ce04fa4a2dbe714615da9 and
21ef272ec1974244710fc639f98674eae3f8b02c.
…Type
This is needed to ensure we find a type if its definition is in a CU
that wasn't indexed. This can happen if the definition is in some
precompiled code (e.g. the c++ standard library) which was built with
different flags than the rest of the binary.
This patch adds support for the new foreign type unit support in
.debug_names. Features include:
- don't manually index foreign TUs if we have info for them
- only use the type unit entries that match the .dwo files when we have
a .dwp file
- fix type unit lookups for .dwo files
- fix crashers that happen due to PeekDIEName() using wrong offsets where an entry had DW_IDX_comp_unit and DW_IDX_type_unit entries and when we had no type unit support, it would cause us to think it was a normal DIE in .debug_info from the main executable.
---------
Co-authored-by: paperchalice <liujunchang97@outlook.com>
This makes sure we try to process declaration DIEs that are erroneously
present in the index. Until bd5c6367bd7, clang was emitting index
entries for declaration DIEs with DW_AT_signature attributes. This makes
sure to avoid returning those DIEs as the definitions of a type, but
also makes sure to pass through DIEs referring to static constexpr
member variables, which is a (probably nonconforming) extension used by
dsymutil.
It adds test cases for both of the scenarios. It is essentially a
recommit of #91808.
and two follow-up commits. The reason is the crash we've discovered when
processing -gsimple-template-names binaries. I'm committing a minimal
reproducer as a separate patch.
This reverts the following commits:
- 51dd4eaaa29683c16151f5168e7f8645acbd6e6c (#92328)
- 3d9d48523977af3590f7dd0edfd258454cb9e9cf (#93839)
- afe6ab7586f7078cc410f6162bd9851e48e2a286 (#94400)
This is a follow-up of
https://github.com/llvm/llvm-project/pull/92328#issuecomment-2145849441
Clang attaches `DW_AT_declaration` to static inline data members and
`dsymutil` indexes these constants. Skipping these caused the expression
evaluator to fail to find such constants when using DWARFv5.
Fixes `TestConstStaticIntegralMember.py` on DWARFv5.
DebugNamesDWARFIndex was jumping through hoops to construct a DIERef
from an index entry only to jump through them back a short while later
to construct a DWARFDIE.
This used to be necessary as the index lookup was a two stage process,
where we first enumerated all matches, and then examined them (so it was
important that the enumeration was cheap -- does not trigger unnecessary
parsing). However, now that the processing is callback based, we are
always immediately examining the DWARFDIE right after finding the entry,
and the DIERef just gets in the way.
This reapplies
9a7262c260
(#90663) and added https://github.com/llvm/llvm-project/pull/91808 as a
fix.
It was causing tests on macos to fail because
`SymbolFileDWARF::GetForwardDeclCompilerTypeToDIE` returned the map
owned by this symol file. When there were two symbol files, two
different maps were created for caching from compiler type to DIE even
if they are for the same module. The solution is to do the same as
`SymbolFileDWARF::GetUniqueDWARFASTTypeMap`: inquery
SymbolFileDWARFDebugMap first to get the shared underlying SymbolFile so
the map is shared among multiple SymbolFileDWARF.
DWARFDebugInfo only knows how to resolve references in its own file, but
in split dwarf, the index entries will refer to DIEs in the separate
(DWO) file. To resolve the DIERef correctly we'd either need to go
through the SymbolFileDWARF to get the full logic for resolving a
DIERef, or use the fact that ToDIERef already looks up the correct unit
while computing its result.
This patch does the latter.
This bug manifested itself in not being able to find type definitions
for types in namespaces, so I've modified one of our type resolving test
cases to run with debug_names, and added a namespaced class into it (it
originally contained only a top-level class).
This commit changes DebugNamesDWARFIndex so that it now overrides
`GetFullyQualifiedType` and attempts to use DW_IDX_parent, when
available, to speed up such queries. When this type of information is
not available, the base-class implementation is used.
With this commit, we now achieve the 4x speedups reported in [1].
[1]:
https://discourse.llvm.org/t/rfc-improve-dwarf-5-debug-names-type-lookup-parsing-speed/74151/38
This is a follow up patch after .debug_names can now emit local type
unit entries when we compile with type units + DWARF5 + .debug_names.
The pull request that added this functionality was:
https://github.com/llvm/llvm-project/pull/70515
This patch makes sure that the DebugNamesDWARFIndex in LLDB will not
manually need to parse type units if they have a valid index. It also
fixes the index to be able to correctly extract name entries that
reference type unit DIEs. Added a test to verify things work as
expected.
While using dwarf5 `.debug_names` in internal large targets, we noticed
a performance issue (around 10 seconds delay) while `lldb-vscode` tries
to show `scopes` for a compile unit. Profiling shows the bottleneck is
inside `DebugNamesDWARFIndex::GetGlobalVariables` which linearly search
all index entries belongs to a compile unit.
This patch improves the performance by using the compile units list to
filter first before checking index entries. This significantly improves
the performance (drops from 10 seconds => under 1 second) in the split
dwarf situation because each compile unit has its own named index.
---------
Co-authored-by: jeffreytan81 <jeffreytan@fb.com>
As a followup of https://github.com/llvm/llvm-project/pull/67851, I'm
defining a new namespace `lldb_plugin::dwarf` for the classes in this
Plugins/SymbolFile/DWARF folder. This change is very NFC and helped me
with exporting the necessary symbols for my out-of-tree language plugin.
The only class that I didn't change is ClangDWARFASTParser, because that
shouldn't be in the same namespace as the generic language-agnostic
dwarf parser.
It would be a good idea if other plugins follow the same namespace
scheme.
In order to recognize namespace aliases as a namespace, the
DW_TAG_imported_declaration has to be taken into account. The name of these DIEs
is already included in all accelerator tables as of D143397.
Two of the three Index classes already handle this correctly:
1. ManualDWARFIndex (as of D143398)
2. AppleDWARFIndex works by default with D143397, since apple has a table
dedicated to namespaces.
This commit updates the third index class, DWARF 5's DebugNamesDWARFIndex.
As a result, it fixes the following test with DWARF 5:
commands/expression/namespace-alias/TestInlineNamespaceAlias.py
Differential Revision: https://reviews.llvm.org/D154730
When LLDB queries the debug names index with a regex, we should use the
`Mangled` class wrapper, which attempts to match regex first against the mangled
name and then against the demangled name. It is important to do so, otherwise
queries like `frame var --regex A::` would never work. This is what is done for
the Apple index as well.
This fixes test/API/lang/cpp/class_static/main.cpp when compiled with DWARF 5.
Differential Revision: https://reviews.llvm.org/D154617
LLDB's implementation of DWARFDataExtractor has a method that returns a
llvm::DWARFDataExtractor. In some cases, like DebugNamesDWARFIndex::Create, we
were passing an LLVM::DWARFDataExtractor to a function that expects a
LLVM:DataExtractor by value. This is causing slicing of the derived class.
While slicing is not inherently bad, it can be dangerous if the constructor of
the derived class mutates the base class in a way that leaves it in an invalid
state after slicing.
Differential Revision: https://reviews.llvm.org/D153913
34a8e6eee666 changed SymbolFileDWARF::GetDwoNum to
SymbolFileDWARF::GetFileIndex but changed the meaning from just DWO to
DWO and OSO which changed the meaning of the assert. The assert was
therefore removed from ManualDWARFIndex::GetGlobalVariables and
ManualDWARFIndex::GetGlobalVariables but was still present in
DebugNamesDWARFIndex::GetGlobalVariables. If we want to reintroduce the
assert, we need something with the old semantics for all 3.
This came out of from https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902
With big binaries we can have .dwp files where .debug_info.dwo section can grow
beyond 4GB. We would like to support this in LLVM and in LLDB.
The plan is to enable manual parsing of cu/tu index in DWARF library
(https://reviews.llvm.org/D137882), and then
switch internal index data structure to 64 bit.
For the second part is to enable 64bit offset support in LLDB with
this patch.
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D138618
This came out of from https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902
With big binaries we can have .dwp files where .debug_info.dwo section can grow
beyond 4GB. We would like to support this in LLVM and in LLDB.
The plan is to enable manual parsing of cu/tu index in DWARF library
(https://reviews.llvm.org/D137882), and then
switch internal index data structure to 64 bit.
For the second part is to enable 64bit offset support in LLDB with
this patch.
Depends on D139955
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D138618
This came out of from https://discourse.llvm.org/t/dwarf-dwp-4gb-limit/63902
With big binaries we can have .dwp files where .debug_info.dwo section can grow
beyond 4GB. We would like to support this in LLVM and in LLDB.
The plan is to enable manual parsing of cu/tu index in DWARF library
(https://reviews.llvm.org/D137882), and then
switch internal index data structure to 64 bit.
For the second part is to enable 64bit offset support in LLDB with
this patch.
Depends on D139955
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D138618
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
This reverts commit 967df65a3610f98a3bc0ec0f2303641d7bad176c.
This fixes test/Shell/SymbolFile/NativePDB/find-functions.cpp. When
looking up functions with the PDB plugins, if we are looking for a
full function name, we should use `GetName` to populate the `name`
field instead of `GetLookupName` since `GetName` has the more
complete information.
This reverts commit befa77e59a7760d8c4fdd177b234e4a59500f61c.
Looks like this broke a SymbolFileNativePDB test. I'll investigate and
resubmit with a fix soon.
Context:
When setting a breakpoint by name, we invoke Module::FindFunctions to
find the function(s) in question. However, we use a Module::LookupInfo
to first process the user-provided name and figure out exactly what
we're looking for. When we actually perform the function lookup, we
search for the basename. After performing the search, we then filter out
the results using Module::LookupInfo::Prune. For example, given
a:🅱️:foo we would first search for all instances of foo and then filter
out the results to just names that have a:🅱️:foo in them. As one can
imagine, this involves a lot of debug info processing that we do not
necessarily need to be doing. Instead of doing one large post-processing
step after finding each instance of `foo`, we can filter them as we go
to save time.
Some numbers:
Debugging LLDB and placing a breakpoint on
llvm::itanium_demangle::StringView::begin without this change takes
approximately 70 seconds and resolves 31,920 DIEs. With this change,
placing the breakpoint takes around 30 seconds and resolves 8 DIEs.
Differential Revision: https://reviews.llvm.org/D129682
This diff introduces a new symbol on-demand which skips
loading a module's debug info unless explicitly asked on
demand. This provides significant performance improvement
for application with dynamic linking mode which has large
number of modules.
The feature can be turned on with:
"settings set symbols.load-on-demand true"
The feature works by creating a new SymbolFileOnDemand class for
each module which wraps the actual SymbolFIle subclass as member
variable. By default, most virtual methods on SymbolFileOnDemand are
skipped so that it looks like there is no debug info for that module.
But once the module's debug info is explicitly requested to
be enabled (in the conditions mentioned below) SymbolFileOnDemand
will allow all methods to pass through and forward to the actual SymbolFile
which would hydrate module's debug info on-demand.
In an internal benchmark, we are seeing more than 95% improvement
for a 3000 modules application.
Currently we are providing several ways to on demand hydrate
a module's debug info:
* Source line breakpoint: matching in supported files
* Stack trace: resolving symbol context for an address
* Symbolic breakpoint: symbol table match guided promotion
* Global variable: symbol table match guided promotion
In all above situations the module's debug info will be on-demand
parsed and indexed.
Some follow-ups for this feature:
* Add a command that allows users to load debug info explicitly while using a
new or existing command when this feature is enabled
* Add settings for "never load any of these executables in Symbols On Demand"
that takes a list of globs
* Add settings for "always load the the debug info for executables in Symbols
On Demand" that takes a list of globs
* Add a new column in "image list" that shows up by default when Symbols On
Demand is enable to show the status for each shlib like "not enabled for
this", "debug info off" and "debug info on" (with a single character to
short string, not the ones I just typed)
Differential Revision: https://reviews.llvm.org/D121631
We have using namespace llvm::dwarf in dwarf.h header globally. Replacing that
with a using namespace within lldb_private::dwarf and moving to a
using namespace lldb_private::dwarf in .cpp files and fully qualified names
in the few header files.
Differential Revision: https://reviews.llvm.org/D120836
Skeleton vs. DWO units mismatch has been fixed in D106270. As they both
have type DWARFUnit it is a bit difficult to debug. So it is better to
make it safe against future changes.
Reviewed By: kimanh, clayborg
Differential Revision: https://reviews.llvm.org/D107659
When going through the CU entries in the name index,
make sure to compare the name entry's CU
offset against the skeleton CU's offset.
Previously there would be a mismatch, since the
wrong offset was compared, and thus no suitable
entry was found.
Reviewed By: jankratochvil
Differential Revision: https://reviews.llvm.org/D106270
If we succeed at gathering global variables for a compile
unit, there is no need to fallback to generating a manual index.
Reviewed By: jankratochvil
Differential Revision: https://reviews.llvm.org/D106355
Pavel Labath wrote in D73206:
The internal representation of DebugNames and Apple indexes is fixed by
the relevant (pseudo-)standards, so we can't really change it. The
question is how to efficiently (and cleanly) convert from the internal
representation to some common thing. The conversion from AppleIndex to
DIERef is trivial (which is not surprising as it was the first and the
overall design was optimized for that). With debug_names, the situation
gets more tricky. The internal representation of debug_names uses
CU-relative DIE offsets, but DIERef wants an absolute offset. That means
the index has to do more work to produce the common representation. And
it needs to do that for all results, even though a lot of the index
users are really interested only in a single entry. With the switch to
user_id_t, _all_ indexes would have to do some extra work to encode it,
only for their users to have to immediately decode it back. Having
a iterator/callback based api would allow us to minimize the impact of
that, as it would only need to happen for the entries that are really
used. And /I think/ we could make it interface returns DWARFDies
directly, and each index converts to that using the most direct approach
available.
Jan Kratochvil:
It also makes all the callers shorter as they no longer need to fetch
DWARFDIE from DIERef (and handling if not found by ReportInvalidDIERef)
but the callers are already served DWARFDIE which they need.
In some cases the DWARFDIE had to be fetched both by callee (DWARFIndex
implementation) and caller.
Differential Revision: https://reviews.llvm.org/D77970
It is an obvious part of D77326.
It removes some needless deep indentation and some redundant statements.
It prepares the code for a more clean next patch - DWARF index callbacks
in D77327.
Summary:
When we added support for type units in dwo files, we changed the
"manual" dwarf index to index _all_ dwarf units in the dwo file instead
of just the split unit belonging to our skeleton unit. This was fine for
dwo files, as they contain only a single compile units and type units do
not have a split type unit which would point to them.
However, this does not work for dwp files because, these files do
contain multiple split compile units, and the current approach means
that each unit gets indexed multiple times (once for each split unit =>
n^2 complexity).
This patch teaches the manual dwarf index to treat dwp files specially.
Any type units in the dwp file added to the main list of compile units
and indexed with them in a single batch. Split compile units in dwp
files are still indexed as a part of their skeleton unit -- this is done
because we need the DW_AT_language attribute from the skeleton unit to
index them properly.
Handling of dwo files remains unchanged -- all units (type and skeleton)
are indexed when we reach the dwo file through the split unit.
Reviewers: clayborg, JDevlieghere, aprantl
Subscribers: arphaman, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D74964
