Remove support for obfuscated bitcode in dsymutil and the DWARF linker.
We no longer support bitcode submissions and the obfuscation support has
been removed from the rest of the compiler.
rdar://123863918
Remove support for obfuscated bitcode in dsymutil and the DWARF linker.
We no longer support bitcode submissions and the obfuscation support has
been removed from the rest of the compiler.
rdar://123863918
The base class llvm::ThreadPoolInterface will be renamed
llvm::ThreadPool in a subsequent commit.
This is a breaking change: clients who use to create a ThreadPool must
now create a DefaultThreadPool instead.
DWARFLinkerImpl::DWARFLinkerImpl initializes
DebugStrStrings/DebugLineStrStrings/CommonSections using GlobalData
but GlobalData is initialized after the three members.
Move GlobalData before.
Fix#81110
This implements the ideas discussed in [1].
To summarize, this commit changes AsmPrinter so that it outputs
DW_IDX_parent information for debug_name entries. It will enable
debuggers to speed up queries for fully qualified types (based on a
DWARFDeclContext) significantly, as debuggers will no longer need to
parse the entire CU in order to inspect the parent chain of a DIE.
Instead, a debugger can simply take the parent DIE offset from the
accelerator table and peek at its name in the debug_info/debug_str
sections.
The implementation uses two types of DW_FORM for the DW_IDX_parent
attribute:
1. DW_FORM_ref4, which points to the accelerator table entry for the
parent.
2. DW_FORM_flag_present, when the entry has a parent that is not in the
table (that is, the parent doesn't have a name, or isn't allowed to be
in the table as per the DWARF spec). This is space-efficient, since it
takes 0 bytes.
The implementation works by:
1. Changing how abbreviations are encoded (so that they encode which
form, if
any, was used to encode IDX_Parent)
2. Creating an MCLabel per accelerator table entry, so that they may be
referred by IDX_parent references.
When all patches related to this are merged, we are able to show that
evaluating an expression such as:
```
lldb --batch -o 'b CodeGenFunction::GenerateCode' -o run -o 'expr Fn' -- \
clang++ -c -g test.cpp -o /dev/null
```
is far faster: from ~5000 ms to ~1500ms.
Building llvm-project + clang with and without this patch, and looking
at its impact on object file size:
```
ls -la $(find build_stage2_Debug_idx_parent_assert_dwarf5 -name \*.cpp.o) | awk '{s+=$5} END {printf "%\047d\n", s}'
11,507,327,592
-la $(find build_stage2_Debug_no_idx_parent_assert_dwarf5 -name \*.cpp.o) | awk '{s+=$5} END {printf "%\047d\n", s}'
11,436,446,616
```
That is, an increase of 0.62% in total object file size.
Looking only at debug_names:
```
$stage1_build/bin/llvm-objdump --section-headers $(find build_stage2_Debug_idx_parent_assert_dwarf5 -name \*.cpp.o) | grep __debug_names | awk '{s+="0x"$3} END {printf "%\047d\n", s}'
440,772,348
$stage1_build/bin/llvm-objdump --section-headers $(find build_stage2_Debug_no_idx_parent_assert_dwarf5 -name \*.cpp.o) | grep __debug_names | awk '{s+="0x"$3} END {printf "%\047d\n", s}'
369,867,920
```
That is an increase of 19%.
DWARF Linkers need to be changed in order to support this. This commit
already brings support to "base" linker, but it does not attempt to
modify the parallel linker. Accelerator entries refer to the
corresponding DIE offset, and this patch also requires the parent DIE
offset -- it's not clear how the parallel linker can access this. It may
be obvious to someone familiar with it, but it would be nice to get help
from its authors.
[1]:
https://discourse.llvm.org/t/rfc-improve-dwarf-5-debug-names-type-lookup-parsing-speed/74151/
This patch is extracted from #74725.
The DwarfStreamer interface looks overcomplicated and has unnecessary
dependencies. This patch avoids creation of DwarfStreamer by DWARFLinker and
simplifies interface.
It was noted that new DWARFLinker libraries do not follow naming
agreement -
https://github.com/llvm/llvm-project/pull/75925#issuecomment-1883301659
This patch rename libraries to match with the agreement.
Rename LLVMDWARFLinkerBase library into the LLVMDWARFLinker. Rename
LLVMDWARFLinker library into the LLVMDWARFLinkerClassic. Correct include
path according to the new directory structure.
This patch creates DWARFLinkerBase library, places DWARFLinker code into
DWARFLinker\Classic, places DWARFLinkerParallel into DWARFLinker\Parallel.
updates BOLT to use new library. This patch is NFC.
There was a strange and seemingly unncessary empty string optimization
in NonRelocatableStringPool that I had to remove in order to support
empty strings in the line_str string table, without unconditionally
forcing an empty string to be added to every debug_line_str table.
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
Enable Type Units with DWARF5 accelerator tables for monolithic DWARF.
Implementation relies on linker to tombstone offset in LocalTU list to
-1 when
it deduplciates type units using COMDAT.
I'm assuming that because sys::path::append defaults to native path
this fails. Since dsymutil is a Darwin tool, we can just hardcode the
UNIX path separator.
Dsymutil already avoids copying textual Swift interface files from the
SDK, since any consumer would have to have a matching SDK installed
anyway. It should also do the same thing with interfaces found in the
toolchain itself, which includes the compiler built-in libraries such as
Swift (=the standard library), and _Concurrency.
rdar://117881604
This is pre-cursor patch to enabling type units with DWARF5 acceleration
tables.
With this change it allows for entries to contain offsets directly, this
way type
units do not need to be preserved until .debug_names is written out.
This reverts commit 122c89b271af30b86536cad7bac64ea9c56615ed. Change does not build, with errors such as:
In file included from ../llvm-project/llvm/tools/dsymutil/DebugMap.h:24,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
../llvm-project/llvm/tools/dsymutil/RelocationMap.h:60:17: error: declaration of ‘llvm::dsymutil::SymbolMapping llvm::dsymutil::ValidReloc::SymbolMapping’ changes meaning of ‘SymbolMapping’ [-fpermissive]
60 | SymbolMapping SymbolMapping;
| ^~~~~~~~~~~~~
../llvm-project/llvm/tools/dsymutil/RelocationMap.h:36:8: note: ‘SymbolMapping’ declared here as ‘struct llvm::dsymutil::SymbolMapping’
36 | struct SymbolMapping {
| ^~~~~~~~~~~~~
In file included from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
../llvm-project/llvm/tools/dsymutil/DebugMap.h:198:32: error: declaration of ‘std::optional<llvm::dsymutil::RelocationMap> llvm::dsymutil::DebugMapObject::RelocationMap’ changes meaning of ‘RelocationMap’ [-fpermissive]
198 | std::optional<RelocationMap> RelocationMap;
| ^~~~~~~~~~~~~
In file included from ../llvm-project/llvm/tools/dsymutil/DebugMap.h:24,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
from ../llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
../llvm-project/llvm/tools/dsymutil/RelocationMap.h:76:7: note: ‘RelocationMap’ declared here as ‘class llvm::dsymutil::RelocationMap’
76 | class RelocationMap {
| ^~~~~~~~~~~~~
This adds support in dsymutil for mergeable libraries [1].
dsymutil reads a new stab emitted by ld, allowing it to operate on
dynamic libraries instead of object files. It also now loads the DWARF
files associated to the libraries, and build the debug map for each
binary from the list of symbols exported by the library. For each Debug
Map Object, there is a new associated Relocation Map which is serialized
from the information retrieved in the original debug_info (or
debug_addr) section of the .o file.
The final DWARF file has multiple compile units, so the offsets
information of the relocations are adjusted relatively to the compile
unit they will end up belonging to, inside the final linked DWARF file.
[1] https://developer.apple.com/documentation/xcode/configuring-your-project-to-use-mergeable-libraries
Differential revision: https://reviews.llvm.org/D158124
Reduce memory usage by only extract unit DIEs when cloning clang
modules. We don't need the full debug info yet at this stage. This
reduces peak memory usage of dsymutil when linking the swift driver by
multiple gigabytes.
rdar://117156180
This patch is extracted from D96035, it adds support for the accelerator
tables to the DWARFLinkerParallel functionality.
Differential Revision: https://reviews.llvm.org/D154793
Remove the paper trail warning from dsymutil and the DWARF linker. The
original purpose of this functionality was to leave a paper trail in the
output artifact about missing object files. The current implementation
however has diverged and is the source of a pretty serious bug:
- In the debug map parser, missing object files are not the only
warnings we emit. When paper trail warnings are enabled, all of them end
up in the dSYM which wasn't the goal.
- When warnings are associated with a object file in the debug map, it
is skipped by the DWARF linker. This only makes sense if the object file
is missing and is obviously incorrect for any other type of warning
(such as a missing symbol).
The combination of the two means that we can generate broken DWARF when
the feature is enabled. AFAIK it was only used by Apple and nobody I
spoke to has relied on it, so rather than fixing the broken behavior I
propose we remove it.
The DWARFLinker library has code to identify ObjC selector names, which is used
by the debug linker to generate accelerator table entries. In the future, we
would like the DWARF verifier to also have access to such code, so that it can
identify these names when verifying accelerator tables (e.g. debug_names).
This patch follows the same intent of D155723, where we also moved code
generating simplified template names.
Since this is moving code around and changing the log, we also replace raw
pointer manipulation with the more expressive
StringRef::{drop_front,take_front,...} methods.
We also change a test so that it verifies its output, and that requires having
dsymutil not write to stdout.
Differential Revision: https://reviews.llvm.org/D158980
This patch is extracted from D96035, it adds support for the existing
DWARFLinker functionality. What is not supported yet:
1. Types deduplication(--odr mode).
2. Modules deduplication.
3. Generation of index tables.
Reland2: temporarily disabled call to "--linker llvm" for tls-variable.test
and location-expression.test as it does not work properly on bigendian
architecture.
Differential Revision: https://reviews.llvm.org/D153268
This reverts commit 0229dd0626b2538c78ebbd2b1bd44c31fbf1cdec.
This introduces two test failures on s390x.
tools/dsymutil/X86/location-expression.test:
warning: cann't load line table.
note: while processing CU1
/builddir/build/BUILD/llvm-18.0.0.src/test/tools/dsymutil/X86/location-expression.test:20:10: error: CHECK: expected string not found in input
# CHECK: DW_AT_name{{.*}}"CU1"
^
<stdin>:34:32: note: scanning from here
0x0000000b: DW_TAG_compile_unit [1] *
^
<stdin>:37:2: note: possible intended match here
DW_AT_name [DW_FORM_strp] ( .debug_str[0x09000000] = )
^
tools/dsymutil/X86/tls-variable.test:
warning: cann't load line table.
note: while processing CU1
/builddir/build/BUILD/llvm-18.0.0.src/test/tools/dsymutil/X86/tls-variable.test:19:10: error: CHECK: expected string not found in input
# CHECK: DW_AT_name{{.*}}"CU1"
^
<stdin>:26:32: note: scanning from here
0x0000000b: DW_TAG_compile_unit
^
<stdin>:29:2: note: possible intended match here
DW_AT_name ()
^
This patch is extracted from D96035, it adds support for the existing
DWARFLinker functionality. What is not supported yet:
1. Types deduplication(--odr mode).
2. Modules deduplication.
3. Generation of index tables.
Differential Revision: https://reviews.llvm.org/D153268
This patch is extracted from D96035, it adds support for the existing
DWARFLinker functionality. What is not supported yet:
1. Types deduplication(--odr mode).
2. Modules deduplication.
3. Generation of index tables.
Differential Revision: https://reviews.llvm.org/D153268
LLDB can benefit from having the base name of functions (i.e. without any
template parameters) as an entry into accelerator tables pointing back in the
DIE for the corresponding function specialization. In fact, some LLDB
functionality is only possible when those entries are present.
The DWARFLinker has been adding such entries for a while now, both with
apple_names and with debug_names. However, this has two side effects:
1. Some LLDB functionality is only possible when dsym bundles are present (i.e.
the linker touched the debug info).
2. The DWARFVerifier doesn't accept debug_name sections created by the linker,
as such names are (usually) neither the AT_name nor the AT_linkage_name of the
DIE.
Based on recent discussion [1], and because the DWARF 5 spec says that:
> A producer may choose to implement additional rules for what names are placed
> in the index
This patch relaxes the checks on the verifier to allow for simplified template
names in the accelerator table. To do so, we move some helper functions from
DWARFLinker into the core lib debug info. This addresses the point 2) above.
This patch also enables addressing point 1) in the future, since the helper
function is now visible to other parts of LLVM.
[1]: https://github.com/llvm/llvm-project/issues/58362
Differential Revision: https://reviews.llvm.org/D155723
instead of DW_ELE_start_length in debug_rnglists section
This patch tries to reduce the size of the debug_rnglist section by
replacing the DW_RLE_start_length opcodes currently emitted by dsymutil
in favor of using DW_RLE_base_addressx + DW_RLE_offset_pair instead.
The DW_RLE_start_length is one AddressSize followed by a ULEB per entry,
whereas, the DW_RLE_base_addressx + DW_RLE_offset_pair will use one ULEB
for the base address, and then the DW_RLE_offset_pair is a pair of
ULEBs. This will be more efficient.
Differential Revision: https://reviews.llvm.org/D156166
With https://reviews.llvm.org/D154638, the ability to emit a .debug_addr
section has been added to dsymutil. With this, instead of emitting a
DW_LLE_baseaddr in the .debug_loclist section, a DW_LLE_baseaddrx can be
emitted instead, which will allow for more indirection.
Differetial Revision: https://reviews.llvm.org/D155724
If the DW_AT_addr_base is not set, dsymutil should not copy any addrx
attributes into the cloned DIE, which will result in no .debug_addr
section being emitted.
Differential Revision: https://reviews.llvm.org/D155666
DWARF5 has support for DW_FORM_addrx, which can be useful for space
savings, but it needs a .debug_addr section to be used. dsymutil does
not have the ability to emit a debug_addr section currently. This patch
adds support for that.
Differential Revision: https://reviews.llvm.org/D154638
The D153869 patch stopped storing stripped template names into
.debug_names accelerator table. This patch restores original
behavior as lldb relies on presenting stripped names. Changes for
llvm-dwarfdump would be done as a separate patch.
Differential Revision: https://reviews.llvm.org/D155070
This patch simplifies line table generation. It removes global
array of all units ranges(RangesTy &getValidAddressRanges()).
The comment says that global array of all units ranges is necessary
to handle corner cases inside line table rows. Removing that
special handling shows that its current usage is handling of
"end of range case" which is already handled correctly
(without special handling). .debug_line tables for clang binary
built with and without this patch are equal.
Differential Revision: https://reviews.llvm.org/D154288
DWARFLinker has a compatibility feature with dsymutil-classic.
It may keep location expression attribute even if does not
reference live address. Current llvm-dwarfdump --verify
reports a error if variable references an address but is not
added into the .debug_names table.
error: Name Index @ 0x0: Entry for DIE @ 0xf35 (DW_TAG_variable) with name seed missing.
DW_TAG_variable
DW_AT_name ("seed")
DW_AT_type (0x00000000000047b7 "uint64_t")
DW_AT_location (DW_OP_addr 0x9ff8) <<<< dead address
DWARFLinker does not add the variable into .debug_names table
because it references dead address. To have a valid variable and
consistent accelerator table it is necessary to remove location expression
referencing dead address. This patch removes dsymutil-classic
compatibilty feature.
Differential Revision: https://reviews.llvm.org/D153988
DWARFLinker puts three names for subprograms into the .apple_names and
.debug_names: short name, linkage name, name without template parameters.
DW_TAG_subprogram
DW_AT_linkage_name "_Z3fooIcEvv"
DW_AT_name "foo<char>"
short name: "foo<char>"
linkage name: "_Z3fooIcEvv"
name without template parameters: "foo"
DWARFv5 does not require stripping template parameters for subprogram name.
Current llvm-dwarfdump --verify reports the error if names stored in
accelerator table do not match with DIE name(name with stripped template
parameters stored in accelerator table does not match with original DIE name).
This patch does not store name without template parameters into the .debug_names table.
Differential Revision: https://reviews.llvm.org/D153869
In preparation for removing the `#include "llvm/ADT/StringExtras.h"`
from the header to source file of `llvm/Support/Error.h`, first add in
all the missing includes that were previously included transitively
through this header.
This patch tries to reduce the size of the debug_loclist section by
replacing the DW_LLE_start_length opcodes currently emitted by dsymutil
in favor of using DW_LLE_base_address + DW_LLE_offset_pair instead.
The DW_LLE_start_length is one AddressSize followed by a ULEB per entry,
whereas, the DW_LLE_base_address + DW_LLE_offset_pair will use one
AddressSize for the base address, and then the DW_LLE_offset_pair is a
pair of ULEBs. This will be more efficient where a loclist fragment has
many entries.
Differential Revision: https://reviews.llvm.org/D153080
