This PR adds support in the gSYM format for call site information and
adds support for loading call sites from a YAML file. The support for
YAML input is mostly for testing purposes - so we have a way to test the
functionality.
Note that this data is not currently used in the gSYM tooling - the
logic to use call sites will be added in a later PR.
The reason why we need call site information in gSYM files is so that we
can support better call stack function disambiguation in the case where
multiple functions have been merged due to optimization (linker ICF).
When resolving a merged function on the callstack, we can use the call
site information of the calling function to narrow down the actual
function that is being called, from the set of all merged functions.
See [this
RFC](https://discourse.llvm.org/t/rfc-extending-gsym-format-with-call-site-information-for-merged-function-disambiguation/80682)
for more details on this change.
raw_string_ostream::flush() is essentially a no-op (also specified in docs).
Don't call it in tests that aren't meant to test 'raw_string_ostream' itself.
p.s. remove a few redundant calls to raw_string_ostream::str()
GsymUtil, like DwarfDump --verify, spews a *lot* of data necessary to
understand/diagnose issues with DWARF data. The trouble is that the kind
of information necessary to make the messages useful also makes them
nearly impossible to easily categorize. I put together a similar output
categorizer (https://github.com/llvm/llvm-project/pull/79648) that will
emit a summary of issues identified at the bottom of the (very verbose)
output, enabling easier tracking of issues as they arise or are
addressed.
There's a single output change, where a message "warning: Unable to
retrieve DWO .debug_info section for some object files. (Remove the
--quiet flag for full output)" was being dumped the first time it was
encountered (in what looks like an attempt to make something easily
grep-able), but rather than keep the output in the same order, that
message is now a 'category' so gets emitted at the end of the output.
The test 'tools/llvm-gsymutil/X86/elf-dwo.yaml' was changed to reflect
this difference.
---------
Co-authored-by: Kevin Frei <freik@meta.com>
llvm-gsymutil allows address ranges to overlap. There was a bug where if
we had debug info for a function with a range like [0x100-0x200) and a
symbol at the same start address yet with a larger range like
[0x100-0x300), we would randomly get either only information from the
first or second entry. This could cause lookups to fail due to the way
the binary search worked.
This patch makes sure that when lookups happen we find the first address
table entry that can match an address, and also ensures that we always
select the first FunctionInfo that could match. FunctionInfo entries are
sorted such that the most debug info rich entries come first. And if we
have two ranges that have the same start address, the smaller range
comes first and the larger one comes next. This patch also adds the
ability to iterate over all function infos with the same start address
to always find a range that contains the address.
Added a unit test to test this functionality that failed prior to this
fix and now succeeds.
Also fix an issue when dumping an entire GSYM file that has duplicate address entries where it used to always print out the binary search match for the FunctionInfo, not the actual data for the address index.
DWARF produced by LTO and BOLT can sometimes be broken where file
indexes are beyond the end of the line table's file list in the
prologue. This patch allows llvm-gsymutil to convert this DWARF without
crashing, and emits errors when:
line table contains entries with an invalid file index (line entry will
be removed) inline functions that have invalid DW_AT_call_file file
indexes when there are no line table entries for a function and we fall
back to making a single line table entry from the functions
DW_AT_decl_file/DW_AT_decl_line attributes, we make sure the
DW_AT_decl_file attribute is valid before emitting it.
Fix line table lookups in line tables with multiple lines with the same
address.
Compilers emit line tables that have multiple line table entries with
the same address. When doing lookups, we always need to use the last
line entry if a lookup address matches the address of one or more line
entries. This is because the size of an address range for a line uses
the next line entry to figure out how big the current line entry is. If
the next line entry has the same address, that means the current line
entry has a size of zero, so no bytes correspond to the line entry.
This patch ensures that lookups will always pick the last matching line
entry when the lookup address matches more than one line entry.
Previous to this fix, if we had a DW_TAG_subprogram that had a
DW_AT_linkage_name that was empty, it would attempt to use this name
which would cause an error to be emitted when saving the gsym file to
disk:
error: DWARF conversion failed: : attempted to encode invalid
FunctionInfo object
This patch fixes this issue and adds a unit test case.
Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an enum.
This patch replaces llvm::support::{big,little,native} with
llvm::endianness::{big,little,native}.
Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
support::endianness with llvm::endianness.
Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
llvm::support::endianness with llvm::endianness.
system_endianness() just returns llvm::endianness::native, a
compile-time constant equivalent to std::native in C++20. This patch
deprecates system_endianness() while replacing all invocations of
system_endianness() with llvm::endianness::native.
While we are at it, this patch replaces
llvm::support::endianness::{big,little} with
llvm::endianness::{big,little} in those statements that happen to call
system_endianness(). It does not go out of its way to replace other
occurrences of llvm::support::endianness::{big,little}.
llvm-gsymutil would emit errors about address ranges for DW_TAG_inlined_subroutine DIEs whose address range didn't exist in the parent inline information. When a DW_TAG_subprogram DIE has more than one address range with a DW_AT_ranges attribute, we emit multiple FunctionInfo objets, one for each range of a function. When we parsed the inline information, it might have inline contribution that appear in any of the function's ranges, and if we were parsing the first range of a function, all inline entries that appeared in other valid ranges of the functions would end up emitting error messages. This patch fixes this by always passing down the full list of ranges, even if they aren't being used in the parse of the information. This eliminates reporting of errors when we shouldn't have been emitting error messages. Added a test to track this and ensure this doesn't regress.
Also we don't warn if we end up with empty inline information if the only top level inline function have been elided where the high and low PC values are the same which indicates that the inline function was elided.
Differential Revision: https://reviews.llvm.org/D157669
The GSYM code alwasy logging to streams even in quiet mode. When in quiet mode we would use the "nulls()" stream to avoid logging to the terminal, but this still caused logging functions to be called on DWARFDie objects and other messages which were quite expensive and not needed if we weren't logging anything. This patch switches some logs in performant areas to be "raw_ostream *" values and if the ostream pointer is NULL, then we don't call the expensive logging functions on DWARFDie and other objects which will improve performance.
Differential Revision: https://reviews.llvm.org/D157466
llvm-gsymutil was maintaining an address ranges collection behind a mutex and having the multi-threaded code access this and hold the mutex was causing slowdown when converting DWARF to GSYM. This patch does the following:
- removes the "Ranges" variable from the GsymCreator and any functions and places that used it
- clients don't try to detect if a function has been added for an address range, we now remove any inferior copies of information in the GsymCreator::finalize() routine as was done before, we just have more items to remove, though performance is greator due to less mutex thread locking
- after I started adding all of the inferior funtion info objects the previous patch that tried to remove infrior debug info had bugs in it, so I replace the removeIfBinary() function in GsymCreator with a more efficient and easier to debug way to do things which copies items from the GsymCreator::Funcs into a new vector of FunctionInfo objects and then replaces GsymCreator::Funcs at the end.
- Sorting of FunctionInfo objects has been modified to also compare InlineInfo objects. We found cases where LTO was ruining inline function address ranges and we ended up with a variety of FunctionInfo objects for the same range that had varying amounts of valid debug info. This patch now ensure that two function info objects with different inline info for the same function address range, the best one will be picked to ensure the greatest fidelity.
- If we detect that a DW_TAG_subprogram has inline functions and after parsing it, we don't end up with any valid inline information, we set the optional to std::nullopt to avoid emitting empty inline information and wasting space.
My tests show a 200% perf increase on M1 macs and a 100% performance increase on linux machines for the same complex large DWARF input binary.
Differential Revision: https://reviews.llvm.org/D156773
If a function contains inline function ranges whose address ranges are not contained in the parent scope, then emit an error message and omit them from the final GSYM. Prior to this we would only test if an inline function's address range was within the concrete function's ranges. If we ran into a case where the inline range was within the function's ranges, but not within one of the parent inline function's ranges, then we would fail to produce a GSYM file and exit with an error.
The current code will emit full details on invalid inline ranges as they are being parsed and will omit any bad ranges from the final GSYM file.
Differential Revision: https://reviews.llvm.org/D155254
Some workflows can generate large GSYM files and sharding GSYM files into segments can help some performant workflows that can take advantage of smaller GSYM files. This patch add a new --segment-size option to llvm-gsymutil. This option can specify a rough size in bytes of how large each segment should be.
Segmented GSYM files contain only the strings and files that are needed for the FunctionInfo objects that are added to each shard. The output file path gets the first address of the first contained function info appended as a suffix to the filename. If a base address of an image is set in the GsymCreator, then all segments will use this same base address which allows lookups for symbolication to happen correctly when the image has been slid in memory.
Code has been addeed to refactor and re-use methods within the GsymCreator to allow for segments to be created easily and tested.
Example of segmenting GSYM files:
$ llvm-gsymutil --convert llvm-gsymutil.dSYM -o llvm-gsymutil.gsym --segment-size 10485760
$ ls -l llvm-gsymutil.gsym-*
-rw-r--r-- 1 gclayton staff 10485839 Feb 9 10:45 llvm-gsymutil.gsym-0x1000030c0
-rw-r--r-- 1 gclayton staff 10485765 Feb 9 10:45 llvm-gsymutil.gsym-0x100668888
-rw-r--r-- 1 gclayton staff 10485881 Feb 9 10:45 llvm-gsymutil.gsym-0x100c948b8
-rw-r--r-- 1 gclayton staff 10485954 Feb 9 10:45 llvm-gsymutil.gsym-0x101659e70
-rw-r--r-- 1 gclayton staff 10485792 Feb 9 10:45 llvm-gsymutil.gsym-0x1022b1dc0
-rw-r--r-- 1 gclayton staff 10485889 Feb 9 10:45 llvm-gsymutil.gsym-0x102a18b10
-rw-r--r-- 1 gclayton staff 10485893 Feb 9 10:45 llvm-gsymutil.gsym-0x1030b05d0
-rw-r--r-- 1 gclayton staff 10485802 Feb 9 10:45 llvm-gsymutil.gsym-0x1037caaac
-rw-r--r-- 1 gclayton staff 10485781 Feb 9 10:45 llvm-gsymutil.gsym-0x103e767a0
-rw-r--r-- 1 gclayton staff 10485832 Feb 9 10:45 llvm-gsymutil.gsym-0x10452d0d4
-rw-r--r-- 1 gclayton staff 10485782 Feb 9 10:45 llvm-gsymutil.gsym-0x104b93310
-rw-r--r-- 1 gclayton staff 6255785 Feb 9 10:45 llvm-gsymutil.gsym-0x10526bf34
Differential Revision: https://reviews.llvm.org/D145448
Some workflows can generate large GSYM files and sharding GSYM files into segments can help some performant workflows that can take advantage of smaller GSYM files. This patch add a new --segment-size option to llvm-gsymutil. This option can specify a rough size in bytes of how large each segment should be.
Segmented GSYM files contain only the strings and files that are needed for the FunctionInfo objects that are added to each shard. The output file path gets the first address of the first contained function info appended as a suffix to the filename. If a base address of an image is set in the GsymCreator, then all segments will use this same base address which allows lookups for symbolication to happen correctly when the image has been slid in memory.
Code has been addeed to refactor and re-use methods within the GsymCreator to allow for segments to be created easily and tested.
Example of segmenting GSYM files:
$ llvm-gsymutil --convert llvm-gsymutil.dSYM -o llvm-gsymutil.gsym --segment-size 10485760
$ ls -l llvm-gsymutil.gsym-*
-rw-r--r-- 1 gclayton staff 10485839 Feb 9 10:45 llvm-gsymutil.gsym-0x1000030c0
-rw-r--r-- 1 gclayton staff 10485765 Feb 9 10:45 llvm-gsymutil.gsym-0x100668888
-rw-r--r-- 1 gclayton staff 10485881 Feb 9 10:45 llvm-gsymutil.gsym-0x100c948b8
-rw-r--r-- 1 gclayton staff 10485954 Feb 9 10:45 llvm-gsymutil.gsym-0x101659e70
-rw-r--r-- 1 gclayton staff 10485792 Feb 9 10:45 llvm-gsymutil.gsym-0x1022b1dc0
-rw-r--r-- 1 gclayton staff 10485889 Feb 9 10:45 llvm-gsymutil.gsym-0x102a18b10
-rw-r--r-- 1 gclayton staff 10485893 Feb 9 10:45 llvm-gsymutil.gsym-0x1030b05d0
-rw-r--r-- 1 gclayton staff 10485802 Feb 9 10:45 llvm-gsymutil.gsym-0x1037caaac
-rw-r--r-- 1 gclayton staff 10485781 Feb 9 10:45 llvm-gsymutil.gsym-0x103e767a0
-rw-r--r-- 1 gclayton staff 10485832 Feb 9 10:45 llvm-gsymutil.gsym-0x10452d0d4
-rw-r--r-- 1 gclayton staff 10485782 Feb 9 10:45 llvm-gsymutil.gsym-0x104b93310
-rw-r--r-- 1 gclayton staff 6255785 Feb 9 10:45 llvm-gsymutil.gsym-0x10526bf34
Differential Revision: https://reviews.llvm.org/D143793
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
The bug was introduced when the AddressRange class was no longer able to modify the End address directly and the entire range of the .text address range that contained the trailing empty symbol was replaced. There was no unit test for this, so it wasn't caught. I fixed the bug and added a unit test for it.
The effects of this bug are serious as the AddressOffsetSize in the header would be incorrectly calculated and an invalid GSYM would be created.
Differential Revision: https://reviews.llvm.org/D127811
llvm-gsymutil has an implementation of AddressRange and AddressRanges
classes. That implementation might be reused in other parts of llvm.
This patch moves AddressRange and AddressRanges classes into llvm/ADT.
Differential Revision: https://reviews.llvm.org/D124350
Some files still contained the old University of Illinois Open Source
Licence header. This patch replaces that with the Apache 2 with LLVM
Exception licence.
Differential Revision: https://reviews.llvm.org/D107528
Symbol tables can have symbols with no size in mach-o files that were failing to get combined into a single entry. This resulted in many duplicate entries for the same address and made gsym files larger.
Differential Revision: https://reviews.llvm.org/D105068
The Length, AbbrOffset and Values fields of the debug_info section are
optional. This patch helps remove them and simplify test cases.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D86857
This patch adds support for emitting multiple abbrev tables. Currently,
compilation units will always reference the first abbrev table.
Reviewed By: jhenderson, labath
Differential Revision: https://reviews.llvm.org/D86194
'InitialLength' is replaced with 'Format' (DWARF32 by default) and 'Length' in this patch.
Besides, test cases for DWARFv4 and DWARFv5, DWARF32 and DWARF64 is
added.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D82622
Summary:
The Offset provides the offset within the function in a SourceLocation struct. This allows us to show the byte offset within a function. We also track offsets within inline functions as well. Updated the lookup tests to verify the offset for functions and inline functions.
0x1000: main + 32 @ /tmp/main.cpp:45
Reviewers: labath, aadsm, serhiy.redko, jankratochvil, xiaobai, wallace, aprantl, JDevlieghere
Subscribers: hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D74680
Summary:
The DWARF transformer is added as a class so it can be unit tested fully.
The DWARF is converted to GSYM format and handles many special cases for functions:
- omit functions in compile units with 4 byte addresses whose address is UINT32_MAX (dead stripped)
- omit functions in compile units with 8 byte addresses whose address is UINT64_MAX (dead stripped)
- omit any functions whose high PC is <= low PC (dead stripped)
- StringTable builder doesn't copy strings, so we need to make backing copies of strings but only when needed. Many strings come from sections in object files and won't need to have backing copies, but some do.
- When a function doesn't have a mangled name, store the fully qualified name by creating a string by traversing the parent decl context DIEs and then. If we don't do this, we end up having cases where some function might appear in the GSYM as "erase" instead of "std::vector<int>::erase".
- omit any functions whose address isn't in the optional TextRanges member variable of DwarfTransformer. This allows object file to register address ranges that are known valid code ranges and can help omit functions that should have been dead stripped, but just had their low PC values set to zero. In this case we have many functions that all appear at address zero and can omit these functions by making sure they fall into good address ranges on the object file. Many compilers do this when the DWARF has a DW_AT_low_pc with a DW_FORM_addr, and a DW_AT_high_pc with a DW_FORM_data4 as the offset from the low PC. In this case the linker can't write the same address to both the high and low PC since there is only a relocation for the DW_AT_low_pc, so many linkers tend to just zero it out.
Reviewers: aprantl, dblaikie, probinson
Subscribers: mgorny, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D74450
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
Lookup functions are designed to not fully decode a FunctionInfo, LineTable or InlineInfo, they decode only what is needed into a LookupResult object. This allows lookups to avoid costly memory allocations and avoid parsing large amounts of information one a suitable match is found.
LookupResult objects contain the address that was looked up, the concrete function address range, the name of the concrete function, and a list of source locations. One for each inline function, and one for the concrete function. This allows one address to turn into multiple frames and improves the signal you get when symbolicating addresses in GSYM files.
Reviewers: labath, aprantl
Subscribers: mgorny, hiraditya, llvm-commits, lldb-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D70993
This patch adds the ability to create GSYM files with GsymCreator, and read them with GsymReader. Full testing has been added for both new classes.
This patch differs from the original patch https://reviews.llvm.org/D53379 in that is uses a StringTableBuilder class from llvm instead of a custom version. Support for big and little endian files has been added. If the endianness matches the current host, we use efficient extraction for the header, address table and address info offset tables.
Differential Revision: https://reviews.llvm.org/D68744
llvm-svn: 374381
This patch adds the llvm::gsym::Header class which appears at the start of a stand alone GSYM file, or in the first bytes of the GSYM data in a GSYM section within a file. Added encode and decode methods with full error handling and full tests.
Differential Revision: https://reviews.llvm.org/D67666
llvm-svn: 372149
This patch adds encoding and decoding of the FunctionInfo objects along with full error handling and tests. Full details of the FunctionInfo encoding format appear in the FunctionInfo.h header file.
Differential Revision: https://reviews.llvm.org/D67506
llvm-svn: 372135
This patch adds the ability to create a gsym::LineTable object, populate it, encode and decode it and test all functionality.
The full format of the LineTable encoding is specified in the header file LineTable.h.
Differential Revision: https://reviews.llvm.org/D66602
llvm-svn: 371657
This patch adds the ability to encode and decode InlineInfo objects and adds test coverage. Error handling is introduced in the encoding and decoding which will be used from here on out for remaining patches.
Differential Revision: https://reviews.llvm.org/D66600
llvm-svn: 370936
The full GSYM patch started with: https://reviews.llvm.org/D53379
This patch add the ability to encode data using the new llvm::gsym::FileWriter class.
FileWriter is a simplified binary data writer class that doesn't require targets, target definitions, architectures, or require any other optional compile time libraries to be enabled via the build process. This class needs the ability to seek to different spots in the binary data that it produces to fix up offsets and sizes in GSYM data. It currently uses std::ostream over llvm::raw_ostream because llvm::raw_ostream doesn't support seeking which is required when encoding and decoding GSYM data.
AddressRange objects are encoded and decoded to be relative to a base address. This will be the FunctionInfo's start address if the AddressRange is directly contained in a FunctionInfo, or a base address of the containing parent AddressRange or AddressRanges. This allows address ranges to be efficiently encoded using ULEB128 encodings as we encode the offset and size of each range instead of full addresses. This also makes encoded addresses easy to relocate as we just need to relocate one base address.
Differential Revision: https://reviews.llvm.org/D63828
llvm-svn: 369587
Delete unnecessary getters of AddressRange.
Simplify AddressRange::size(): Start <= End check should be checked in an upper layer.
Delete isContiguousWith() that doesn't make sense.
Simplify AddressRanges::insert. Delete commented code. Fix it when more than 1 ranges are to be deleted.
Delete trailing newline.
llvm-svn: 364637
