The section headers for XCOFF files have a subtype flag for Dwarf
sections. This PR updates obj2yaml, yaml2obj, and llvm-readobj so that
they recognize the subtype.
yaml2obj creates invalid object files even when the input was created by
obj2yaml using a valid object file. On the other hand, yaml2obj is used
to intentionally create invalid object files for testing purposes.
This update balances using specified input values when provided and
computing file offsets and sizes if necessary.
XCOFF encodes a symbol type and alignment in a single 8-bit field. It is
easier to read and write YAML files if the fields can be specified
separately. This PR causes obj2yaml to write the fields separately and
allows yaml2obj to read either the single combined field or the separate
fields.
Today `-split-machine-functions` and `-fbasic-block-sections={all,list}`
cannot be combined with `-basic-block-sections=labels` (the labels
option will be ignored).
The inconsistency comes from the way basic block address map -- the
underlying mechanism for basic block labels -- encodes basic block
addresses
(https://lists.llvm.org/pipermail/llvm-dev/2020-July/143512.html).
Specifically, basic block offsets are computed relative to the function
begin symbol. This relies on functions being contiguous which is not the
case for MFS and basic block section binaries. This means Propeller
cannot use binary profiles collected from these binaries, which limits
the applicability of Propeller for iterative optimization.
To make the `SHT_LLVM_BB_ADDR_MAP` feature work with basic block section
binaries, we propose modifying the encoding of this section as follows.
First let us review the current encoding which emits the address of each
function and its number of basic blocks, followed by basic block entries
for each basic block.
| | |
|--|--|
| Address of the function | Function Address |
| Number of basic blocks in this function | NumBlocks |
| BB entry 1
| BB entry 2
| ...
| BB entry #NumBlocks
To make this work for basic block sections, we treat each basic block
section similar to a function, except that basic block sections of the
same function must be encapsulated in the same structure so we can map
all of them to their single function.
We modify the encoding to first emit the number of basic block sections
(BB ranges) in the function. Then we emit the address map of each basic
block section section as before: the base address of the section, its
number of blocks, and BB entries for its basic block. The first section
in the BB address map is always the function entry section.
| | |
|--|--|
| Number of sections for this function | NumBBRanges |
| Section 1 begin address | BaseAddress[1] |
| Number of basic blocks in section 1 | NumBlocks[1] |
| BB entries for Section 1
|..................|
| Section #NumBBRanges begin address | BaseAddress[NumBBRanges] |
| Number of basic blocks in section #NumBBRanges |
NumBlocks[NumBBRanges] |
| BB entries for Section #NumBBRanges
The encoding of basic block entries remains as before with the minor
change that each basic block offset is now computed relative to the
begin symbol of its containing BB section.
This patch adds a new boolean codegen option `-basic-block-address-map`.
Correspondingly, the front-end flag `-fbasic-block-address-map` and LLD
flag `--lto-basic-block-address-map` are introduced.
Analogously, we add a new TargetOption field `BBAddrMap`. This means BB
address maps are either generated for all functions in the compiling
unit, or for none (depending on `TargetOptions::BBAddrMap`).
This patch keeps the functionality of the old
`-fbasic-block-sections=labels` option but does not remove it. A
subsequent patch will remove the obsolete option.
We refactor the `BasicBlockSections` pass by separating the BB address
map and BB sections handing to their own functions (named
`handleBBAddrMap` and `handleBBSections`). `handleBBSections` renumbers
basic blocks and places them in their assigned sections.
`handleBBAddrMap` is invoked after `handleBBSections` (if requested) and
only renumbers the blocks.
- New tests added:
- Two tests basic-block-address-map-with-basic-block-sections.ll and
basic-block-address-map-with-mfs.ll to exercise the combination of
`-basic-block-address-map` with `-basic-block-sections=list` and
'-split-machine-functions`.
- A driver sanity test for the `-fbasic-block-address-map` option
(basic-block-address-map.c).
- An LLD test for testing the `--lto-basic-block-address-map` option.
This reuses the LLVM IR from `lld/test/ELF/lto/basic-block-sections.ll`.
- Renamed and modified the two existing codegen tests for basic block
address map (`basic-block-sections-labels-functions-sections.ll` and
`basic-block-sections-labels.ll`)
- Removed `SHT_LLVM_BB_ADDR_MAP_V0` tests. Full deprecation of
`SHT_LLVM_BB_ADDR_MAP_V0` and `SHT_LLVM_BB_ADDR_MAP` version less than 2
will happen in a separate PR in a few months.
This PR implements part 1 of yaml2obj for the GOFF Object File Format.
It adds support for the header and end records.
---------
Co-authored-by: Yusra Syeda <yusra.syeda@ibm.com>
There can be zero padding bytes at a section end for file alignment in
PECOFF. Exclude those padding bytes when reading the section data.
Differential Revision: https://reviews.llvm.org/D157059
For a relocation, we don't differentiate the two cases:
* the symbol index is 0
* the symbol index is non zero, the type is not STT_SECTION, and the name is empty. Clang generates such local symbols for RISC-V linker relaxation.
So we may print
```
Offset Info Type Symbol's Value Symbol's Name + Addend
000000000000001c 0000000100000039 R_RISCV_32_PCREL 0000000000000000 0
// llvm-readobj
0x1C R_RISCV_32_PCREL - 0x0
```
while GNU readelf prints "<null>", which is clearer. Let's match the GNU behavior.
Related to https://reviews.llvm.org/D81842
```
000000000000001c 0000000100000039 R_RISCV_32_PCREL 0000000000000000 <null> + 0
// llvm-readobj
0x1C R_RISCV_32_PCREL <null> 0x0
```
Reviewed By: jhenderson, kito-cheng
Differential Revision: https://reviews.llvm.org/D155353
Let Propeller use specialized IDs for basic blocks, instead of MBB number.
This allows optimizations not just prior to asm-printer, but throughout the entire codegen.
This patch only implements the functionality under the new `LLVM_BB_ADDR_MAP` version, but the old version is still being used. A later patch will change the used version.
####Background
Today Propeller uses machine basic block (MBB) numbers, which already exist, to map native assembly to machine IR. This is done as follows.
- Basic block addresses are captured and dumped into the `LLVM_BB_ADDR_MAP` section just before the AsmPrinter pass which writes out object files. This ensures that we have a mapping that is close to assembly.
- Profiling mapping works by taking a virtual address of an instruction and looking up the `LLVM_BB_ADDR_MAP` section to find the MBB number it corresponds to.
- While this works well today, we need to do better when we scale Propeller to target other Machine IR optimizations like spill code optimization. Register allocation happens earlier in the Machine IR pipeline and we need an annotation mechanism that is valid at that point.
- The current scheme will not work in this scenario because the MBB number of a particular basic block is not fixed and changes over the course of codegen (via renumbering, adding, and removing the basic blocks).
- In other words, the volatile MBB numbers do not provide a one-to-one correspondence throughout the lifetime of Machine IR. Profile annotation using MBB numbers is restricted to a fixed point; only valid at the exact point where it was dumped.
- Further, the object file can only be dumped before AsmPrinter and cannot be dumped at an arbitrary point in the Machine IR pass pipeline. Hence, MBB numbers are not suitable and we need something else.
####Solution
We propose using fixed unique incremental MBB IDs for basic blocks instead of volatile MBB numbers. These IDs are assigned upon the creation of machine basic blocks. We modify `MachineFunction::CreateMachineBasicBlock` to assign the fixed ID to every newly created basic block. It assigns `MachineFunction::NextMBBID` to the MBB ID and then increments it, which ensures having unique IDs.
To ensure correct profile attribution, multiple equivalent compilations must generate the same Propeller IDs. This is guaranteed as long as the MachineFunction passes run in the same order. Since the `NextBBID` variable is scoped to `MachineFunction`, interleaving of codegen for different functions won't cause any inconsistencies.
The new encoding is generated under the new version number 2 and we keep backward-compatibility with older versions.
####Impact on Size of the `LLVM_BB_ADDR_MAP` Section
Emitting the Propeller ID results in a 23% increase in the size of the `LLVM_BB_ADDR_MAP` section for the clang binary.
Reviewed By: tmsriram
Differential Revision: https://reviews.llvm.org/D100808
Let Propeller use specialized IDs for basic blocks, instead of MBB number.
This allows optimizations not just prior to asm-printer, but throughout the entire codegen.
This patch only implements the functionality under the new `LLVM_BB_ADDR_MAP` version, but the old version is still being used. A later patch will change the used version.
####Background
Today Propeller uses machine basic block (MBB) numbers, which already exist, to map native assembly to machine IR. This is done as follows.
- Basic block addresses are captured and dumped into the `LLVM_BB_ADDR_MAP` section just before the AsmPrinter pass which writes out object files. This ensures that we have a mapping that is close to assembly.
- Profiling mapping works by taking a virtual address of an instruction and looking up the `LLVM_BB_ADDR_MAP` section to find the MBB number it corresponds to.
- While this works well today, we need to do better when we scale Propeller to target other Machine IR optimizations like spill code optimization. Register allocation happens earlier in the Machine IR pipeline and we need an annotation mechanism that is valid at that point.
- The current scheme will not work in this scenario because the MBB number of a particular basic block is not fixed and changes over the course of codegen (via renumbering, adding, and removing the basic blocks).
- In other words, the volatile MBB numbers do not provide a one-to-one correspondence throughout the lifetime of Machine IR. Profile annotation using MBB numbers is restricted to a fixed point; only valid at the exact point where it was dumped.
- Further, the object file can only be dumped before AsmPrinter and cannot be dumped at an arbitrary point in the Machine IR pass pipeline. Hence, MBB numbers are not suitable and we need something else.
####Solution
We propose using fixed unique incremental MBB IDs for basic blocks instead of volatile MBB numbers. These IDs are assigned upon the creation of machine basic blocks. We modify `MachineFunction::CreateMachineBasicBlock` to assign the fixed ID to every newly created basic block. It assigns `MachineFunction::NextMBBID` to the MBB ID and then increments it, which ensures having unique IDs.
To ensure correct profile attribution, multiple equivalent compilations must generate the same Propeller IDs. This is guaranteed as long as the MachineFunction passes run in the same order. Since the `NextBBID` variable is scoped to `MachineFunction`, interleaving of codegen for different functions won't cause any inconsistencies.
The new encoding is generated under the new version number 2 and we keep backward-compatibility with older versions.
####Impact on Size of the `LLVM_BB_ADDR_MAP` Section
Emitting the Propeller ID results in a 23% increase in the size of the `LLVM_BB_ADDR_MAP` section for the clang binary.
Reviewed By: tmsriram
Differential Revision: https://reviews.llvm.org/D100808
These directives define per-test lit substitutions. The concept was
discussed at
<https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10>.
For example, the following directives can be inserted into a test file
to define `%{cflags}` and `%{fcflags}` substitutions with empty
initial values, which serve as the parameters of another newly defined
`%{check}` substitution:
```
// DEFINE: %{cflags} =
// DEFINE: %{fcflags} =
// DEFINE: %{check} = %clang_cc1 %{cflags} -emit-llvm -o - %s | \
// DEFINE: FileCheck %{fcflags} %s
```
The following directives then redefine the parameters before each use
of `%{check}`:
```
// REDEFINE: %{cflags} = -foo
// REDEFINE: %{fcflags} = -check-prefix=FOO
// RUN: %{check}
// REDEFINE: %{cflags} = -bar
// REDEFINE: %{fcflags} = -check-prefix=BAR
// RUN: %{check}
```
Of course, `%{check}` would typically be more elaborate, increasing
the benefit of the reuse.
One issue is that the strings `DEFINE:` and `REDEFINE:` already appear
in 5 tests. This patch adjusts those tests not to use those strings.
Our prediction is that, in the vast majority of cases, if a test
author mistakenly uses one of those strings for another purpose, the
text appearing after the string will not happen to have the syntax
required for these directives. Thus, the test author will discover
the mistake immediately when lit reports the syntax error.
This patch also expands the documentation on existing lit substitution
behavior.
Reviewed By: jhenderson, MaskRay, awarzynski
Differential Revision: https://reviews.llvm.org/D132513
If you're having trouble getting a yaml2obj macro expansion to do what
you want, it's useful to be able to print the output of the
preprocessing to see what your macros expanded to //before// going
into the YAML processing phase.
yaml2obj has its own preprocessing system which isn't the same as any
other well-known thing like cpp. So there's no way to do this macro
expansion via another tool: yaml2obj will have to do it itself.
In this commit I add an `-E` flag to yaml2obj to do that.
Differential Revision: https://reviews.llvm.org/D130981
This is a resurrection of D106421 with the change that it keeps backward-compatibility. This means decoding the previous version of `LLVM_BB_ADDR_MAP` will work. This is required as the profile mapping tool is not released with LLVM (AutoFDO). As suggested by @jhenderson we rename the original section type value to `SHT_LLVM_BB_ADDR_MAP_V0` and assign a new value to the `SHT_LLVM_BB_ADDR_MAP` section type. The new encoding adds a version byte to each function entry to specify the encoding version for that function. This patch also adds a feature byte to be used with more flexibility in the future. An use-case example for the feature field is encoding multi-section functions more concisely using a different format.
Conceptually, the new encoding emits basic block offsets and sizes as label differences between each two consecutive basic block begin and end label. When decoding, offsets must be aggregated along with basic block sizes to calculate the final offsets of basic blocks relative to the function address.
This encoding uses smaller values compared to the existing one (offsets relative to function symbol).
Smaller values tend to occupy fewer bytes in ULEB128 encoding. As a result, we get about 17% total reduction in the size of the bb-address-map section (from about 11MB to 9MB for the clang PGO binary).
The extra two bytes (version and feature fields) incur a small 3% size overhead to the `LLVM_BB_ADDR_MAP` section size.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D121346
Summary:
Address post-commit review comments in the https://reviews.llvm.org/D82549, including
changed file name from llvm/test/tools/llvm-readobj/XCOFF/xcoff-auxiliary-header.test --> llvm/test/tools/llvm-readobj/XCOFF/auxiliary-header.test
replaced macro define by using lambda function.
added a helper function to reduce the duplicated check and print error code.
Reviewer : James Henderson
Differential Revision: https://reviews.llvm.org/D116220
Instead of the GNU extension `SHF_GNU_RETAIN`, Solaris provides equivalent
functionality with `SHF_SUNW_NODISCARD`. This patch implements the necessary
support.
Tested on `sparcv9-sun-solaris2.11`, `amd64-pc-solaris2.11`, and
`x86_64-pc-linux-gnu`.
Differential Revision: https://reviews.llvm.org/D107955
Summary: Fix the build failure on MSVC by making the `T` and `U` of the function
'T llvm::Optional<T>::getValueOr<llvm::yaml::Hex32>(U &&) const &' the same.
Differential Revision: https://reviews.llvm.org/D111487
Summary:
This patch adds yaml2obj supporting for the auxiliary
file header of XCOFF.
Reviewed By: DiggerLin, jhenderson
Differential Revision: https://reviews.llvm.org/D111487
POSIX does not define the exact output from od tool.
While most implementations use lower case characters in hex output,
the z/OS USS implementation uses upper case characters.
To avoid LIT failures, the FileCheck option to ignore the case must
be used when checking hex bytes.
Reviewed By: abhina.sreeskantharajan
Differential Revision: https://reviews.llvm.org/D111427
Summary: Add the SectionIndex field for symbol.
1: a symbol can reference a section by SectionName or SectionIndex.
2: a symbol can reference a section by both SectionName and SectionIndex.
3: if both Section and SectionIndex are specified, but the two values refer
to different sections, an error will be reported.
4: an invalid SectionIndex is allowed.
5: if a symbol references a non-existent section by SectionName, an error will be reported.
Reviewed By: jhenderson, Higuoxing
Differential Revision: https://reviews.llvm.org/D109566
Summary: The patch adds support for yaml2obj customizing the string table.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D107421
Allow variable number of directories, as allowed by the
specification. NumberOfRvaAndSize will default to 16 if not specified,
as in the past.
Reviewed by: jhenderson
Differential Revision: https://reviews.llvm.org/D108825
item of StringTable.
Summary: For the string table in XCOFF, the first 4 bytes
contains the length of the string table, so we should
print the string entries from fifth bytes. This patch
also adds tests for llvm-readobj dumping the string
table.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D105522
Users should generally observe no difference as long as they don't use
unintended option forms. Behavior changes:
* `-t=d` is removed. Use `-t d` instead.
* `--demangle=false` and `--demangle=0` cannot be used. Omit the option or use `--no-demangle`. Other flag-style options don't have `--no-` forms.
* `--help-list` is removed. This is a `cl::` specific option.
* llvm-readobj now supports grouped short options as well.
* `--color` is removed. This is generally not useful (only apply to errors/warnings) but was inherited from Support.
Some adjustment to the canonical forms
(usually from GNU readelf; currently llvm-readobj has too many redundant aliases):
* --dyn-syms is canonical. --dyn-symbols is a hidden alias
* --file-header is canonical. --file-headers is a hidden alias
* --histogram is canonical. --elf-hash-histogram is a hidden alias
* --relocs is canonical. --relocations is a hidden alias
* --section-groups is canonical. --elf-section-groups is a hidden alias
OptTable avoids global option collision if we decide to support multiplexing for binary utilities.
* Most one-dash long options are still supported. `-dt, -sd, -st, -sr` are dropped due to their conflict with grouped short options.
* `--section-mapping=false` (D57365) is strange but is kept for now.
* Many `cl::opt` variables were unnecessarily external. I added `static` whenever appropriate.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D105532
Summary: The patch adds the StringTable dumping to
llvm-readobj. Currently only XCOFF is supported.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D104613
llvm-readobj is an internal testing tool for binary formats. Its output and
command line options do not need to be stable. It isn't supposed to be part of a
build process.
llvm-readelf was created as a user-facing utility and its interface intends to
be compatible with GNU readelf (unless there are good reasons not to).
The two tools have mostly compatible options. -s and -t are noticeable
exceptions due to history. I think the cost of keeping the inconsistency
overweighs the little history-compatible benefit and hinders transition from
cl::opt to OptTable, so let's change it.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D105055
Currently when .llvm.call-graph-profile is created by llvm it explicitly encodes the symbol indices. This section is basically a black box for post processing tools. For example, if we run strip -s on the object files the symbol table changes, but indices in that section do not. In non-visible behavior indices point to wrong symbols. The visible behavior indices point outside of Symbol table: "invalid symbol index".
This patch changes the format by using R_*_NONE relocations to indicate the from/to symbols. The Frequency (Weight) will still be in the .llvm.call-graph-profile, but symbol information will be in relocation section. In LLD information from both sections is used to reconstruct call graph profile. Relocations themselves will never be applied.
With this approach post processing tools that handle relocations correctly work for this section also. Tools can add/remove symbols and as long as they handle relocation sections with this approach information stays correct.
Doing a quick experiment with clang-13.
The size went up from 107KB to 322KB, aggregate of all the input sections. Size of clang-13 binary is ~118MB. For users of -fprofile-use/-fprofile-sample-use the size of object files will go up slightly, it will not impact final binary size.
Reviewed By: jhenderson, MaskRay
Differential Revision: https://reviews.llvm.org/D104080
Summary: This patch, as a follow-up of D95505, adds
support for writing the long symbol name by implementing
the StringTable. Only XCOFF32 is suppoted now.
Reviewed By: jhenderson, shchenz
Differential Revision: https://reviews.llvm.org/D103455
This patch adds support for a new field in the FileHeader, which states
the name to use for the section header string table. This also allows
combining the string table with another string table in the object, e.g.
the symbol name string table. The field is optional. By default,
.shstrtab will continue to be used.
This partially fixes https://bugs.llvm.org/show_bug.cgi?id=50506.
Reviewed by: Higuoxing
Differential Revision: https://reviews.llvm.org/D104035
There was an off-by-one error caused by an index (which included an
index for the null section header) being used to check against the size
of a list of sections (which didn't include the null section header).
This is a partial fix for https://bugs.llvm.org/show_bug.cgi?id=50506.
Reviewed by: MaskRay
Differential Revision: https://reviews.llvm.org/D104098
Summary: The patch implements the mapping of the Yaml
information to XCOFF object file to enable the yaml2obj
tool for XCOFF. Currently only 32-bit is supported.
Reviewed By: jhenderson, shchenz
Differential Revision: https://reviews.llvm.org/D95505
This patch uses the errno python library to print out the correct error messages instead of hardcoding the error message per platform.
Reviewed By: jhenderson, ASDenysPetrov
Differential Revision: https://reviews.llvm.org/D97472
As discussed in D95511, this allows us to encode invalid BBAddrMap
sections to be used in more rigorous testing.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D96831
On z/OS, other error messages are not matched correctly in lit tests.
```
EDC5121I Invalid argument.
EDC5111I Permission denied.
```
This patch adds a lit substitution to fix it.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D95808
On z/OS, the following error message is not matched correctly in lit tests.
```
EDC5129I No such file or directory.
```
This patch uses a lit config substitution to check for platform specific error messages.
Reviewed By: muiez, jhenderson
Differential Revision: https://reviews.llvm.org/D95246
Currently we don't allow the following definition:
```
Sections:
- Type: SectionHeaderTable
- Name: .foo
Type: SHT_PROGBITS
```
We report an error: "SectionHeaderTable can't be empty. Use 'NoHeaders' key to drop the section header table".
It was implemented in this way earlier, when `SectionHeaderTable`
was a dedicated key outside of the `Sections` list. And we did not
allow to select where the table is written.
Currently it makes sense to allow it, because a user might
want to place the default section header table at an arbitrary position,
e.g. before other sections. In this case it is not convenient and error prone
to require specifying all sections:
```
Sections:
- Type: SectionHeaderTable
Sections:
- Name: .foo
- Name: .strtab
- Name: .shstrtab
- Name: .foo
Type: SHT_PROGBITS
```
This patch allows empty SectionHeaderTable definitions.
Differential revision: https://reviews.llvm.org/D95341
This was discussed in D93678 thread.
Currently we have one special chunk - Fill.
This patch re implements the "SectionHeaderTable" key to become a special chunk too.
With that we are able to place the section header table at any location,
just like we place sections.
Differential revision: https://reviews.llvm.org/D95140
This makes the following improvements.
For `SHT_GNU_versym`:
* yaml2obj: set `sh_link` to index of `.dynsym` section automatically.
For `SHT_GNU_verdef`:
* yaml2obj: set `sh_link` to index of `.dynstr` section automatically.
* yaml2obj: set `sh_info` field automatically.
* obj2yaml: don't dump the `Info` field when its value matches the number of version definitions.
For `SHT_GNU_verneed`:
* yaml2obj: set `sh_link` to index of `.dynstr` section automatically.
* yaml2obj: set `sh_info` field automatically.
* obj2yaml: don't dump the `Info` field when its value matches the number of version dependencies.
Also, simplifies few test cases.
Differential revision: https://reviews.llvm.org/D94956
On z/OS, the following error message is not matched correctly in lit tests. This patch updates the CHECK expression to match successfully.
```
EDC5129I No such file or directory.
```
Reviewed By: muiez
Differential Revision: https://reviews.llvm.org/D94239
