Some unit tests has incorrect DW_AT_type offset since they are manual crafted, fix them to the correct offset.
Reviewed By: Amir, ayermolo
Differential Revision: https://reviews.llvm.org/D129828
Add a test for `-icp-inline` knob, which ensures that ICP is only performed for
functions that can be subsequently inlined.
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D129803
We previously support split jump table, where some jump table entries
target different fragments of same function. In this fix, we provide
support for another type of intra-indirect transfer: landing pad.
When C++ exception handling is used, compiler emits .gcc_except_table
that describes the location of catch block (landing pad) for specific
range that potentially invokes a throw(). Normally landing pads reside
in the function, but with -fsplit-machine-functions, landing pads can
be moved to another fragment. The intuition is, landing pads are rarely
executed, so compiler can move them to .cold section.
This update will mark all fragments that have landing pad to another
fragment as non-simple, and later propagate non-simple to all related
fragments.
This update also includes one manual test case: split-landing-pad.s
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D128561
There are two assumptions regarding jump table:
(a) It is accessed by only one fragment, say, Parent
(b) All entries target instructions in Parent
For (a), BOLT stores jump table entries as relative offset to Parent.
For (b), BOLT treats jump table entries target somewhere out of Parent
as INVALID_OFFSET, including fragment of same split function.
In this update, we extend (a) and (b) to include fragment of same split
functinon. For (a), we store jump table entries in absolute offset
instead. In addition, jump table will store all fragments that access
it. A fragment uses this information to only create label for jump table
entries that target to that fragment.
For (b), using absolute offset allows jump table entries to target
fragments of same split function, i.e., extend support for split jump
table. This can be done using relocation (fragment start/size) and
fragment detection heuristics (e.g., using symbol name pattern for
non-stripped binaries).
For jump table targets that can only be reached by one fragment, we
mark them as local label; otherwise, they would be the secondary
function entry to the target fragment.
Test Plan
```
ninja check-bolt
```
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D128474
Add -experimental-shrink-wrapping flag to control when we
want to move callee-saved registers even when addresses of the stack
frame are captured and used in pointer arithmetic, making it more
challenging to do alias analysis to prove that we do not access
optimized stack positions. This alias analysis is not yet implemented,
hence, it is experimental. In practice, though, no compiler would emit
code to do pointer arithmetic to access a saved callee-saved register
unless there is a memory bug or we are failing to identify a
callee-saved reg, so I'm not sure how useful it would be to formally
prove that.
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D126115
Add the option to run -equalize-bb-counts before shrink
wrapping to avoid unnecessarily optimizing some CFGs where profile is
inaccurate but we can prove two blocks have the same frequency.
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D126113
Change how function score is calculated and provide more
detailed statistics when reporting back frame optimizer and shrink
wrapping results. In this new statistics, we provide dynamic coverage
numbers. The main metric for shrink wrapping is the number of executed
stores that were saved because of shrink wrapping (push instructions
that were either entirely moved away from the hot block or converted
to a stack adjustment instruction). There is still a number of reduced
load instructions (pop) that we are not counting at the moment. Also
update alloc combiner to report dynamic numbers, as well as frame
optimizer.
For debugging purposes, we also include a list of top 10 functions
optimized by shrink wrapping. These changes are aimed at better
understanding the impact of shrink wrapping in a given binary.
We also remove an assertion in dataflow analysis to do not choke on
empty functions (which makes no sense).
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D126111
Added support for mixing monolithic DWARF5 with legacy DWARF, and monolithic legacy and DWARF5 split dwarf.
Reviewed By: maksfb
Differential Revision: https://reviews.llvm.org/D128232
For test purposes, we want to split functions at a random split point
to be able to test different layouts without relying on the profile.
This patch introduces an option, that randomly chooses a split point
to partition blocks of a function into hot and cold regions.
Reviewed By: Amir, yota9
Differential Revision: https://reviews.llvm.org/D128773
The SplitFunctions pass does not distinguish between various splitting
modes anymore. This change updates the command line interface to
reflect this behavior by deprecating values passed to the
--split-function option.
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D128558
DWARF 5 added two new attributes DW_AT_call_pc and DW_AT_call_return_pc.
Adding support for them.
Reviewed By: maksfb
Differential Revision: https://reviews.llvm.org/D128526
Allow cold fragment to get new address.
Our previous assumption is that a fragment (.cold) is only reached
through the main fragment of same function. In addition, .cold fragment
must be reached through either (a) direct transfer, or (b) split jump
table. For (a), we perform a simple fix-up. For (b), we currently mark
all relevant fragments as non-simple. Therefore, there is no need to
get new address for .cold fragment.
This is not always the case, as function entry can be rarely executed,
and is placed in .text.cold segment. Essentially we cannot tell which
the source-level function entry is based on hot and cold segments,
so we must treat each fragment a function on its own. Therfore, we
remove the assertion that a function entry cannot be cold fragment.
Test Plan:
```
ninja check-bolt
```
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D128111
Resolve a crash related to split functions
Due to split function optimization, a function can be divided to two
fragments, and both fragments can access same jump table. This
violates the assumption that a jump table can only have one parent
function, which causes a crash during instrumentation.
We want to support the case: different functions cannot access same
jump tables, but different fragments of same function can!
As all fragments are from same function, we point JT::Parent to one
specific fragment. Right now it is the first disassembled fragment, but
we can point it to the function's main fragment later.
Functions are disassembled sequentially. Previously, at the end of
processing a function, JT::OffsetEntries is cleared, so other fragment
can no longer reuse JT::OffsetEntries. To extend the support for split
function, we only clear JT::OffsetEntries after all functions are
disassembled.
Let say A.hot and A.cold access JT of three targets {X, Y, Z}, where
X and Y are in A.hot, and Z is in A.cold. Suppose that A.hot is
disassembled first, JT::OffsetEntries = {X',Y',INVALID_OFFSET}. When
A.cold is disassembled, it cannot reuse JT::OffsetEntries above due to
different fragment start. A simple solution:
A.hot = {X',Y',INVALID_OFFSET}
A.cold = {INVALID_OFFSET, INVALID_OFFSET, INVALID_OFFSET}
We update the assertion to allow different fragments of same function
to get the same JumpTable object.
Potential improvements:
A.hot = {X',Y',INVALID_OFFSET}
A.cold = {INVALID_OFFSET, INVALID_OFFSET, Z'}
The main issue is A.hot and A.cold have separate CFGs, thus jump table
targets are still constrained within fragment bounds.
Future improvements:
A.hot = {X, Y, Z}
A.cold = {X, Y, Z}
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D127924
Replace a single dash with a double dash for options that have more
than a single letter.
llvm-bolt-wrapper.py has special treatment for output options such as
"-o" and "-w" causing issues when a single dash is used, e.g. for
"-write-dwp". The wrapper can be fixed as well, but using a double dash
has other advantages as well.
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D127538
Mark fragments related to split jump table as non-simple.
A function could be splitted into hot and cold fragments. A split jump table is
challenging for correctly reconstructing control flow graphs, so it was marked
as ignored. This update marks those fragments as non-simple, allowing them
to be printed and partial control flow graph construction.
Test Plan:
```
llvm-lit -a tools/bolt/test/X86/split-func-icf.s
```
This test has two functions (main, main2), each has a jump table target to the
same cold portion main2.cold.1(*2). We try to print out only this cold portion.
If it is ignored, it cannot be printed. If it is non-simple, it can be printed. We
verify that it can be printed.
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D127464
The linker can convert instructions with GOTPCRELX relocations into a
form that uses an absolute addressing with an immediate. BOLT needs to
recognize such conversions and symbolize the immediates.
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D126747
I put it into wrong directory. As the result it is failing for aarch64. Moving
the test under X86. Orignial diff D126999.
Differential Revision: https://reviews.llvm.org/D127417
When we generate split dwarf with -fdebug-types-section we will have
.debug_types.dwo sections. These go into TU Index when we run llvm-dwp. BOLT was
not handling DWP input correctly with this section.
Added support for handling DWP with TU Index as an input and output for DWARF4.
Added support for handling DWP with TU Index as an input for DWARF5
Reviewed By: maksfb
Differential Revision: https://reviews.llvm.org/D126087
Replace "cache+" with "ext-tsp" in all BOLT tests
Test Plan:
```
ninja check-bolt
grep -rnw . -e "cache+"
```
no more tests containing "cache+"
"cache+" and "ext-tsp" are aliases
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D126714
After D126484, order in .debug-line-str and .debug-line is different. Changed
test accordingly.
Differential Revision: https://reviews.llvm.org/D126733
Summary:
While disassembling instructions, we need to replace certain immediate
operands with symbols. This symbolizing process relies on reading
relocations against instructions. However, some X86 instructions can
have multiple immediate operands and up to two relocations against
them. Thus, correctly matching a relocation to an operand is not
always possible without knowing the operand offset within the
instruction.
Luckily, LLVM provides an interface for passing the required info from
the disassembler via a virtual MCSymbolizer class. Creating a
target-specific version allows a precise matching of relocations to
operands.
This diff adds X86MCSymbolizer class that performs X86-specific
symbolizing (currently limited to non-branch instructions).
Reviewers: yota9, Amir, ayermolo, rafauler, zr33
Differential Revision: https://reviews.llvm.org/D120928
Fix a bug where shrink-wrapping would use wrong stack offsets
because the stack was being aligned with an AND instruction, hence,
making its true offsets only available during runtime (we can't
statically determine where are the stack elements and we must give up
on this case).
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D126110
Add a new testcase that reproduces a bug when BOLTing current
trunk LLD bootstrapped with trunk clang. This makes it official
that we do not support this transformation but are working on
it. When the support is ready, XFAIL should be removed.
Reviewed By: maksfb, Amir, yota9
Differential Revision: https://reviews.llvm.org/D125843
When a profile is collected in a BOLTed binary, the generated
profile is tagged with a header string "boltedcollection" in the first
line of the fdata file. Fix merge-fdata to recognize this header
string and preserve it into the output.
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D125591
As BOLT support for monolithic and split DWARF5 is added, remove DWARF version
override for BOLT tests.
Reviewed By: ayermolo
Differential Revision: https://reviews.llvm.org/D125366
Address X86 tests failures on AArch64 builder:
https://lab.llvm.org/staging/#/builders/211/builds/82
Inputs fail to cross-compile due to a missing header:
```
/usr/include/stdio.h:27:10: fatal error: 'bits/libc-header-start.h' file not found
#include <bits/libc-header-start.h>
```
As inputs are linked with `-nostdlib` anyway, don't include stdio.h.
Reviewed By: yota9
Differential Revision: https://reviews.llvm.org/D124863
Address X86 tests failures on AArch64 builder:
https://lab.llvm.org/staging/#/builders/211/builds/82
Inputs fail to cross-compile due to a missing header:
```
/usr/include/stdio.h:27:10: fatal error: 'bits/libc-header-start.h' file not found
#include <bits/libc-header-start.h>
```
As inputs are linked with `-nostdlib` anyway, don't include stdio.h.
Reviewed By: yota9
Differential Revision: https://reviews.llvm.org/D124863
Added implementation to support DWARF5 in monolithic mode.
Next step DWARF5 split dwarf support.
Reviewed By: maksfb
Differential Revision: https://reviews.llvm.org/D121876
LLVM with LTO can generate function names in the form
func.llvm.<number>, where <number> could vary based on the compilation
environment. As a result, if a profiled binary originated from a
different build than a corresponding binary used for BOLT optimization,
then profiles for such LTO functions will be ignored.
To fix the problem, use "fuzzy" matching with "func.llvm.*" form.
Reviewed By: yota9, Amir
Differential Revision: https://reviews.llvm.org/D124117
Looks like implementation in llvm changed, and now we need to process error
being returned.
Reviewed By: maksfb
Differential Revision: https://reviews.llvm.org/D124133
Align with an upstream change D120305 to make PIE the default on linux-gnu.
Add `-no-pie` to tests that require it.
Reviewed By: maksfb, yota9
Differential Revision: https://reviews.llvm.org/D123329
BOLT expects PC-relative relocations in data sections to reference code
and the relocated data to form a jump table. However, there are cases
where PC-relative addressing is used for data-to-data references
(e.g. clang-15 can generate such code). BOLT should recognize and ignore
such relocations. Otherwise, they will be considered relocations not
claimed by any jump table and cause a failure in the strict mode.
Reviewed By: yota9, Amir
Differential Revision: https://reviews.llvm.org/D123650
Check for supported target architecture instead of the host arch when
deciding to execute non-runtime tests.
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D122498
It seems the earlier implementation does not follow the description
in LoopRotationPass.h: It rotates loops even if they are already laid out
correctly. The diff adjusts the behaviour.
Given that the impact of LoopInversionPass is minor, this change won't
yield significant perf differences. Tested on clang-10: there seems to be a
0.1%-0.3% cpu win and a small reduction of branch misses.
**Before:**
BOLT-INFO: 120 Functions were reordered by LoopInversionPass
**After:**
BOLT-INFO: 79 Functions were reordered by LoopInversionPass
Reviewed By: yota9
Differential Revision: https://reviews.llvm.org/D121921
Since LLVM MC now preserves redundant AdSize override prefix (0x67), remove it
in BOLT explicitly (-x86-strip-redundant-adsize, on by default).
Test Plan:
`bin/llvm-lit -a bolt/test/X86/addr32.s`
Reviewed By: rafauler
Differential Revision: https://reviews.llvm.org/D120975
