Currently, many unnecessary samples are populated into MemSamples,
including zero-initialized samples and samples in which the PC address
is not contained in any BinaryFunction. But these samples are totally
skipped during processing and the whole MemSamples vector is cleared
immediately after processing. So, we could just stop populating these
samples into MemSamples, which would reduce maximum resident set size
when processing a large perf.data.
Hot text mover functions are placed in special sections (e.g.,
.never_hugify) to avoid being placed on hot/huge pages. Folding them
with functions from other sections could defeat this purpose.
Add a check in ICF's isIdenticalWith() to prevent folding when either
function is a hot text mover.
When handling relocation in one function referencing code or
data defined in another function, we should check if relocation
target is constant island or not, and get the referenced symbol
accordingly for both cases.
On x86-64, PLT optimization does not require the binary to be linked
with -znow because indirect calls through GOT work correctly with lazy
binding. At runtime, the dynamic linker's resolver will populate the GOT
entry on the first call, just like with a regular PLT call.
This change removes the -znow requirement specifically for x86-64 while
keeping it for other architectures. I haven't checked RISV-V, but it's
still necessary on AArch64.
This patch adds comprehensive assembler (MC layer) support for the
Mach-O object file format on RISC-V targets, enabling assembly and
disassembly of RISC-V code targeting Apple platforms.
Key changes:
- Define RISC-V-specific Mach-O relocation types in BinaryFormat/MachO.h
- Implement RISCVMachObjectWriter with full relocation handling for:
- PCREL_HI/LO pairs for PC-relative addressing
- GOT relocations for external symbols
- Branch relocations (CALL, unconditional/conditional branches)
- Data section relocations
Test files include llvm-otool dumps to verify the generated relocations.
This code is based on code originally written by Tim Northover.
Having it in the X86 subdirectory only affects tests in that directory.
That's however not sufficient as for example runtime/X86/pie-exceptions-split.test is affected but
isn't located in the X86 directory.
This essentially fixes the fix for the original commit by guarding it properly for when the X86
target has been built and the flag is recognized.
Fixes: 6c48fbc1dcfbd44a47f126f21e575340b67aac06
This patch adds the patchPLTEntryForBTI to enable patching PLT entries
generated by LLD.
## Context:
To keep BTI consistent, targets of stubs inserted in LongJmp need to be
patched. As PLTs are not optimized and emitted by BOLT, this patch adds
a helper for patching them in the original .plt section.
For PLTs generated by LLD, this is safe as LLD inserts extra nops to
PLTs which don't already contain a BTI.
PLT entry before patching:
```
adrp x16, Page(&(.got.plt[n]))
ldr x17, [x16, Offset(&(.got.plt[n]))]
add x16, x16, Offset(&(.got.plt[n]))
br x17
nop
nop
```
PLT entry after patching:
```
bti c
adrp x16, Page(&(.got.plt[n]))
ldr x17, [x16, Offset(&(.got.plt[n]))]
add x16, x16, Offset(&(.got.plt[n]))
br x17
nop
```
## Safety considerations:
The PLT entry can become incorrect if shifting the ADRP moves it
across a page boundary.
The PLT entry is 24 bytes, and page size is 4096 (or 16384) bytes.
Their GCD is 8 bytes, meaning that shifting the ADRP is safe, as long as
it's shifted by less than 8 bytes.
The introduced function only shifts the ADRP by one instruction (4
bytes),
meaning there is no need to recompute the ADRP offset.
PLT entries are PseudoFunctions, and are not disassembled or emitted.
For BTI, we need to check the first MCInst of PLT entries, to see
if indirectly calling them is safe or not.
This patch disassembles PLTs for binaries using BTI, while not changing
the behaviour for binaries without BTI.
The PLTs are only disassembled, not emitted.
---------
Co-authored-by: Paschalis Mpeis <paschalis.mpeis@arm.com>
Corrected various spelling mistakes such as 'occurred', 'receiver',
'initialized', 'length', and others in comments, variable names,
function names, and documentation throughout the project. These
changes improve code readability and maintain consistency in naming
and documentation.
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
There is no guarantee that PatchOffset is suitably aligned for uint32_t,
and in BOLT's own tests, it is not aligned for uint32_t.
Fixes test failures seen with LLVM_USE_SANITIZER=Undefined.
On RHEL8 we get the following error that may originate from the changed
typo:
```
clang: warning: argument unused during compilation: '-ffreestanding' [-Wunused-command-line-argument]
ld.lld: error: relocation R_AARCH64_ADR_PREL_LO21 cannot be used against symbol '_jmptbl2'; recompile with -fPIC
>>> defined in /tmp/epilogue-determination-7bd9d4.o
>>> referenced by /tmp/epilogue-determination-7bd9d4.o:(.text+0x54)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
Since this code was last reviewed, more relaxations have been added to
the psABI for existing standard relocations that LLVM didn't have marked
as relaxable.
This change ensures that LLVM marks the following relocations (and their
respective fixups) as relaxable:
- `R_RISCV_JAL`
- `R_RISCV_GOT_HI20`
- `R_RISCV_TPREL_HI20`
- `R_RISCV_TLSDESC_HI20`
This also updates the linker relaxation test to use `-NEXT` to check all
the output lines.
There are lots more emitted relocations, not only due to JAL being
relaxable, but branches are now also marked linker relaxable because
they can be turned into `b<cc>; jal` during assembly relaxation, which
may also be marked relaxable.
A build with LLVM_USE_SANITIZER=Undefined showed:
bolt/lib/Target/AArch64/AArch64MCPlusBuilder.cpp:2277:60:
runtime error: left shift of negative value -32768
This showed up in bolt/test/AArch64/veneer-lite-mode.s.
It is valid for ADRP's operand to be negative, and not valid to shift it
like that. To perform this shift reliably, cast the value to unsigned.
Until instrumentation support is added, the feature should be
disabled for BTI binaries. An error message is added to explain
the situation.
Meanwhile, users can choose sampling-based profiling methods.
Added a TODO comment explaining missing steps.
While the current max memory size is sufficient for most binaries, a few
binaries may encouter insufficient allocated memory space.
Allow specify the max memory size of the instrumentation bump allocator.
Several BTI-related functions are checking that a call MCInst has one
non-annotation operand.
This patch changes these checks to use MCPlus::getNumPrimeOperands,
instead of getNumOperands.
Testing:
added annotations to existing gtests to serve as regression
tests. These now also explicitly check getNumOperands and getNumPrimeOperands
usage on the annotated MCInsts.
There's no need for a full definition of `BinaryBasicBlock` in
`MCPlusBuilder.h`. Use `InstructionListType::iterator` instead of
`BinaryBasicBlock::iterator` in `findMemcpySizeInBytes()`.
Update the comments in PAuthGadgetScanner.cpp to better describe the
current version of the code. Along the way, shorten identifier names
that are redundant taking their context into account:
`RegsToTrackInstsFor` (made `RegsToTrack`) and `getNumTrackedRegisters`
(made `getNumRegisters`).
Co-authored-by: Kristof Beyls <kristof.beyls@arm.com>
Add utils/docker-tests/Dockerfile to facilitate in-tre and out-of-tree
testing.
Builds perf from source to work around an Ubuntu 24.04 issue.
To reproduce a specific issue adjust the Dockerfile like:
```
RUN git clone https://github.com/llvm/llvm-project
RUN cd llvm-project && git checkout <SHA>
```
Replace the code along these lines
BitVector Tmp = LHS;
Tmp &= RHS;
return Tmp.any();
and
BitVector Tmp = LHS;
Tmp.reset(RHS);
return Tmp.none();
with `LHS.anyCommon(RHS)` and `LHS.subsetOf(RHS)`, correspondingly,
which do not require creating temporary BitVector and can return early.
Top-level (binary) functions don't have a unique GUID mapping, with
different
causes namely coroutine fragments sharing the same parent source
function GUID.
Replace the top-level inline tree node GUID lookup with probe lookup
coupled
with walk up the inline tree.
Test Plan: added test-coro-probes.yaml
This patch adds BTI landing pads to ShortJmp/LongJmp targets in the
LongJmp pass when optimizing BTI binaries.
BOLT does not have the ability to add BTI to all types of functions.
This patch aims to insert the landing pad where possible, and emit an
error and exit where it currently is not.
BOLT cannot insert BTIs into several function "types", including:
- ignored functions,
- PLT functions,
- other functions without a CFG.
Additional context:
In #161206, BOLT gained the ability to decode the .note.gnu.property
section, and warn about lack of BTI support for BOLT. However, this
warning is misleading: the emitted binary may not need extra BTI landing
pads.
With this patch, the emitted binary will be "BTI-safe".
- Add an enum to encode BTI variants in function arguments.
- Remove updateBTIVariant as createBTI can be used for the same
purpose.
- Remove a test case that checked against invalid BTI variants, as
those are now unrepresentable.
Add `-Wl,-z,defs` in `%cflags` to make linking more consistent across
distros. Aims to catch typos in handwritten assembly tests. If
intentional, individual tests can override this with: `-Wl,-z,undefs`
Tests changed:
- bolt/test/AArch64/bf_min_alignment.s: fixed a typo
- bolt/test/AArch64/tls.c: overriding behaviour in all clang
invocations, making the test more consistent.
- Overrides to allow undefined symbols in:
- bolt/test/AArch64/epilogue-determination.s
- bolt/test/AArch64/exceptions-plt.cpp
- bolt/test/AArch64/ifunc.test
- bolt/test/AArch64/plt-call.test
- bolt/test/AArch64/text-data.c
- bolt/test/AArch64/unsupported-passes.test
- bolt/test/AArch64/unsupported-passes.test
Several lit tests are failing on macOS, as they are linux-specific,
but not using the "REQUIRES: system-linux" setting.
This patch adds the missing requirements to tests failing on macOS.
few tests are broken on ubuntu, need find out the cause
This reverts commit 999c9382571d6aadf9b786263862bf4085dd2dba.
Co-authored-by: yavtuk <yavtuk@ya.ru>
This patch fixed the issue related to load literal
for AArch64 (bolt/test/AArch64/materialize-constant.s),
address range for literal is limited +/- 1MB,
emitCI puts the constants by the end of function and
the one is out of available range.
SimplifyRODataLoads is enabled by default for X86 & AArch64
Add hashing support for MCSpecifierExpr, which is frequently used as an
operand in AArch64 instructions. Proper hashing reduces collisions when
placing functions into buckets, resulting in shorter Identical Code
Folding (ICF) pass runtime.
In one benchmark, the ICF wall time decreased from 272s to 124s.
On AArch64, a larger cold code size can result in more veneers,
increasing potential overhead for hot code. This change minimizes cold
code size when the `--use-compact-aligner` option (default) is enabled.
The order in which functions are emitted into the output file does not
always reflect the order in which they will appear in the final file.
This discrepancy occurs because code is emitted to different sections,
such as `.text`, `.text.cold`, `.text.mover`, etc.
To make passes that rely on the relative function order - such as
`LongJmpPass` - more precise and functional, sort the output functions
to reflect their final layout, and validate the layout after code
sections are sorted.
Note that, at this time, we only directly change the order of the main
fragments of functions. The order of other fragments, such as cold and
warm sections relative to other fragment types, is determined by the
section order.
To gain better control over the functions that go into the output file
and their order, introduce `BinaryContext::getOutputBinaryFunctions()`.
The new API returns a modifiable list of functions in output order.
This list is filled by a new `PopulateOutputFunctions` pass and includes
emittable functions from the input file, plus functions added by BOLT
(injected functions).
The new functionality allows to freely intermix input functions with
injected ones in the output, which will be used in new PRs.
The new function replaces `BinaryContext::getSortedFunctions()`, but
unlike its predecessor, it includes injected functions in the returned
list.
If a basic block contains a load into LR from stack and has no
instruction saving LR onto stack, we just assume the basic block
is an epilogue.
This is not meant to accurately recognize epilogue in all possible
cases, but to have BOLT be conservative on treating basic block as
epilogue and then turning indirect branch with unknown control flow
to tail call.
This function contains most of the logic for BTI:
- it takes the BasicBlock and the instruction used to jump to it.
- Then it checks if the first non-pseudo instruction is a sufficient
landing pad for the used call.
- if not, it generates the correct BTI instruction.
Also introduce the isCallCoveredByBTI helper to simplify the logic.
Assign output sections for injected functions explicitly, and don't
reassign in AssignSections pass.
This change is a prerequisite for further PRs where veneer functions are
created as injected functions and their code section depends on their
placement.
If a veneer is not disassembled in lite mode, the veneer elimination
pass will not recognize it as such and the call to such veneer will
remain unchanged.
Later, we may need to insert a new veneer for such code ending up with a
double veneer.
To avoid such suboptimal code generation, always disassemble veneers and
guarantee that they are converted to direct calls in BOLT.
