Introduce `__builtin_allow_sanitize_check("name")` which returns true if
the specified sanitizer is enabled for the function (after inlining).
Supported sanitizers are "address", "thread", "memory", "hwaddress", and
their "kernel-" variants, matching the names of the
`no_sanitize("name")`
usage.
This builtin enables conditional execution of explicit checks only when
the sanitizer is enabled, respecting `no_sanitize` attributes, even when
used from `always_inline` functions that may be used in sanitized or
no_sanitize functions.
Since we must defer until after inlining and cannot determine the result
statically, Clang must lower to the `llvm.allow.sanitize.*` intrinsics,
which are then resolved by the `LowerAllowCheckPass`.
*Original Motivation:* The Linux kernel has a number of low-level
primitives that use inline assembly not visible to the sanitizers, but
use explicitly inserted checks to avoid coverage loss. Many of those
low-level helpers, however, are also used from so-called `noinstr`
functions, which use `no_sanitize(..)` to prohibit instrumentation;
these are used for very brittle code (such as when the kernel sets up a
task context *before* normal memory is accessible), and any
instrumentation, incl. from explicit instrumentation, is prohibited.
Many such helpers themselves are macros or `always_inline`, however, are
unable to be used from such brittle contexts because they contain
explicit instrumentation. This requires awkward workarounds to avoid the
instrumentation.
The ideal solution is this new builtin, that can be used to determine if
instrumentation is enabled in a given function or not, which the helper
can then use to insert instrumentation only where instrumentation is
allowed.
A recent such case came up in [1], where file-level instrumentation had
already been disabled for KASAN and KCSAN, which had not been necessary
if the new builtin were available.
[1] https://lore.kernel.org/all/20251208-gcov-inline-noinstr-v1-0-623c48ca5714@google.com/
Single-use AddEmitPasses is inlined into RunCodegenPipeline for clarity
in comparing the parameters to the plugin and the parameters passed to
addPassesToEmitFile.
Pull Request: https://github.com/llvm/llvm-project/pull/171872
This permits pass plugins to use llvm:🆑:opt. Additionally, add a test
of -fpass-plugin, this was previously not tested at all.
I'm not sure whether using the LLVM Bye.so in the tests is possible this
way (e.g., if Clang is built standalone).
Reland after #173279.
Pull Request: https://github.com/llvm/llvm-project/pull/173287
This avoid pulling in the entire Passes library with all passes as
dependencies when just referring to PassPlugin, which is in fact
independent of the Passes themselves.
Pull Request: https://github.com/llvm/llvm-project/pull/173279
This permits pass plugins to use llvm:🆑:opt. Additionally, add a test
of -fpass-plugin, this was previously not tested at all.
I'm not sure whether using the LLVM Bye.so in the tests is possible this
way (e.g., if Clang is built standalone).
Pull Request: https://github.com/llvm/llvm-project/pull/171868
This avoid pulling in the entire Passes library with all passes as
dependencies when just referring to PassPlugin, which is in fact
independent of the Passes themselves.
Pull Request: https://github.com/llvm/llvm-project/pull/172478
This PR introduces a new mechanism for enforcing a sandbox around
filesystem reads coming from the compiler. A fatal error is raised
whenever the `llvm::sys::fs`, `llvm::MemoryBuffer::getFile*()` APIs get
used directly instead of going through the "blessed" virtual interface
of `llvm::vfs::FileSystem`.
This patch adds Clang support for speculative devirtualization and
integrates the related pass into the pass pipeline.
It's building on the LLVM backend implementation from PR #159048.
Speculative devirtualization transforms an indirect call (the virtual
function) to a guarded direct call.
It is guarded by a comparison of the virtual function pointer to the
expected target.
This optimization is still safe without LTO because it doesn't do direct
calls, it's conditional according to the function ptr.
This optimization:
- Opt-in: Disabled by default, enabled via `-fdevirtualize-speculatively`
- Works in non-LTO mode
- Handles publicly-visible objects.
- Uses guarded devirtualization with fallback to indirect calls when the
speculation is incorrect.
For this C++ example:
```
class Base {
public:
__attribute__((noinline))
virtual void virtual_function1() { asm volatile("NOP"); }
virtual void virtual_function2() { asm volatile("NOP"); }
};
class Derived : public Base {
public:
void virtual_function2() override { asm volatile("NOP"); }
};
__attribute__((noinline))
void foo(Base *BV) {
BV->virtual_function1();
}
void bar() {
Base *b = new Derived();
foo(b);
}
```
Here is the IR without enabling speculative devirtualization:
```
define dso_local void @_Z3fooP4Base(ptr noundef %BV) local_unnamed_addr #0 {
entry:
%vtable = load ptr, ptr %BV, align 8, !tbaa !6
%0 = load ptr, ptr %vtable, align 8
tail call void %0(ptr noundef nonnull align 8 dereferenceable(8) %BV)
ret void
}
```
IR after enabling speculative devirtualization:
```
define dso_local void @_Z3fooP4Base(ptr noundef %BV) local_unnamed_addr #0 {
entry:
%vtable = load ptr, ptr %BV, align 8, !tbaa !12
%0 = load ptr, ptr %vtable, align 8
%1 = icmp eq ptr %0, @_ZN4Base17virtual_function1Ev
br i1 %1, label %if.true.direct_targ, label %if.false.orig_indirect, !prof !15
if.true.direct_targ: ; preds = %entry
tail call void @_ZN4Base17virtual_function1Ev(ptr noundef nonnull align 8 dereferenceable(8) %BV)
br label %if.end.icp
if.false.orig_indirect: ; preds = %entry
tail call void %0(ptr noundef nonnull align 8 dereferenceable(8) %BV)
br label %if.end.icp
if.end.icp: ; preds = %if.false.orig_indirect, %if.true.direct_targ
ret void
}
```
The libcall lowering decisions should be program dependent,
depending on the current module's RuntimeLibcallInfo. We need
another related analysis derived from that plus the current
function's subtarget to provide concrete lowering decisions.
This takes on a somewhat unusual form. It's a Module analysis,
with a lookup keyed on the subtarget. This is a separate module
analysis from RuntimeLibraryAnalysis to avoid that depending on
codegen. It's not a function pass to avoid depending on any
particular function, to avoid repeated subtarget map lookups in
most of the use passes, and to avoid any recomputation in the
common case of one subtarget (and keeps it reusable across
repeated compilations).
This also switches ExpandFp and PreISelIntrinsicLowering as
a sample function and module pass. Note this is not yet wired
up to SelectionDAG, which is still using the LibcallLoweringInfo
constructed inside of TargetLowering.
Remove explicit insertion of the AllocTokenPass, which is now handled by
the PassBuilder. Emit AllocToken configuration as LLVM module flags to
persist into the backend.
Specifically, this also means it will now be handled by LTO backend
phases; this avoids interference with other optimizations (e.g. PGHO)
and enable late heap-allocation optimizations with LTO enabled.
Unconditionally add AllocTokenPass to the optimization pipelines, and
ensure that it runs last in LTO backend pipelines. The latter ensures
that AllocToken instrumentation can be moved later in the LTO pipeline
to avoid interference with other optimizations (e.g. PGHO) and enable
late heap-allocation optimizations.
In preparation of removing AllocTokenPass being added by Clang, add
support for AllocTokenPass to read configuration options from LLVM
module flags.
To optimize given the pass is now runs unconditionally, only retrieve
TargetLibraryInfo and OptimizationRemarkEmitter when necessary.
d076608d58d1ec55016eb747a995511e3a3f72aa moved some deps around to avoid
cycles and left clang/Frontend/FrontendDiagnostic.h as a shim that
simply includes clang/Basic/DiagnosticFrontend.h. This PR inlines it so
that nothing in tree still includes clang/Frontend/FrontendDiagnostic.h.
Doing this will help prevent future layering issues. See #162865.
Frontend already depends on Basic, so no new deps need to be added
anywhere except for places that do strict dep checking.
After the base branch was moved to main, this somehow ended up
adding a second definition of RTLCI, instead of modifying the
existing one.
Also fix other build error with gcc bots.
This fixes the -fveclib flag getting lost on its way to the backend.
Previously this was its own cl::opt with a random boolean. Move the
flag handling into CommandFlags with other backend ABI-ish options,
and have clang directly set it, rather than forcing it to go through
command line parsing.
Prior to de68181d7f, codegen used TargetLibraryInfo to find the vector
function. Clang has special handling for TargetLibraryInfo, where it
would
directly construct one with the vector library in the pass pipeline.
RuntimeLibcallsInfo currently is not used as an analysis in codegen, and
needs to know the vector library when constructed.
RuntimeLibraryAnalysis could follow the same trick that
TargetLibraryInfo is using in the future, but a lot more boilerplate changes
are needed to thread that analysis through codegen. Ideally this would come
from an IR module flag, and nothing would be in TargetOptions. For now, it's
better for all of these sorts of controls to be consistent.
Update all uses of variadic `.Cases` to use the initializer list
overload instead. I plan to mark variadic `.Cases` as deprecated in a
followup PR.
For more context, see https://github.com/llvm/llvm-project/pull/163117.
Implement code generation for `__builtin_infer_alloc_token()`. The
`AllocToken` pass is now registered to run unconditionally in the
optimization pipeline. This ensures that all instances of the
`llvm.alloc.token.id` intrinsic are lowered to constant token IDs,
regardless of whether `-fsanitize=alloc-token` is enabled. This
guarantees that the builtin always resolves to a token value, providing
a consistent and reliable mechanism for compile-time token querying.
This completes `__builtin_infer_alloc_token(<malloc-args>, ...)` to
allow compile-time querying of the token ID, where the builtin arguments
mirror those normally passed to any allocation function. The argument
expressions are unevaluated operands. For type-based token modes, the
same type inference logic is used as for untyped allocation calls.
For example the ID that is passed to (with `-fsanitize=alloc-token`):
some_malloc(sizeof(Type), ...)
is equivalent to the token ID returned by
__builtin_infer_alloc_token(sizeof(Type), ...)
The builtin provides a mechanism to pass or compare token IDs in code
that needs to be explicitly allocation token-aware (such as inside an
allocator, or through wrapper macros).
A more concrete demonstration of __builtin_infer_alloc_token's use is
enabling type-aware Slab allocations in the Linux kernel:
https://lore.kernel.org/all/20250825154505.1558444-1-elver@google.com/
Notably, any kind of allocation-call rewriting is a poor fit for the
Linux kernel's kmalloc-family functions, which are macros that wrap
(multiple) layers of inline and non-inline wrapper functions. Given the
Linux kernel defines its own allocation APIs, the more explicit builtin
gives the right level of control over where the type inference happens
and the resulting token is passed.
This PR passes the VFS to LLVM's sanitizer passes from Clang, so that
the configuration files can be loaded in the same way all other compiler
inputs are.
Implement KCFI (Kernel Control Flow Integrity) backend support for
ARM32, Thumb2, and Thumb1. The Linux kernel has supported ARM KCFI via
Clang's generic KCFI implementation, but this has finally started to
[cause problems](https://github.com/ClangBuiltLinux/linux/issues/2124)
so it's time to get the KCFI operand bundle lowering working on ARM.
Supports patchable-function-prefix with adjusted load offsets. Provides
an instruction size worst case estimate of how large the KCFI bundle is
so that range-limited instructions (e.g. cbz) know how big the indirect
calls can become.
ARM implementation notes:
- Four-instruction EOR sequence builds the 32-bit type ID byte-by-byte
to work within ARM's modified immediate encoding constraints.
- Scratch register selection: r12 (IP) is preferred, r3 used as fallback
when r12 holds the call target. r3 gets spilled/reloaded if it is
being used as a call argument.
- UDF trap encoding: 0x8000 | (0x1F << 5) | target_reg_index, similar
to aarch64's trap encoding.
Thumb2 implementation notes:
- Logically the same as ARM
- UDF trap encoding: 0x80 | target_reg_index
Thumb1 implementation notes:
- Due to register pressure, 2 scratch registers are needed: r3 and r2,
which get spilled/reloaded if they are being used as call args.
- Instead of EOR, add/lsl sequence to load immediate, followed by
a compare.
- No trap encoding.
Update tests to validate all three sub targets.
Move the `AllocTokenMax` from `CodeGenOptions` and introduces a new
`AllocTokenMode` to `LangOptions`. Note, `-falloc-token-mode=`
deliberately remains an internal experimental option.
This refactoring is necessary because these options influence frontend
behavior, specifically constexpr evaluation of `__builtin_infer_alloc_token`.
Placing them in `LangOptions` makes them accessible during semantic analysis,
which occurs before codegen.
> clang -flto -funified-lto -save-temps test.c
Fails with the following error message:
```bash
module flag identifiers must be unique (or of 'require' type)
!"UnifiedLTO"
fatal error: error in backend: Broken module found, compilation aborted!
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
```
Here is what the driver does when `-save-temps` flag is set:
> clang -flto -funified-lto -save-temps test.c -ccc-print-phases
> +- 0: input, "test.c", c
> +- 1: preprocessor, {0}, cpp-output
> +- 2: compiler, {1}, ir
> +- 3: backend, {2}, lto-bc
> 4: linker, {3}, image
The IR output of "compiler" step has "UnifiedLTO" module flag. "backend"
step adds another module flag with "UnifiedLTO" identifier, which
invalidates the LLVM IR module.
This PR starts using the correct VFS in `GCOVProfilerPass` instead of
using the real FS directly. This matches compiler's behavior for other
input files.
This PR loads the path from `-fembed-offload-object=<path>` through the
VFS rather than going straight to the real file system. This matches the
behavior of other input files of the compiler. This technically changes
behavior in that `-fembed-offload-object=-` no longer loads the file
from stdin, but I don't think that was the intention of the original
code anyways.
This PR loads the path from `-fbasic-block-sections=list=<path>` through
the VFS rather than going straight to the real file system. This matches
the behavior of other input files of the compiler.
Some LLVM passes need access to the filesystem to read configuration
files and similar. In some places, this is achieved by grabbing the VFS
from `PGOOptions`, but some passes don't have access to these and resort
to just calling `vfs::getRealFileSystem()`. This PR allows setting the
VFS directly on `PassBuilder` that's able to pass it down to all passes
that need it.
Currently there are two serialization modes for bitstream Remarks:
standalone and separate. The separate mode splits remark metadata (e.g.
the string table) from actual remark data. The metadata is written into
the object file by the AsmPrinter, while the remark data is stored in a
separate remarks file. This means we can't use bitstream remarks with
tools like opt that don't generate an object file. Also, it is confusing
to post-process bitstream remarks files, because only the standalone
files can be read by llvm-remarkutil. We always need to use dsymutil
to convert the separate files to standalone files, which only works for
MachO. It is not possible for clang/opt to directly emit bitstream
remark files in standalone mode, because the string table can only be
serialized after all remarks were emitted.
Therefore, this change completely removes the separate serialization
mode. Instead, the remark string table is now always written to the end
of the remarks file. This requires us to tell the serializer when to
finalize remark serialization. This automatically happens when the
serializer goes out of scope. However, often the remark file goes out of
scope before the serializer is destroyed. To diagnose this, I have added
an assert to alert users that they need to explicitly call
finalizeLLVMOptimizationRemarks.
This change paves the way for further improvements to the remark
infrastructure, including more tooling (e.g. #159784), size optimizations
for bitstream remarks, and more.
Pull Request: https://github.com/llvm/llvm-project/pull/156715
This reverts commit 895cda70a95529fd22aac05eee7c34f7624996af.
And fix attempt: 06f671e57a574ba1c5127038eff8e8773273790e.
Performance regressions and broken sanitizers, see #142686.
This patch adds the flag -fexperimental-loop-fuse to the clang and flang
drivers. This is primarily useful for experiments as we envision to
enable the pass one day.
The options are based on the same principles and reason on which we have
`floop-interchange`.
---------
Co-authored-by: Madhur Amilkanthwar <madhura@nvidia.com>
Remove "approx-func-fp-math" attribute and related command line option,
users should always use afn flag in IR.
Resolve FIXME in `TargetMachine::resetTargetOptions` partially.
This PR removes the command line parsing workaround introduced in
https://github.com/llvm/llvm-project/pull/146342 by moving
`LangOptions::ExceptionHandling` to `CodeGenOptions` that get parsed
even for IR input. Additionally, this improves layering, where the
codegen library now checks `CodeGenOptions` instead of `LangOptions` for
exception handling. (This got enabled by
https://github.com/llvm/llvm-project/pull/146422.)
In a previous commit, the llvm-config-defined macro
LLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING was renamed to
LLVM_ENABLE_DEBUGLOC_TRACKING_COVERAGE. One instance of this in Clang
remains unchanged; this patch renames it, and adds an explicit
llvm-config inclusion to ensure the define doesn't silently get removed.
NFC outside of coverage tracking builds, which we do not currently test.
## Purpose
This patch is one in a series of code-mods that annotate LLVM’s public
interface for export. This patch ensures a few `cl::opt` declarations
are properly annotated with `LLVM_ABI`. The annotations currently have
no meaningful impact on the LLVM build; however, they are a prerequisite
to support an LLVM Windows DLL (shared library) build.
## Background
This effort is tracked in #109483. Additional context is provided in
[this
discourse](https://discourse.llvm.org/t/psa-annotating-llvm-public-interface/85307),
and documentation for `LLVM_ABI` and related annotations is found in the
LLVM repo
[here](https://github.com/llvm/llvm-project/blob/main/llvm/docs/InterfaceExportAnnotations.rst).
## Overview
- Remove local `extern` declarations of `llvm::PrintPipelinePasses`
because it is already correctly declared with an `LLVM_ABI` annotation
in `llvm\Passes\PassBuilder.h`. Leaving these declarations results in a
gcc compile warning unless they are also annotated with `LLVM_ABI`.
- Similarly, remove local `extern` declarations of
`ProfileSummaryCutoffHot` and `UseContextLessSummary` from
`llvm/tools/llvm-profgen/ProfileGenerator.cpp` since they are declared
with `LLVM_ABI` in `llvm\ProfileData\ProfileCommon.h`.
- Explicitly annotate the extern declaration of `ProfileCorrelate` in
`clang/lib/CodeGen/BackendUtil.cpp` since it is not declared in a
header. The definition of `ProfileCorrelate` in
`llvm\lib\Transforms\Instrumentation\InstrProfiling.cpp` is already
annotated with `LLVM_ABI`.
## Validation
Local builds and tests to validate cross-platform compatibility. This
included llvm, clang, and lldb on the following configurations:
- Windows with MSVC
- Windows with Clang
- Linux with GCC
- Linux with Clang
- Darwin with Clang
This PR resubmits the changes from #136098, which was previously
reverted due to a build failure during the linking stage:
```
undefined reference to `llvm::DebugInfoCorrelate'
undefined reference to `llvm::ProfileCorrelate'
```
The root cause was that `llvm/lib/Frontend/Driver/CodeGenOptions.cpp`
references symbols from the `Instrumentation` component, but the
`LINK_COMPONENTS` in the `llvm/lib/Frontend/CMakeLists.txt` for
`LLVMFrontendDriver` did not include it. As a result, linking failed in
configurations where these components were not transitively linked.
### Fix:
This updated patch explicitly adds `Instrumentation` to
`LINK_COMPONENTS` in the relevant `llvm/lib/Frontend/CMakeLists.txt`
file to ensure the required symbols are properly resolved.
---------
Co-authored-by: ict-ql <168183727+ict-ql@users.noreply.github.com>
Co-authored-by: Chyaka <52224511+liliumshade@users.noreply.github.com>
Co-authored-by: Tarun Prabhu <tarunprabhu@gmail.com>
When using `-fcs-generate-profile` with distributed thin-lto in the same
fashion we do for local thin-lto, we hit the following assertion:
6041c745f3/llvm/lib/Support/PGOOptions.cpp (L36)
Using local thin-lto with LLD for MachO, we set the missing path
automatically to a default value: https://reviews.llvm.org/D151589. In
this fix we add the same behavior.
---------
Co-authored-by: Nuri Amari <nuriamari@fb.com>
Most of the recent development on the MemProfiler has been on the Use part. The instrumentation has been quite stable for a while. As the complexity of the use grows (with undrifting, diagnostics etc) I figured it would be good to separate these two implementations.
This patch implements IR-based Profile-Guided Optimization support in
Flang through the following flags:
- `-fprofile-generate` for instrumentation-based profile generation
- `-fprofile-use=<dir>/file` for profile-guided optimization
Resolves#74216 (implements IR PGO support phase)
**Key changes:**
- Frontend flag handling aligned with Clang/GCC semantics
- Instrumentation hooks into LLVM PGO infrastructure
- LIT tests verifying:
- Instrumentation metadata generation
- Profile loading from specified path
- Branch weight attribution (IR checks)
**Tests:**
- Added gcc-flag-compatibility.f90 test module verifying:
- Flag parsing boundary conditions
- IR-level profile annotation consistency
- Profile input path normalization rules
- SPEC2006 benchmark results will be shared in comments
For details on LLVM's PGO framework, refer to [Clang PGO
Documentation](https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization).
This implementation was developed by [XSCC Compiler
Team](https://github.com/orgs/OpenXiangShan/teams/xscc).
---------
Co-authored-by: ict-ql <168183727+ict-ql@users.noreply.github.com>
Co-authored-by: Tom Eccles <t@freedommail.info>
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
Normally -fsanitize-coverage=stack-depth inserts inline arithmetic to
update thread_local __sancov_lowest_stack. To support stack depth
tracking in the Linux kernel, which does not implement traditional
thread_local storage, provide the option to call a function instead.
This matches the existing "stackleak" implementation that is supported
in Linux via a GCC plugin. To make this coverage more performant, a
minimum estimated stack depth can be chosen to enable the callback mode,
skipping instrumentation of functions with smaller stacks.
With -fsanitize-coverage-stack-depth-callback-min set greater than 0,
the __sanitize_cov_stack_depth() callback will be injected when the
estimated stack depth is greater than or equal to the given minimum.
Reapplied after fixing the config issue that was causing issues following
the previous merge.
This reverts commit fdbf073a86573c9ac4d595fac8e06d252ce1469f.
