This adds a new function-level `flatten` LLVM IR attribute and
implements support for it in the AlwaysInliner pass, bringing LLVM's
behavior in line with GCC.
Previously, the `flatten` attribute only existed as a Clang attribute,
which was lowered to `alwaysinline` on individual call sites. Per the
RFC discussion [1], the consensus was to match GCC semantics:
recursively inline the entire call tree into the
flattened function, rather than just immediate call sites.
This PR:
- Adds the `flatten` function attribute to LLVM IR
- Implements recursive inlining of all viable callees in AlwaysInliner
- Uses inline history tracking to detect and stop at recursive call
cycles
- Emits optimization remarks when inlining is skipped due to recursion
A follow-up patch will update Clang to emit the LLVM `flatten` attribute
on
functions instead of marking individual call sites with `alwaysinline`.
[1]
https://discourse.llvm.org/t/rfc-function-level-flatten-depth-attribute-for-depth-limited-inlining
Adds support for `atomicrmw` `fminimumnum`/`fmaximumnum` operations.
These were added to C++ in P3008, and are exposed in libc++ in #186716 .
Adding LLVM IR support for these unblocks work in both backends with HW
support, and frontends.
This patch relands https://github.com/llvm/llvm-project/pull/178666. The
original version caused CI failures due to the missing target triple in
`llvm/test/CodeGen/X86/byte-constants.ll`. CI should be green now.
Now that CondBrInst and UncondBrInst are explicit subclasses, use them
instead.
HotColdSplitting was trying to inspect prof metadata also on
unconditional branches, fix this.
Also introduce C API cast functions and deprecate LLVMIsConditional in
favor of LLVMIsACondBrInst.
This patch covers all LLVM uses outside of Transforms, Analysis,
CodeGen/Target, SandboxIR, Frontend/OpenMP, tools, examples.
When building the combined summary index during a thin link, we already
performed a memory optimization for non-prevailing copies of a function
by not recording their allocation and callsite info in the associated
function summary. We can save on the thin link time as well by avoiding
building the memprof summary structures just to throw them away later
in the non-prevailing case.
The reason we were eagerly building these structures is that the memprof
summaries *precede* the corresponding function summary record, and we
don't know whether this is the prevailing copy of the function until we
parse the function summary record. To facilitate the new handling, we
emit the memprof summary records *after* the corresponding function
summary record. The bitcode summary version is bumped, and the reader is
changed to support both versions, for backwards compatibility. Note that
there is already a memprof test that tests an older record type and will
also test reading of the legacy version of the ordering:
(llvm/test/ThinLTO/X86/memprof-old-alloc-context-summary.ll.
To make the new handling even more efficient, the lookup/insertion of
stack IDs in the combined summary index and the caching of their
corresponding stack index in the StackIdToIndex map is made lazy.
This resulted in a 27% reduction in thin link time for a large target
(21% without the lazy insertion change).
BranchInst currently represents both unconditional and conditional
branches. However, these are quite different operations that are often
handled separately. Therefore, split them into separate opcodes and
classes to allow distinguishing these operations in the type system.
Additionally, this also slightly improves compile-time performance.
Currently for thin-lto, the imported static global values (functions,
variables, etc) will be promoted/renamed from e.g., foo() to
foo.llvm.(). Such a renaming caused difficulties in live patching
since function name is changed ([1]).
It is possible that some global value names have to be promoted to avoid
name collision and linker failure. But in practice, majority of name
promotions can be avoided.
In [2], the suggestion is that thin-lto pre-link decides whether
a particular global value needs name promotion or not. If yes, later on
in thinBackend() the name will be promoted.
I compiled a particular linux kernel version (latest bpf-next tree)
and found 1216 global values with suffix .llvm.. With this patch,
the number of promoted functions is 2, 98% reduction from the
original kernel build.
If some native objects are not participating with LTO, name promotions
have to be done to avoid potential linker issues. So the current
implementation cannot be on by default. But in certain cases, e.g., linux kernel
build, people can enable lld flag --lto-whole-program-visibility to reduce the
number of functions like foo.llvm.().
For ThinLTOCodeGenerator.cpp which is used by llvm-lto tool and a
few other rare cases, reducing the number of renaming due to promotion,
is not implemented as lld flag '-lto-whole-program-visibility' is not
supported in ThinLTOCodeGenerator.cpp for now. In summary, this pull
request only supports llvm-lto2 style workflow.
The feature is off by default. To enable the future, lld flag
'-lto-whole-program-visibility' and llvm flag
'-always-rename-promoted-locals=false' are needed.
The link [3] has more context for the pull request discussions.
[1] https://lpc.events/event/19/contributions/2212
[2] https://discourse.llvm.org/t/rfc-avoid-functions-like-foo-llvm-for-kernel-live-patch/89400
[3] https://github.com/llvm/llvm-project/pull/178587
Currently for thin-lto, the imported static global values (functions,
variables, etc) will be promoted/renamed from e.g., foo() to
foo.llvm.<hash>(). Such a renaming caused difficulties in live patching
since function name is changed ([1]).
It is possible that some global value names have to be promoted to avoid
name collision and linker failure. But in practice, majority of name
promotions can be avoided.
In [2], the suggestion is that thin-lto pre-link decides whether
a particular global value needs name promotion or not. If yes, later on
in thinBackend() the name will be promoted.
I compiled a particular linux kernel version (latest bpf-next tree)
and found 1216 global values with suffix .llvm.<hash>. With this patch,
the number of promoted functions is 2, 98% reduction from the
original kernel build.
If some native objects are not participating with LTO, name promotions
have to be done to avoid potential linker issues. So the current
implementation cannot be on by default. But in certain cases, e.g., linux kernel
build, people can enable lld flag --lto-whole-program-visibility to reduce the
number of functions like foo.llvm.<hash>().
For ThinLTOCodeGenerator.cpp which is used by llvm-lto tool and a
few other rare cases, reducing the number of renaming due to promotion,
is not implemented as lld flag '-lto-whole-program-visibility' is not supported
in ThinLTOCodeGenerator.cpp for now. In summary, this pull request
only supports llvm-lto2 style workflow.
[1] https://lpc.events/event/19/contributions/2212
[2] https://discourse.llvm.org/t/rfc-avoid-functions-like-foo-llvm-for-kernel-live-patch/89400
Introduce StackIdToIndex to ModuleSummaryIndexBitcodeReader to cache the
mapping from module-local stack id indices to the global index in the
ModuleSummaryIndex's StackIds vector. This avoids repeated hash lookups
when processing callsite and allocation records.
This reduced the thin link time for a large target built with memprof
by ~16%.
Also add assertions to ensure STACK_IDS records are processed once and
that the cache is empty initially.
This change turns the `"nooutline"` attribute into an enum attribute
called `nooutline`, and adds an auto-upgrader for bitcode to make the
same change to existing IR.
This IR attribute disables both the Machine Outliner (enabled at Oz for
some targets), and the IR Outliner (disabled by default).
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
This is an attempt to merge https://reviews.llvm.org/D144006 with LTO
fix.
The last merge attempt was
https://github.com/llvm/llvm-project/pull/75385.
The issue with it was investigated in
https://github.com/llvm/llvm-project/pull/75385#issuecomment-2386684121.
The problem happens when
1. Several modules are being linked.
2. There are several DISubprograms that initially belong to different
modules but represent the same source code function (for example, a
function included from the same source code file).
3. Some of such DISubprograms survive IR linking. It may happen if one
of them is inlined somewhere or if the functions that have these
DISubprograms attached have internal linkage.
4. Each of these DISubprograms has a local type that corresponds to the
same source code type. These types are initially from different modules,
but have the same ODR identifier.
If the same (in the sense of ODR identifier/ODR uniquing rules) local
type is present in two modules, and these modules are linked together,
the type gets uniqued. A DIType, that happens to be loaded first,
survives linking, and the references on other types with the same ODR
identifier from the modules loaded later are replaced with the
references on the DIType loaded first. Since defintion subprograms, in
scope of which these types are located, are not deduplicated, the linker
output may contain multiple DISubprogram's having the same (uniqued)
type in their retainedNodes lists.
Further compilation of such modules causes crashes.
To tackle that,
* previous solution to handle LTO linking with local types in
retainedNodes is removed (cloneLocalTypes() function),
* for each loaded distinct (definition) DISubprogram, its retainedNodes
list is scanned after loading, and DITypes with a scope of another
subprogram are removed. If something from a Function corresponding to
the DISubprogram references uniqued type, we rely on cross-CU links.
Additionally:
* a check is added to Verifier to report about local types located in a
wrong retainedNodes list,
Original commit message follows.
---------
RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544
Similar to imported declarations, the patch tracks function-local types in
DISubprogram's 'retainedNodes' field. DwarfDebug is adjusted in accordance with
the aforementioned metadata change and provided a support of function-local
types scoped within a lexical block.
The patch assumes that DICompileUnit's 'enums field' no longer tracks local
types and DwarfDebug would assert if any locally-scoped types get placed there.
Authored-by: Kristina Bessonova <kbessonova@accesssoftek.com>
Co-authored-by: Jeremy Morse <jeremy.morse@sony.com>
This removes most of the handling of the relative block frequency
support added in 2018 in c73cec84c99e5a63dca961fef67998a677c53a3c, which
was disabled by default and never utilized in the thin link as expected.
Support for reading old Bitcode containing the record is maintained as
required for backwards compatibility requirements, as is the support for
parsing old LLVM assembly containing that information. Tests ensure that
this backwards compatibility is maintained.
This came up in the context of redundant BFI/DT computations which
existed largely for the purpose of computing this information
and are being addressed in PR176646.
This patch makes the dead_on_return parameter attribute optionally require a number
of bytes to be passed in to specify the number of bytes known to be dead
upon function return/unwind. This is aimed at enabling annotating the
this pointer in C++ destructors with dead_on_return in clang. We need
this to handle cases like the following:
```
struct X {
int n;
~X() {
this[n].n = 0;
}
};
void f() {
X xs[] = {42, -1};
}
```
Where we only certain that sizeof(X) bytes are dead upon return of ~X.
Otherwise DSE would be able to eliminate the store in ~X which would not
be correct.
This patch only does the wiring within IR. Future patches will make
clang emit correct sizing information and update DSE to only delete
stores to objects marked dead_on_return that are provably in bounds of
the number of bytes specified to be dead_on_return.
Reviewers: nikic, alinas, antoniofrighetto
Pull Request: https://github.com/llvm/llvm-project/pull/171712
SwitchInst case values must be ConstantInt, which have no use list.
Therefore it is not necessary to store these as Use, instead store them
more efficiently as a simple array of pointers after the uses, similar
to how PHINode stores basic blocks.
After this change, the successors of all terminators are stored
consecutively in the operand list. This is preparatory work for
improving the performance of successor access.
Add new C API functions so that switch case values remain accessible
from bindings for other languages.
While this could also be achieved by merely changing the order of
operands (i.e., first all successors, then all constants), doing so
would increase the asymptotic runtime of addCase from O(1) to O(n)
(i.e., adding n cases would be O(n^2)), because it would need to shift
all constants by one slot. Having null/invalid operands is also a bad
idea and would cause much more breakage.
Pull Request: https://github.com/llvm/llvm-project/pull/170984
One of the most common mistakes when working with the LLVM C API is to
mix functions that work on the global context and those that work on an
explicit context. This often results in seemingly nonsensical errors
because types from different contexts are mixed.
We have considered the APIs working on the global context to be obsolete
for a long time already, and do not add any new APIs using the global
context. However, the fact that these still exist (and have shorter
names) continues to cause issues.
This PR proposes to deprecate these APIs, with intent to remove them at
some point in the future.
RFC:
https://discourse.llvm.org/t/rfc-deprecate-c-api-functions-using-the-global-context/88639
SwitchInst case values must be ConstantInt, which have no use list.
Therefore it is not necessary to store these as Use, instead store them
more efficiently as a simple array of pointers after the uses, similar
to how PHINode stores basic blocks.
After this change, the successors of all terminators are stored
consecutively in the operand list. This is preparatory work for
improving the performance of successor access.
Deactivation symbol operands are supported in the code generator by
building on the previously added support for IRELATIVE relocations.
Reviewers: ojhunt, fmayer, ahmedbougacha, nikic, efriedma-quic
Reviewed By: fmayer
Pull Request: https://github.com/llvm/llvm-project/pull/133537
For swift async code, we need to use a debug intrinsic that behaves like
an llvm.dbg.declare but can take any location type rather than just a
pointer or integer.
To solve this, a new debug instrinsic called llvm.dbg.declare_value has
been created, which behaves exactly like an llvm.dbg.declare but can
take non pointer and integer location types.
More information here:
https://discourse.llvm.org/t/rfc-introduce-new-llvm-dbg-coroframe-entry-intrinsic/88269
This is the first patch as part of a stack of patches, with the one
succeeding it being: https://github.com/llvm/llvm-project/pull/168134
Also add a corresponding intrinsic property that can be used to mark
intrinsics that do not introduce poison, for example simple arithmetic
intrinsics that propagate poison just like a simple arithmetic
instruction.
As a smoke test this patch adds the new property to
llvm.amdgcn.fmul.legacy.
DW_TAG_base_type DIEs are permitted to have both byte_size and bit_size
attributes "If the value of an object of the given type does not fully
occupy the storage described by a byte size attribute"
* Add DataSizeInBits to DIBasicType (`DIBasicType(... dataSize: n ...)` in IR).
* Change Clang to add DataSizeInBits to _BitInt type metadata.
* Change LLVM to add DW_AT_bit_size to base_type DIEs that have non-zero
DataSizeInBits.
TODO: Do we need to emit DW_AT_data_bit_offset for big endian targets?
See discussion on the PR.
Fixes [#61952](https://github.com/llvm/llvm-project/issues/61952)
---------
Co-authored-by: David Stenberg <david.stenberg@ericsson.com>
Depends on:
* https://github.com/llvm/llvm-project/pull/165401
We weren't testing `DIObjCProperty` roundtripping. So this was never
caught.
The consequence of this is that the `setter:` would have the getter name
and `getter:` would have the setter name.
Module flag is used to indicate the feature to be propagated to the
function. As now the frontend emits all attributes accordingly let's
help the auto upgrade to only do work when old and new bitcodes are
merged.
Depends on #82819 and #86031
In preparation for a follow on change that will require checking every
time a new summary is added to the SummaryList for a GUID, make the
SummaryList private and require all accesses to go through one of two
new interfaces. Most changes are to access the list via the read only
getSummaryList() method, and the few that add new summaries (e.g. while
building the combined summary) use the new addSummary() method.
This patch moves the `preserve-bc-uselistorder` and
`preserve-ll-uselistorder` options out of individual tools(opt, llvm-as,
llvm-dis, llvm-link, llvm-extract) and make them global defaults for
AsmWriter and BitcodeWriter.
These options are useful when we use `-print-*` options to dump LLVM IR.
This patch sets up `DICompileUnit` to support the DWARFv6
`DW_AT_language_name` and `DW_AT_language_version` attributes (which are
set to replace `DW_AT_language`). This patch changes the
`DICompileUnit::SourceLanguage` field type to a `DISourceLanguageName`
that encapsulates the notion of "versioned vs. unversioned name". A
"versioned" name is one that has an associated version stored separately
in `DISourceLanguageName::Version`.
This patch just changes all the clients of the `getSourceLanguage` API
to the expect a `DISourceLanguageName`. Currently they all just `assert`
(via `DISourceLanguageName::getUnversionedName`) that we're dealing with
"unversioned names" (i.e., the pre-DWARFv6 language codes). In follow-up
patches (e.g., draft is at
https://github.com/llvm/llvm-project/pull/162261), when we start
emitting versioned language codes, the `getUnversionedName` calls can
then be adjusted to `getName`.
**Implementation considerations**
* We could have added a new member to `DICompileUnit` alongside the
existing `SourceLanguage` field. I don't think this would have made the
transition any simpler (clients would still need to be aware of
"versioned" vs. "unversioned" language names). I felt that encapsulating
this inside a `DISourceLanguageName` was easier to reason about for
maintainers.
* Currently DISourceLanguageName is a `12` byte structure. We could
probably pack all the info inside a `uint64_t` (16-bits for the name,
32-bits for the version, 1-bit for answering the `hasVersionedName`).
Just to keep the prototype simple I used a `std::optional`. But since
the guts of the structure are hidden, we can always change the layout to
a more compact representation instead.
**How to review**
* The new `DISourceLanguageName` structure is defined in
`DebugInfoMetadata.h`. All the other changes fall out from changing the
`DICompileUnit::SourceLanguage` from `unsigned` to
`DISourceLanguageName`.
Add a new named module-level frontend-annotated metadata that
specifies the TBAA node for an integer access, for which, C/C++
`errno` accesses are guaranteed to use (under strict aliasing).
This should allow LLVM to prove the involved memory location/
accesses may not alias `errno`; thus, to perform optimizations
around errno-writing libcalls (store-to-load forwarding amongst
others).
Previous discussion: https://discourse.llvm.org/t/rfc-modelling-errno-memory-effects/82972.
Teach the IR parser and writer to support metadata on ifuncs, and update
documentation.
In PR #153049, we have a use case of attaching the `!associated`
metadata to an ifunc.
Since an ifunc is similar to a function declaration, it seems natural to
allow metadata on ifuncs.
Currently, the metadata API allows adding Metadata to
llvm::GlobalObject, so the in-memory IR allows for metadata on ifuncs,
but the IR reader/writer is not aware of that.
---------
Co-authored-by: Wael Yehia <wyehia@ca.ibm.com>
- Remove forward declaration of `llvm::Argument` and include Argument.h
instead.
- Restrict scope of anonymous namespaces to just class declarations.
- Move local static function out of anonymous namespace.
- Remove a redundant assert when indexing a vector.
In order to better see what's going on during ThinLTO linking, this PR
adds more profile tags when using `--time-trace` on a `lld-link.exe`
invocation.
After PR, linking `clang.exe`:
<img width="3839" height="2026" alt="Capture d’écran 2025-09-02 082021"
src="https://github.com/user-attachments/assets/bf0c85ba-2f85-4bbf-a5c1-800039b56910"
/>
Linking a custom (Unreal Engine game) binary gives a completly
different picture, probably because of using Unity files, and the sheer
amount of input files (here, providing over 60 GB of .OBJs/.LIBs).
<img width="1940" height="1008" alt="Capture d’écran 2025-09-02 102048"
src="https://github.com/user-attachments/assets/60b28630-7995-45ce-9e8c-13f3cb5312e0"
/>
MaxAlignment is used to produce the abbreviation for MODULE_CODE_GLOBALVAR
and is not used for anything related to function alignments, so stop
combining function alignments and rename it to make its purpose clearer.
Reviewers: teresajohnson
Reviewed By: teresajohnson
Pull Request: https://github.com/llvm/llvm-project/pull/155341
This introduces a new `ptrtoaddr` instruction which is similar to
`ptrtoint` but has two differences:
1) Unlike `ptrtoint`, `ptrtoaddr` does not capture provenance
2) `ptrtoaddr` only extracts (and then extends/truncates) the low
index-width bits of the pointer
For most architectures, difference 2) does not matter since index (address)
width and pointer representation width are the same, but this does make a
difference for architectures that have pointers that aren't just plain
integer addresses such as AMDGPU fat pointers or CHERI capabilities.
This commit introduces textual and bitcode IR support as well as basic code
generation, but optimization passes do not handle the new instruction yet
so it may result in worse code than using ptrtoint. Follow-up changes will
update capture tracking, etc. for the new instruction.
RFC: https://discourse.llvm.org/t/clarifiying-the-semantics-of-ptrtoint/83987/54
Reviewed By: nikic
Pull Request: https://github.com/llvm/llvm-project/pull/139357
