Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
Relying on the semantics of the type here is a bit potentially awkward,
since the full allocated space may be accessible to the user if desired,
since that space is defined to be a part of the type's sizeof
computation (e.g. for a `memcpy(gv, gv, sizeof(*gv))` or when making an
array of them). It also gets in the way of removing getAllocatedType
from AllocaInst entirely (they are converted to GlobalVariable
sometimes). It was originally added in
519561f418c77dcf46fd6d96d25d884fa07fd7da, though "correct size" is a
difficult thing to define.
The frontend (in clang) appears to always emit the full type size here,
so there seems like this shouldn't be visible change to clang users.
This is still a bit awkward though, since a global is defined to be any
size that is bigger than this unless it has a known initizalizer,
rendering the test still incomplete here against the IR semantics.
This patch changes the memset lowering to match the optimized memcpy lowering.
The memset lowering now queries TTI.getMemcpyLoopLoweringType for a preferred
memory access type. If that type is larger than a byte, the memset is lowered
into two loops: a main loop that stores a sufficiently wide vector splat of the
SetValue with the preferred memory access type and a residual loop that covers
the remaining bytes individually. If the memset size is statically known, the
residual loop is replaced by a sequence of stores.
This improves memset performance on gfx1030 (AMDGPU) in microbenchmarks by
around 7-20x.
I'm planning similar treatment for memset.pattern as a follow-up PR.
For SWDEV-543208.
This is an attempt to merge https://reviews.llvm.org/D144006 with LTO
fix.
The last merge attempt was
https://github.com/llvm/llvm-project/pull/75385.
The issue with it was investigated in
https://github.com/llvm/llvm-project/pull/75385#issuecomment-2386684121.
The problem happens when
1. Several modules are being linked.
2. There are several DISubprograms that initially belong to different
modules but represent the same source code function (for example, a
function included from the same source code file).
3. Some of such DISubprograms survive IR linking. It may happen if one
of them is inlined somewhere or if the functions that have these
DISubprograms attached have internal linkage.
4. Each of these DISubprograms has a local type that corresponds to the
same source code type. These types are initially from different modules,
but have the same ODR identifier.
If the same (in the sense of ODR identifier/ODR uniquing rules) local
type is present in two modules, and these modules are linked together,
the type gets uniqued. A DIType, that happens to be loaded first,
survives linking, and the references on other types with the same ODR
identifier from the modules loaded later are replaced with the
references on the DIType loaded first. Since defintion subprograms, in
scope of which these types are located, are not deduplicated, the linker
output may contain multiple DISubprogram's having the same (uniqued)
type in their retainedNodes lists.
Further compilation of such modules causes crashes.
To tackle that,
* previous solution to handle LTO linking with local types in
retainedNodes is removed (cloneLocalTypes() function),
* for each loaded distinct (definition) DISubprogram, its retainedNodes
list is scanned after loading, and DITypes with a scope of another
subprogram are removed. If something from a Function corresponding to
the DISubprogram references uniqued type, we rely on cross-CU links.
Additionally:
* a check is added to Verifier to report about local types located in a
wrong retainedNodes list,
Original commit message follows.
---------
RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544
Similar to imported declarations, the patch tracks function-local types in
DISubprogram's 'retainedNodes' field. DwarfDebug is adjusted in accordance with
the aforementioned metadata change and provided a support of function-local
types scoped within a lexical block.
The patch assumes that DICompileUnit's 'enums field' no longer tracks local
types and DwarfDebug would assert if any locally-scoped types get placed there.
Authored-by: Kristina Bessonova <kbessonova@accesssoftek.com>
Co-authored-by: Jeremy Morse <jeremy.morse@sony.com>
ptrauth intrinsic to safely construct relative ptr for swift coroutines.
A ptrauth intrinsic for swift co-routine support that allows creation of
signed pointer
from offset stored at address relative to the pointer.
Following C-like pseudo code (ignoring keys,discriminators) explains its
operation:
let rawptr = PACauth(inputptr);
return PACsign( rawptr + signextend64( *(int32*)(rawptr+addend) ))
What: Authenticate a signed pointer, load a 32bit value at offset
'addend' from pointer,
add this value to pointer, sign this new pointer.
builtin: __builtin_ptrauth_auth_load_relative_and_sign
intrinsic: ptrauth_auth_resign_load_relative
https://github.com/llvm/llvm-project/pull/178895 caused a clang
crash(see https://lab.llvm.org/buildbot/#/builders/210/builds/8229),
reverted in 6d52d2683c2ceb9ab75810730c3ced2509c32bc5.
The crash is assertion `DT && "DT should be available to update
LoopInfo!"' failed.
ad8d5349d4/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp (L1106)
```
#7 0x00007f5a380254e3 __assert_perror_fail (/usr/lib/libc.so.6+0x254e3)
#8 0x0000563df5d8fde1 UpdateAnalysisInformation(llvm::BasicBlock*, llvm::BasicBlock*, llvm::ArrayRef<llvm::BasicBlock*>, llvm::DomTreeUpdater*, llvm::DominatorTree*, llvm::LoopInfo*, llvm::MemorySSAUpdater*, bool, bool&) BasicBlockUtils.cpp:0:0
#9 0x0000563df5d8f3bb llvm::splitBlockBefore(llvm::BasicBlock*, llvm::ilist_iterator_w_bits<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void, true, llvm::BasicBlock>, false, false>, llvm::DomTreeUpdater*, llvm::LoopInfo*, llvm::MemorySSAUpdater*, llvm::Twine const&)
#10 0x0000563df5d8cb08 llvm::SplitEdge(llvm::BasicBlock*, llvm::BasicBlock*, llvm::DominatorTree*, llvm::LoopInfo*, llvm::MemorySSAUpdater*, llvm::Twine const&)
#11 0x0000563df4ff5b59 (anonymous namespace)::CodeGenPrepare::splitLargeGEPOffsets()::$_1::operator()(long, llvm::Value*, llvm::GetElementPtrInst*) const CodeGenPrepare.cpp:0:0
#12 0x0000563df4fc0ec8 (anonymous namespace)::CodeGenPrepare::_run(llvm::Function&) CodeGenPrepare.cpp:0:0
#13 0x0000563df4fbb36c (anonymous namespace)::CodeGenPrepareLegacyPass::runOnFunction(llvm::Function&) CodeGenPrepare.cpp:0:0
```
I think this happened when we get DominatorTree with `DT.get()` in
`splitLargeGEPOffsets()` but `DT.reset()` already setting it to nullptr
in
ad8d5349d4/llvm/lib/CodeGen/CodeGenPrepare.cpp (L660).
To fix this assertion failure, use `getDT()` for
`splitLargeGEPOffsets()` to build the DominatorTree if it is set to
nullptr by `DT.reset()`.
I don't have a RSIC-V environment, so no reproducer. Checked that the
crash is fixed by rerunning buildbot with this patch
https://lab.llvm.org/buildbot/#/builders/210/builds/8248
06dfbb50d70eea4ae38d655842626a0b9b224d5c fixed dominator update for
entry block in `SplitBlockPredecessors()`, this patch fixes dominator
tree update for entry block in `splitBlockBefore()` with
`UpdateAnalysisInformation()`.
We never need to use this conditionally (and it doesn't really make
sense, as the behavior is substantially different). Force the use of
separate APIs instead of a boolean argument.
This pass does not change the CFG, so mark all CFG analyses as
preserved, instead of DT in particular. This matches what the NewPM
implementation does.
(This currently has no direct benefit as nearby passes end up
invalidating things anyway.)
Darwin reserves slot 231 for storing a pointer to the history ring
buffer. It also uses bits 60-62 to store the size of the ring buffer
rdar://168176496
This patch improves multi-exit loop unrolling by taking into account
branch probability and not only other exit being deopting one.
This implementation uses branch metadata directly because of unstable
state of BPI in this part of code (runtime unrolling invalidates the
state of the map and using BPI in my tests has caused errors).
If branch probability metadata are not present then the current deopt
heuristic is still used.
---------
Co-authored-by: Marek Sedlacek <msedlacek@azul.com>
Previously, `gatherIncomingValuesToPhi` populated the `IncomingValues`
map with *all* non-undef incoming values from the PHI node. For PHI
nodes with a large number of incoming blocks, this caused the
`SmallDenseMap` to grow significantly, triggering expensive resizing and
rehashing operations, even when the caller
(`redirectValuesFromPredecessorsToPhi`) was only interested in a small
subset of predecessors.
This patch optimizes the logic to prevent this unnecessary map growth.
Instead of collecting all values, we now:
1. Initialize the `IncomingValues` map specifically for the blocks in
`BBPreds` (setting them to `nullptr` initially).
2. Iterate through the PHI node and update the map entries only if the
incoming block is already present in the map.
This ensures that the size of the map is bounded by the size of
`BBPreds`. Since `BBPreds` is typically small, this change keeps the map
within the `SmallDenseMap`'s inline storage in most cases, eliminating
heap allocations and resizing overhead for large PHI nodes.
The `selectIncomingValueForBlock` and `replaceUndefValuesInPhi` helpers
are also updated to handle map entries where the value is `nullptr`.
Replace uses of getAllocatedType() in PromoteMemoryToRegister.cpp with
type tracking from actual loads and stores. This makes the promotion
logic more semantic - it now checks that all loads/stores use a
consistent type rather than requiring them to match the alloca's
declared type.
Changes:
- isAllocaPromotable() now tracks the first load/store type seen and
ensures all subsequent accesses use the same type
- AllocaInfo gains a ValueType field populated during AnalyzeAlloca()
- PromoteMem2Reg tracks AllocaValueTypes alongside other per-alloca info
- PHI nodes and UndefValues are created using the tracked type
This is semantically more permissive - an alloca declared as i64 but
only accessed as i32 is now promotable. This is correct because the
alloca is just a blob of memory; what matters for Mem2Reg is consistent
access patterns.
Test changes:
- asan-stack-safety.ll: Changed loads/stores to volatile to prevent
promotion while preserving ASAN stack safety analysis behavior
- SPIRV pointer tests (array-skips-gep.ll, load-struct.ll,
store-struct.ll, store-to-array-first-element.ll): Added escape calls to
prevent alloca promotion, as these tests verify SPIRV backend handling
of Function-storage-class pointers
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Replace the `isSized()` and `isScalableTy()` type predicate checks with
a more direct `getAllocationSize()` query. The `isSized` check seems
unnecessary since the verifier already rejects unsized alloca.
For context, I'm working on deleting (or at least minimizing uses of)
the getAllocatedType function, since many optimizations need to destroy
that info and just produce byte sized allocas. It is for the same
reason that opaque pointers were introduced–memory doesn't inherently
have a provenience type, only uses (load and stores) do.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Previously, for performance reasons, we would only use precise lifetimes
if they cover all reachable exits. Now, if they do not, we use precise
lifetimes in addition to untagging at every exit that is not dominated
by them.
This is the behavior of ASan.
Reviewers: vitalybuka, pcc
Pull Request: https://github.com/llvm/llvm-project/pull/174875
Fixes the miscompilation discussed for the PR #164882 as part of
generalizing the optimization for the issue #150120.
Without this commit, MoveAutoInit moves the store instruction to a
different branch which does not dominate the user dominator node. This
results in UB at runtime. The example in the test case is specifically
for an irreducible loop, in which all the predecessor may not dominate
user dominator head.
To fix this problem, we've introduced a new check to verify if the
predecessor of the user dominator node does in fact dominate user
dominator node before deciding that it is the node where the instruction
will be moved to.
This PR improves the propagation of profile metadata within the
ExpandIRInsts pass. When lowering large integer division operations, the
pass now ensures that branch weights are correctly attached to the
generated control flow, preventing the loss of profile data during IR
expansion.
This PR improves signed and unsigned division/remainder for non-native
bit widths (e.g., `sdiv/udiv i129`, `srem/urem i129`) and implemented
Heuristic-Based Branch Weights labeling using established heuristics for
edge cases e.g., `Division-by-zero guards` and `Magnitude comparisons
between dividends and divisors`.
It also adds detailed comments within the expansion logic to explain the
rationale behind specific branch weight choices and the underlying
mathematical invariants.
Please refer to the implementation details in the source code for the
specific branch weight values and the logic governing their application.
Co-authored-by: Jonas Devlieghere <jonas@devlieghere.com>
This adds the analogous metadata to the nofpclass attribute
to assert values are not a certain set of floating-point classes.
This allows the same information to be expressed if a function
argument is passed indirectly. This matches the bitmask encoding
of nofpclass.
I also think this should be allowed for stores to symmetrically handle
sret, but leave that for later.
Alternatively we could add a more expressive !fprange metadata,
but that would be much more complex. It's useful to match the attribute,
and more annotations can always be added.
Fixes#133560
Returns uint64_t to simplify callers. The goal is eventually replace
getValueType with this query, which should return the known minimum
reference-able size, as provided (instead of a Type) during create.
Additionally the common isSized query would be replaced with an
isExactKnownSize query to test if that size is an exact definition.
This is helpful for tests with a lot of test cases and grepping for
instructions with missing profile data isn't feasible because it doesn't
account for things like vectors which are exempt.
Tracking issue: #147390
When a conditional branch has both successors pointing to the same block
(e.g., `br i1 %cond, label %bb, label %bb`), `ControlFlowHub::finalize`
generates duplicate `Delete` updates for the same CFG edge. This can
cause assertion in `fix-irreducible` pass.
Fixes#176553.
getUpper returns 1 more than the maxium value included in the range.
This may be 0. We should not use this in a umin. Instead we should
get the maximum value included in the range and use that for the umin.
Then convert that to Upper for the new range by adding 1.
The test was manually reduced from a downstream failure, but I couldn't
get it behave exactly the same way without more instructions. It should
be enough to show an incorrect range being calculated.
Fixes#176471
Reduce the size of a DomTreeNodeBase from 80 to 56 bytes by not storing
the children in a SmallVector. Instead, store children as forward-linked
list. This also avoids extra allocations for nodes with many children.
Additionally, DomTreeNodeBase is now trivially destructible.
A lot of code depends on the order of nodes in the dominator tree, so
make sure that the order is the same when inserting nodes. (Not having
to do this would save 8 bytes per node.)
NewGVN uses the order of nodes in the dominator tree in a way that is
not entirely clear to me (https://reviews.llvm.org/D28129). I kept the
semantics as, but now this is the only external user of
addChild/removeChild, which actually should be private.
https://llvm-compile-time-tracker.com/compare.php?from=263802c56b4db3fc9b6ed9fd313499cb03ca44da&to=43e0c0c5b663b3a4067252fc0addbaccefd0014d&stat=instructions:u
Removed unused functions in order to fix 'unused function' warnings, as
mentioned in PR 171889. This involved the two original functions
```ControlConditions::isEquivalent(const ControlConditions &Other)
const``` and ```ControlConditions::collectControlConditions(const
llvm::BasicBlock&, const llvm::BasicBlock&, const llvm::DominatorTree&,
const llvm::PostDominatorTree&, unsigned int)``` plus all the functions
that became unused as the result of deleting the two original ones.
Co-authored-by: Szymon Sobieszek <szymon.sobieszek1@huawei.com>
When LoopUnroll copies the original loop's latch to the corresponding
non-latch branch in an unrolled iteration, any `!llvm.loop` is copied
along with it, but `!llvm.loop` is useless and misleading there. This
patch discards it.
e06831a3b29d did the same for LoopPeel.
Based on Michael Maitland's previous work:
https://github.com/llvm/llvm-project/pull/121222
This PR uses the existing recurrences code instead of introducing a
new pass just for CSA autovec. I've also made recipes that are more
generic.
Corrected various spelling mistakes such as 'occurred', 'receiver',
'initialized', 'length', and others in comments, variable names,
function names, and documentation throughout the project. These
changes improve code readability and maintain consistency in naming
and documentation.
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
If the mem intrinsics have value profile information associated, we can synthesize branch weights when converting them (the intrinsics) to loops.
Issue #147390
We currently encode an estimated trip count of 0 as the latch having branch probabilities 0-0. That's an invalid pair of weights. The probability of a branch is computed as a fraction of its corresponding weight and the sum of the weights. In fact, `BranchProbabilityInfo::calcMetadataWeights` will convert this to a 1-1, meaning 50% - 50%, which isn't quite what we want. To indicate the loop is never taken, we just need to initialize the exit probability to non-zero (hence, 1)
Related: https://reviews.llvm.org/D67905
Issue #147390
Add a new metadata node `!implicit.ref` to represent an implicit
dependency between 2 symbols. The metadata is unique to AIX and gets
lowered to a relocation that adds an explicit link between the section
the global that the metadata is placed on is allocated in, to the
asscoiated symbol. This relocation will cause the associated symbol to
remain live if the section is not garbage collected. This is used mainly
for compiler features where there is some hidden runtime dependency
between the symbols that isn't otherwise obvious to the linker.
IRCE may fail to prove predicates when loop bounds are rewritten by
LoopConstrainer::applyLoopGuards(). In such cases, simple predicates
(e.g. %start < %limit) become obscured by strengthened SCEV expressions,
even though they are trivially implied by the guard in the loop
preheader.
See: https://github.com/llvm/llvm-project/issues/167827
This change makes IRCE attempt to prove predicates both on the original
SCEVs and on the guarded SCEVs, preventing incorrect rejection of loops
with safe bounds.
Previously, we use `std::map<BasicBlock *, unsigned> PredCount` to track
excess incoming blocks and removed them one by one using
`removeIncomingValue`.
Since `PredCount` use `BasicBlock *` as key, the iteration order depends
on the memory addresses of the blocks. With
`PHINode::removeIncomingValue()` changed to use the swapping strategy,
the order in which operands are removed affects the final order of the
remaining operands in the PHI node. This will cause non-determinism in
compiles.
This patch uses `PHINode::removeIncomingValueIf()` to remove invalid
incoming blocks that no longer go to `NewBB` block, fixes the
non-determinism.
