Ensure that frontends can request both `PragmaEnable` and `PragmaFull`
semantics on a loop.
FYI: to the best of my knowledge, it's not possible to toggle both
`PragmaEnable` and `PragmaFull` via `clang`.
When RewriteExprTree modifies instruction operands, call
`salvageDebugInfo()` before `setOperand()` so debug value expressions
are rewritten while original operands are still intact. This preserves
variable accessibility in debuggers instead of dropping values to
poison.
Fixes#60532Fixes#61272
The implementation of getIntrinsicInstrCost in BasicTTIImpl
assumes that for some intrinsics if we're using custom
lowering for the equivalent DAG node that the cost needs to
be 2, instead of 1 for legal ops. However, even though we
use custom lowering for these scalable vector operations
when SVE is available, we still end up generating the same
efficient codegen as fixed-width. This patch deals with a
few obvious intrinsics that we know get lowered to something
sensible and return the same cost as NEON, i.e. 1.
Currently the logic for introducing a header mask and predicating the
vector loop region is done inside introduceMasksAndLinearize.
This splits the tail folding part out into an individual VPlan transform
so that VPlanPredicator.cpp doesn't need to worry about tail folding,
which seemed to be a temporary measure according to a comment in
VPlanTransforms.h.
To perform tail folding independently, this splits the "body" of the
vector loop region between the phis in the header and the branch + iv
increment in the latch:
Before:
```
+-------------------------------------------+
|%iv = ... |
|... |
|%iv.next = add %iv, vfxuf |
|branch-on-count %iv.next, vector-trip-count|
+-------------------------------------------+
```
After:
```
+-------------------------------------------+
|%iv = ... |
|%wide.iv = widen-canonical-iv ... |
|%header-mask = icmp ule %wide.iv, BTC |---+
|branch-on-cond %header-mask | |
+-------------------------------------------+ |
| |
v |
+-------------------------------------------+ |
|... | |
+-------------------------------------------+ |
| |
v |
+-------------------------------------------+ |
|%iv.next = add %iv, vfxuf |<--+
|branch-on-count %iv.next, vector-trip-count|
+-------------------------------------------+
```
Phis are then inserted in the latch for any value in the loop body that
have outside uses, with poison as their incoming value from the header
edge.
The motivation for this is to allow us to share the same "predicate all
successor blocks" type of predication we do for tail folding, but for
early-exit loops in #172454. This may also allow us to directly emit an
EVL based header mask, instead of having to match + transform the
existing header mask in addExplicitVectorLength.
This also allows us to eventually handle recurrences in the same
transform, avoiding the need to special case tail folding in
addReductionResultComputation.
Summary:
This PR simply changes the behavior of the `wchar_size` flag. Currently,
we emit this in all cases for all targets. This causes problems during
LLVM-IR linking, specifically because this would vary between Linux and
Windows in unintuitive ways. Now we have an llvm::Triple helper to
determine the size from the known values. The module flag will only be
emitted if these do not match (indicating a non-standard environment).
In addition to fixing AMDGCN bitcode linking, this also means we don't
need to bloat *every* IR module compiled by clang with this flag. The
changed tests reflects this, one less unnecessary piece of metadata.
Similarly to what already being done in GVN (fb632ed2377d280b581b8d4653b855e60d611f77),
when a dominating equality condition of two pointers holds, and the
value being stored is implied by such a condition, ensure the store
may be removed by leveraging `canReplacePointersIfEqual`, subject to
the known approximations.
Fixes: https://github.com/llvm/llvm-project/issues/184088.
When -globals-aa is used, `analyzeLoadAvailiability` calls
`analyzeLoadClobberringStore` which in turn calls
`isFirstClassAggregateOrScalableType` which is using
`isa<ScalableVectorType>`. Ideally it should use type's isScalableType()
method.
The crash does not occur when default (BasicAA) is used because
`analyzeLoadAvailiability` function does not take the load clobbering
path as BasicAA returns `Def`.
Fixes#159368
This reverts commit ea3fdc5972db7f2d459e543307af05c357f2be26.
Re-enable const-folding for maxnum/minnum in the middle-end, GlobalISel,
and SelectionDAG.
Re-enable optimizations that depend on maxnum/minnum sNaN semantics in
InstCombine and DAGCombiner.
Now that maxnum(x, sNaN) is specified to non-deterministically produce
either NaN or x, these constant-foldings and optimizations are now valid
again according to the newly clarified semantics in #172012 .
This patch teaches the backend how to lower the FCBRT DAG node to the
vector math library function when using ArmPL. This is similar to what
we already do for llvm.pow/FPOW, however the only way to expose this is
via a DAG combine that converts
FPOW(<2 x double> %x, <2 x double> <double 1.0/3.0, double 1.0/3.0>)
into
FCBRT(<2 x double> %x)
when the appropriate fast math flags are present on the node. I've
updated the DAG combine to handle vector types and only perform the
transformation if there exists a vector library variant of cbrt.
In investigating #156233, it came up that select folds like here:
https://alive2.llvm.org/ce/z/Y6jzj6 cannot be carried out, or easily
fixed for now, because integer reductions do not propagate noundef, even
if their arguments are noundef. This patch adds this propagation.
Replace coroutine frame struct type with a simple byte array and use
offset-based ptradd operations instead of struct GEP for all field
access. Alloca types have largely lost all meaning to LLVM (even this
pass merged them and used an arbitrary type to represent all of them),
and so they just makes the code to construct alloca more difficult and
less flexible.
Key changes:
- Remove LayoutFieldIndex from frame field tracking
- Remove StructType usage - frame is now a byte array
- Replace all CreateStructGEP/CreateConstInBoundsGEP with CreatePtrAdd
- Store ResumeOffset/DestroyOffset in SwitchLowering for reuse
- Remove Shape.FrameTy, use Shape.FrameSize directly
Bug fix: Uses pointer size and alignment from data layout for header
pointer offsets in debug info instead of hardcoded 8 byte.
Optimization: Replaces load+store patterns with CreateMemCpy for copying
allocas to the frame more efficiently.
Optimization: Add missing inbounds annotations on existing ptradd calls.
Improvement: Preserve debug info of every alloca, even overlapping ones.
The frame type is now completely opaque at the IR level. All structure
is implicit through computed offsets. Debug info still provides detailed
field information using explicit offsets.
See further information in dependent PR
https://github.com/llvm/llvm-project/pull/178358 as well.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Update groupChecks to always use DepCands to try and merge runtime
checks. DepCands contains the dependency partition, grouping together
all accessed pointers to he same underlying objects.
If we computed the dependencies, We only need to check accesses to the
same underlying object, if there is an unknown dependency for this
underlying object; otherwise we already proved that all accesses withing
the underlying object are safe w.r.t. vectorization and we only need to
check that accesses to the underlying object don't overlap with accesses
to other underlying objects.
To ensure runtime checks are generated for the case with unknown
dependencies, remove equivalence classes containing accesses involved in
unknown dependencies.
This reduces the number of runtime checks needed in case non-constant
dependence distances are found, and is in preparation for removing the
restriction that the accesses need to have the same stride which was
added in https://github.com/llvm/llvm-project/pull/88039.
PR: https://github.com/llvm/llvm-project/pull/91196
When a memory-reading or memory-writing instruction is not a
LoadInst/StoreInst, the dyn_cast to Ld/St returns nullptr, which is then
passed to recordAnalysis. This causes the optimization remark to fall
back to the loop header location instead of pointing at the actual
problematic instruction.
Pass &I (the actual Instruction) instead.
For following code sequence:
```
%cmp = icmp ugt i64 %idx, 1
tail call void @llvm.assume(i1 %cmp)
```
CVP may replace all use of `%cmp` with `true`, and the information about
`%idx` is lost. This commit stop CVP from doing that, so that later pass
can exploit the `llvm.assume`.
Fixes https://github.com/llvm/llvm-project/issues/90206.
When vectorising loops containing calls to the llvm.pow intrinsic we
currently end up with direct calls to the vector math library equivalent
of llvm.pow. This causes problems later on because we then miss out on
optimisations in the backend for things like
pow(x, 0.25)
pow(x, 0.75)
etc.
There are a few different ways of trying to solve this, but I think the
cleanest way is to leave the call in the intrinsic form and then teach
the backend how to lower the FPOW DAG node to the vector math library
function. This is similar to what we already do for llvm.sincos/FSINCOS
today.
I've done this by removing the entries in Analysis/VecFuncs.def that map
the intrinsic to the vector math functions, whilst also teaching the
cost model that the vector intrinsic form of llvm.pow is cheap in the
presence of the library.
The current LangRef wording says that the sign of a zero argument or
result is "insignificant", which is not really clear on what this means.
Alive2 models this as non-deterministically flipping the zero sign bits
for both inputs and outputs.
This PR proposes to specify this flag as non-deterministically flipping
inputs only. A consequence of this is that fabs is guaranteed to have an
unset sign bit even if the input is zero (that is,
https://alive2.llvm.org/ce/z/irCftQ is no longer a valid transform), and
that the copysign result sign only depends on the second operand (that
is, https://alive2.llvm.org/ce/z/VnHdfh is no longer a valid transform).
These rules are still more liberal than we'd really like them to be, but
at least avoid some of the issues with nsz.
This is based on the discussion in:
https://discourse.llvm.org/t/rfc-clarify-the-behavior-of-fp-operations-on-bit-strings-with-nsz-flag/85981
VPExpandSCEVRecipe may become unused after VPlan optimizations. This
patch removes VPExpandSCEVRecipes with no users before expansion in
expandSCEVs, avoiding generating dead code during VPlan execution.
Remove the use of function attributes no-nans-fp-math and
no-signed-zeros-fp-math in FP min/max reduction detection. The required
fast-math flags nnan and nsz should be present on the intrinsic calls,
fcmp and select instructions themselves.
This helps canonicalize some address calculation. This would further
help immediate folding into memory load instructions in the backend.
The order changes to v_mad_u32_u24 is just because
@llvm.amdgcn.mul.u24.i32 was used in codegen prepare after this change.
It does not really change anything important.
If the select/zext comparison has negate predicate and is used in
several places, it should not be considered as a candidate for inversed
zext/select pattern, it will be replaced by a negate vector predicate,
leading to an incorrect codegen for other uses
Licm can now sink reverse VPInstructions outside the loop region; they
won't be considered when computing costs. Account for that in
planContainsAdditionalSimplifications.
Fixes https://github.com/llvm/llvm-project/issues/183592.
This removes the restriction requiring a single predicated early exit.
Using MaskedCond, we only combine early-exit conditions with block
masks from non-exiting control flow.
This means we have to ensure that we check the early exit conditions in
program order, to make sure we take the first exit in program order that
exits at the first lane for the combined exit condition.
To do so, sort the exits by their reverse post-order numbers.
Depends on https://github.com/llvm/llvm-project/pull/182395
PR: https://github.com/llvm/llvm-project/pull/182396
