This patch removes the extra logical-and in `x && (x && y)` and `x && (y && x)` to `x && y`.
This helps to simplify mask calculation in the FindLast reduction and
exposes more opportunities to replace to EVL.
PR link: https://github.com/llvm/llvm-project/pull/185806
This patch tries to drastically simplify resume value handling for the
scalar loop when vectorizing the epilogue.
It uses a simpler, uniform approach for updating all resume values in
the scalar loop:
1. Create ResumeForEpilogue recipes for all scalar resume phis in the
main loop (the epilogue plan will have exactly the same scalar resume
phis, in exactly the same order)
2. Update ::execute for ResumeForEpilogue to set the underlying value
when executing. This is not super clean, but allows easy lookup of the
generated IR value when we update the resume phis in the epilogue. Once
we connect the 2 plans together explicitly, this can be removed.
3. Use the list of ResumeForEpilogue VPInstructions from the main loop
to update the resume/bypass values from the epilogue.
This simplifies the code quite a bit, makes it more robust (should fix
https://github.com/llvm/llvm-project/issues/179407) and also fixes a
mis-compile in the existing tests (see change in
llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub-epilogue-vec.ll,
where previously we would incorrectly resume using the start value when
the epilogue iteration check failed)
In some cases, we get simpler code, due to additional CSE, in some cases
the induction end value computations get moved from the epilogue
iteration check to the vector preheader. We could try to sink the
instructions as cleanup, but it is probably not worth the trouble.
Fixes https://github.com/llvm/llvm-project/issues/179407.
The patch was intially landed as bd5f9384, but then reverted due to an
underlying issue in narrowInterleaveGroups, described in #185860. The
issue has since been fixed. The reland is simply a conflict-resolved
version of the original patch, which includes an additonal test update.
WidenCast is very similar to Widen recipes.
Fixes#128062.
Restrict PHI nodes that getRangeRef is allowed to recursively examine so
we don't need a "visited" set. And fix createSCEVIter so it creates all
the relevant SCEV nodes before getRangeRef tries to examine them.
The tests that are affected have induction variables that aren't
AddRecs. (Other cases are theoretically affected, but don't seem to show
up in our tests.)
Previously, VPlanPredicator only reused the mask of the loop header when
a block post-dominates the header. This patch generalizes the
optimization to reuse the mask of immediate dominator when a block
post-dominates its immediate dominator.
This reduces more redundant mask computations, simplifies the generated
code, and improves EVL tail folding.
Based on #173265Fix#173260
Some VPlan transforms, like vectorizing fmin without fast-math,
introduce AnyOfs, which have costs assigned in the VPlan-based cost
model, but not the legacy cost model. Account for their cost like done
for other similar VPInstrctions, like EVL.
Fixes https://github.com/llvm/llvm-project/issues/185867.
Add test coverage for mis-compile in https://github.com/llvm/llvm-project/issues/185860.
Currently a wide load gets incorrectly narrowed to a single-scalar one
with scalable VFs, which only loads one element instead of the needed
vscale elements.
When encountering a control-flow join, VPPredicator emit a disjunction
over the incoming edge masks as the entry mask of the joining block.
However, such a complex mask is not always necessary. If the block is
control-flow equivalent to the header block, we can directly use the
header block’s entry mask as the entry mask of that block.
This patch introduces a VPlan post-dominator tree to determine whether a
block is control-flow equivalent to the header block, and simplifies the
computation of block masks accordingly.
Based on #178724
I found the logic of this function quite hard to reason about. This
patch attempts to rectify this by splitting out matching an extended
reduction operand and traversing reduction chain.
- `matchExtendedReductionOperand()` contains all the logic to match an
extended operand.
- `getScaledReductions()` validates each operation in the chain,
starting backwards from the exit value, walking up through the operand
that is not extended.
When narrowInterleaveGroups transforms a plan, VF and VFxUF are
materialized (replaced with concrete values). This patch also
materializes the VectorTripCount in the same transform.
This ensures that VectorTripCount is properly computed when the narrow
interleave transform is applied, instead of using the original VF
+ UF to compute the vector trip count. The previous behavior generated
correct code, but executed fewer iterations in the vector loop.
The change also enables stricter verification prevent accesses of UF,
VF, VFxUF etc after materialization as follow-up.
Note that in some cases we no miss branch folding, but that should be
addressed separately, https://github.com/llvm/llvm-project/pull/181252
Fixes one of the violations accessing a VectorTripCount after UF and VF
being materialized
PR: https://github.com/llvm/llvm-project/pull/182146
This duplicates #182313 with some very small modifications on top, as
@dheaton-arm is unable
to finish the PR and I'm unable to push to his branch.
Expands support for the `FindLast` Reccurence Kind to floating-point and
pointer types, thereby
enabling conditional scalar assignment (CSA) for these types.
Originally authored by @dheaton-arm
---------
Co-authored-by: Damian Heaton <Damian.Heaton@arm.com>
This reverts commit d7e037c8383e66e5c07897f144f6d8ef47258682.
Recommit with a small fix to properly handle ordered reductions when
connecting the epilogue.
Original message:
Replace manual region dissolution code in
simplifyBranchConditionForVFAndUF with using general
removeBranchOnConst. simplifyBranchConditionForVFAndUF now just creates
a (BranchOnCond true) or updates BranchOnTwoConds.
The loop then gets automatically removed by running removeBranchOnConst.
This removes a bunch of special logic to handle header phi replacements
and CFG updates. With the new code, there's no restriction on what kind
of header phi recipes the loop contains.
Note that VPEVLBasedIVRecipe needs to be marked as readnone. This is
technically unrelated, but I could not find an independent test that
would be impacted.
The code to deal with epilogue resume values now needs updating, because
we may simplify a reduction directly to the start value.
PR: https://github.com/llvm/llvm-project/pull/181252
Remove updateScalarResumePhis and create extracts for live-outs early in
addInitialSkeleton. Instead of extracting the from the header phi
recipes for the resume values (which is incorrect), extract the last
lane of the backedege value.
Then update optimizeInductionExitUsers to optimize both the scalar
resume values for IVs and IV exit values together.
This removes the need to pass state between transforms and addresses a
TODO.
PR: https://github.com/llvm/llvm-project/pull/174239
This extends the existing support to work with arbitrary interleave
factors. The main change here is reworking the ExtractLastActive
VPInstruction to take a variable amount of arguments and handling it in
unrollRecipeByUF and VPInstruction::generate.
The select condition for all mask/data values in a find-last recurrence
is the true if the mask for any part is true. Because of this the masks
for inactive parts will be updated to all-false when the parts with
active lanes are updated. This ensures the mask/data for last active
element always corresponds to the greatest part with an active lane.
This means finding the last element in the middle block simply requires
chaining the `extract.last.active` to forward the result from the last
active part through any inactive parts ahead of it.
Canonicalize GEP source element types from `%T` to `[sizeof(%T) x i8]`.
This is intended to flush out any remaining places that rely on GEP
element types, as part of the `ptradd` migration. The impact of this
change is expected to be fairly minimal (we might enable a few more
hoist/sink style folds that depend on equal GEP types).
I have a patch I want to post that improves blend masks, but it ends up
with a weird diff in this test stemming from the branch on false.
This replaces it with an external boolean. This should still test
scalarizing a blend which I believe is the original intent.
The implementation of getIntrinsicInstrCost in BasicTTIImpl
assumes that for some intrinsics if we're using custom
lowering for the equivalent DAG node that the cost needs to
be 2, instead of 1 for legal ops. However, even though we
use custom lowering for these scalable vector operations
when SVE is available, we still end up generating the same
efficient codegen as fixed-width. This patch deals with a
few obvious intrinsics that we know get lowered to something
sensible and return the same cost as NEON, i.e. 1.
Currently the logic for introducing a header mask and predicating the
vector loop region is done inside introduceMasksAndLinearize.
This splits the tail folding part out into an individual VPlan transform
so that VPlanPredicator.cpp doesn't need to worry about tail folding,
which seemed to be a temporary measure according to a comment in
VPlanTransforms.h.
To perform tail folding independently, this splits the "body" of the
vector loop region between the phis in the header and the branch + iv
increment in the latch:
Before:
```
+-------------------------------------------+
|%iv = ... |
|... |
|%iv.next = add %iv, vfxuf |
|branch-on-count %iv.next, vector-trip-count|
+-------------------------------------------+
```
After:
```
+-------------------------------------------+
|%iv = ... |
|%wide.iv = widen-canonical-iv ... |
|%header-mask = icmp ule %wide.iv, BTC |---+
|branch-on-cond %header-mask | |
+-------------------------------------------+ |
| |
v |
+-------------------------------------------+ |
|... | |
+-------------------------------------------+ |
| |
v |
+-------------------------------------------+ |
|%iv.next = add %iv, vfxuf |<--+
|branch-on-count %iv.next, vector-trip-count|
+-------------------------------------------+
```
Phis are then inserted in the latch for any value in the loop body that
have outside uses, with poison as their incoming value from the header
edge.
The motivation for this is to allow us to share the same "predicate all
successor blocks" type of predication we do for tail folding, but for
early-exit loops in #172454. This may also allow us to directly emit an
EVL based header mask, instead of having to match + transform the
existing header mask in addExplicitVectorLength.
This also allows us to eventually handle recurrences in the same
transform, avoiding the need to special case tail folding in
addReductionResultComputation.
Update groupChecks to always use DepCands to try and merge runtime
checks. DepCands contains the dependency partition, grouping together
all accessed pointers to he same underlying objects.
If we computed the dependencies, We only need to check accesses to the
same underlying object, if there is an unknown dependency for this
underlying object; otherwise we already proved that all accesses withing
the underlying object are safe w.r.t. vectorization and we only need to
check that accesses to the underlying object don't overlap with accesses
to other underlying objects.
To ensure runtime checks are generated for the case with unknown
dependencies, remove equivalence classes containing accesses involved in
unknown dependencies.
This reduces the number of runtime checks needed in case non-constant
dependence distances are found, and is in preparation for removing the
restriction that the accesses need to have the same stride which was
added in https://github.com/llvm/llvm-project/pull/88039.
PR: https://github.com/llvm/llvm-project/pull/91196
When a memory-reading or memory-writing instruction is not a
LoadInst/StoreInst, the dyn_cast to Ld/St returns nullptr, which is then
passed to recordAnalysis. This causes the optimization remark to fall
back to the loop header location instead of pointing at the actual
problematic instruction.
Pass &I (the actual Instruction) instead.
When vectorising loops containing calls to the llvm.pow intrinsic we
currently end up with direct calls to the vector math library equivalent
of llvm.pow. This causes problems later on because we then miss out on
optimisations in the backend for things like
pow(x, 0.25)
pow(x, 0.75)
etc.
There are a few different ways of trying to solve this, but I think the
cleanest way is to leave the call in the intrinsic form and then teach
the backend how to lower the FPOW DAG node to the vector math library
function. This is similar to what we already do for llvm.sincos/FSINCOS
today.
I've done this by removing the entries in Analysis/VecFuncs.def that map
the intrinsic to the vector math functions, whilst also teaching the
cost model that the vector intrinsic form of llvm.pow is cheap in the
presence of the library.
VPExpandSCEVRecipe may become unused after VPlan optimizations. This
patch removes VPExpandSCEVRecipes with no users before expansion in
expandSCEVs, avoiding generating dead code during VPlan execution.
Remove the use of function attributes no-nans-fp-math and
no-signed-zeros-fp-math in FP min/max reduction detection. The required
fast-math flags nnan and nsz should be present on the intrinsic calls,
fcmp and select instructions themselves.
Licm can now sink reverse VPInstructions outside the loop region; they
won't be considered when computing costs. Account for that in
planContainsAdditionalSimplifications.
Fixes https://github.com/llvm/llvm-project/issues/183592.
This removes the restriction requiring a single predicated early exit.
Using MaskedCond, we only combine early-exit conditions with block
masks from non-exiting control flow.
This means we have to ensure that we check the early exit conditions in
program order, to make sure we take the first exit in program order that
exits at the first lane for the combined exit condition.
To do so, sort the exits by their reverse post-order numbers.
Depends on https://github.com/llvm/llvm-project/pull/182395
PR: https://github.com/llvm/llvm-project/pull/182396
