Add an alternative to test VPlan in more isolation via a new
`vplan-test-transform` option, which builds VPlan0 for each loop in the
input IR and then can invoke a set of transforms on it.
In order to allow different recipe types to be created, a new
widen-from-metadata transform is added, which transforms VPInstructions
to different recipes, based on custom !vplan.widen metadata. Currently
this supports creating widen & replicate recipes, but can easily be
extended in the future.
Currently the handling is intentionally bare-bones, to be extended
gradually as needed.
PR: https://github.com/llvm/llvm-project/pull/178522
When canonicalizing GEPs to have a single index, only ignore the first
leading zero index. This fixes a hole in the canonicalization in case
there was a struct zero offset, which can't be removed by the the
leading zero stripping.
This reverts commit f8f6965ceece9e330ddb66db5f402ecfb5e3ad34.
This is causing infinte loops interacting with other transforms. See
discussion on #182647 .
This enables vectorization of epilogue loops produced by LoopVectorizer on
SystemZ.
LoopVectorizationCostModel::isEpilogueVectorizationProfitable() and
TTI.preferEpilogueVectorization() have been refactored slightly so that
targets can override preferEpilogueVectorization(ElementCount Iters) and
directly control this, whereas before this depended on
TTI.getMaxInterleaveFactor() as well.
The Iters passed to preferEpilogueVectorization() reflects the total number
of scalar iterations performed in the vectorized loop (including interleaving).
The default implementation of preferEpilogueVectorization() now subsumes
the old check against getMaxInterleaveFactor(). This patch should be NFC for
other targets.
For FindLast reduction selecting an IV, we can avoid the horizontal
AnyOf in the vector loop, by introducing an independent boolean
reduction to track if the condition was ever true in the loop. If it was
never true in the loop, we select the start value, otherwise the select
the min/max of the FindIV reduction, as required by the predicate.
The main advantage of this approach is that we have 2 independent
reductions, that do not require a horizontal AnyOf reduction in the
loop.
Currently this requires a non-wrapping IV, but this can be relaxed in
the future by selecting a canonical IV, which is then transformed to the
specific derived IV for the reduction after the loop.
Depends on https://github.com/llvm/llvm-project/pull/177870.
PR: https://github.com/llvm/llvm-project/pull/172569
#149042 added last-active-lane and removed the restriction that we
couldn't tail fold loops that had outside users (in AllowedExit).
However we still have a restriction that IVs can't have outside users.
This was added separately to the AllowedExit restriction in #81609, but
it looks like #149042 didn't remove it.
AFAICT we currently extract the correct lane for IVs, so this PR relaxes
the restriction. This helps a good few loops get tail folded in
llvm-test-suite.
-force-tail-folding-style=none was added to pr5881-scev-expansion.ll to
preserve the original scev expansion, since otherwise we end up with a
cttz.elts(false, false, true, true) that blocks SCEV analysis. We should
probably teach ConstantFolding to fold it.
In order to be able to create selects for reduction phis through tail
folding in foldTailByMasking (#176143), make VPReductionPHIRecipe an
instance of VPIRFlags and plumb the FMFs from the original RdxDesc.
This allows us to remove more uses of the RecurrenceDescriptor in
addReductionResultComputation, which should help untie it from
LoopVectorizationLegality.
Extend handleMultiUseReductions to support strict predicates (>, <),
matching the first index instead of the last for non-strict predicates.
Builds on top of https://github.com/llvm/llvm-project/pull/141431.
FindLast reductions with strict predicates are adjusted to compute the
correct result as follows:
1. Find the first canonical indices corresponding to partial min/max
values, using loop reductions.
2. Find which of the partial min/max values are equal to the overall
min/max value.
3. Select among the canonical indices those corresponding to the overall
min/max value.
4. Find the first canonical index of overall min/max and scale it back to
the original IV using VPDerivedIVRecipe.
5. If the overall min/max equals the starting min/max, the condition in
the loop was always false, due to being strict; return the original start
value in that case.
When the vectorized epilogue loop uses partial reductions, the PHI node
in the loop must start at 0 (because for partial sub-reductions the
sub is done in the middle block) and the compute-reduction-result must
subtract from the partial result (as calculated in the middle block of
the main vector loop), instead of subtracting from the original init
value.
This fixes the issue as reported on #178919 by @aeubanks.
This is groundwork for #151300, which aims to support first-faulting
loads in non-tail-folded early-exit loops.
Per #175900, we need a variable-length stepping transform that can
shared between EVL and non-EVL loops.
The idea is to have an EVL-independent counter and transform for
tracking the cumulative number of processed elements.
This patch renames the existing counter (VPEVLBasedIVPHIRecipe) and
transform (canonicalizeEVLLoops) to be EVL-independent:
- Rename VPEVLBasedIVPHIRecipe to VPCurrentIterationRecipe to
reflect its general purpose of tracking processed element count.
- Rename canonicalizeEVLLoops to convertToVariableLengthStep.
This is NFC.
Right now memory dependencies checks and speculation for unit-strideness
are performed somewhat simultaneously. This is wrong because:
* Ideally, if accesses aren't unit-strided in runtime we might want to
take a version with gather/strided load (longer term). Those two loops
should share legality checks and the dispatch based on stride should
only happen after the legality condition has been satisfied.
* Even if we don't generate multiple vector loops (current situation),
not vectorizing at all is worse than generating gather-only vector loop.
This PR adds a test for the latter as that could be a first step in
adding full support for the former.
This isn't target-specific, but gathers aren't supported in generic
target and result in very ugly scalarized code/CHECKs, hence put the
test under RISCV/.
Co-authored-by: Florian Hahn <flo@fhahn.com>
Addresses review comments from
https://github.com/llvm/llvm-project/pull/180898#pullrequestreview-3791945590.
We don't need to recursively collect direct users of the header mask, we
can do that as a separate step so that the main worklist loop only
handles potentially reassociatable candidates.
Also add back mention of tail folding to comment and a TODO.
After 6f253e87dd, VFxUF may have been replaced by UF, in which case the
simplification is no longer correct. Tighten check to make sure the
increment is still what we expect.
Fixes a miscompile in the added test case.
The legacy cost model skips gather/scatters when determining the
predicated stores in useEmulatedMaskMemRefHack. Match the behavior in
the VPlan-based implementation. This fixes a cost divergence in the
attached test.
Directly unroll VectorEndPointerRecipe following 0636225b ([VPlan]
Directly unroll VectorPointerRecipe, #168886). It allows us to leverage
existing VPlan simplifications to optimize.
Co-authored-by: Luke Lau <luke@igalia.com>
Co-authored-by: Florian Hahn <flo@fhahn.com>
This reverts commit 8d29d09309654541fb2861524276ada6a3ebf84c.
The underlying issue causing the revert has been fixed independently
as 301fa24671256734df6b7ee65f23ad885400108e.
Original message:
Move narrowInterleaveGroups to to general VPlan optimization stage.
To do so, narrowInterleaveGroups now has to find a suitable VF where all
interleave groups are consecutive and saturate the full vector width.
If such a VF is found, the original VPlan is split into 2:
a) a new clone which contains all VFs of Plan, except VFToOptimize, and
b) the original Plan with VFToOptimize as single VF.
The original Plan is then optimized. If a new copy for the other VFs has
been created, it is returned and the caller has to add it to the list of
candidate plans.
Together with https://github.com/llvm/llvm-project/pull/149702, this
allows to take the narrowed interleave groups into account when
computing costs to choose the best VF and interleave count.
One example where we currently miss interleaving/unrolling when
narrowing interleave groups is https://godbolt.org/z/Yz77zbacz
PR: https://github.com/llvm/llvm-project/pull/149706
Compute the number of predicated stores directly in VPlan instead of
using CM.useEmulatedMaskMemRefHack(), which will only account for the
number of predicated stores for the last VF the legacy cost model
considered.
Fixes https://github.com/llvm/llvm-project/issues/181183
Building on top of the recent changes to introduce BranchOnTwoConds,
this patch adds support for vectorizing loops with multiple early exits,
all dominating a countable latch. The early exits must form a
dominance chain, so we can simply check which early exit has been taken
in dominance order.
Currently LoopVectorizationLegality ensures that all exits other than
the latch must be uncountable. handleUncountableEarlyExits now collects
those uncountable exits and processes each exit.
In the vector region, we compute if any exit has been taken, by taking
the OR of all early exit conditions (EarlyExitConds) and checking if
there's
any active lane.
If the early exit is taken, we exit the loop and compute which early
exit
has been taken. The first taken early exit is the one where its exit
condition is true in the first active lane of EarlyExitConds.
We create a chain of dispatch blocks outside the loop to check this for
the early exit blocks ordered by dominance.
Depends on https://github.com/llvm/llvm-project/pull/174016.
PR: https://github.com/llvm/llvm-project/pull/174864
narrowToSingleScalarRecipes' operands check is a bit too restrictive by
permitting a single user. Factor out and reuse the existing
introduces-broadcast logic to improve results.
In some cases, LV gets simplifyable IR as input. Directly apply
simplifications on the initial VPlan0 to avoid vectorization in cases
where the loop body can be folded away.
Using the end-to-end pipeline, this is relatively rare, but when
reducing test cases, the reduction often ends up with cases with trivial
folds. Rejecting those will result in more robust & realistic test
cases.
As follow-up, I also plan to add initial dead recipe removal.
Depends on https://github.com/llvm/llvm-project/pull/176795.
PR: https://github.com/llvm/llvm-project/pull/176828
Recipes can be sunk now. In those cases, the sunk recipes are outside
the loop region, but may not be uniform across VF and UF.
Update the code to only exit early if the recipe is defined before the
region. Without DT available, the easiest way to check is just if it is
in the entry/preheader block.
Fixes https://github.com/llvm/llvm-project/issues/181002.
Fixes https://github.com/llvm/llvm-project/issues/180781.
Switch tests from using `-debug[-only=LoopVectorize]` to
`-vplan-print-after` as that provides better control at what step in the
pipeline we want to check the VPlan (I'm using `optimize$` for now to
preserve previous state).
Then, update `-vplan-print-after*` to print what function the loop
belongs to. That enables us to simplify VPlan UTC support as the output
of the updated tests contains the VPlan dump only - no special
filtering/extraction is necessary anymore.
Add a new helper function `matchEquivZeroRHS()` that recognizes
comparisons with constants that are equivalent to comparisons with zero,
and transforms the predicate accordingly.
This handles the following transformations:
- icmp sgt X, -1 --> icmp sge X, 0
- icmp sle X, -1 --> icmp slt X, 0
- icmp [us]ge X, 1 --> icmp [us]gt X, 0
- icmp [us]lt X, 1 --> icmp [us]le X, 0
This enables more optimization opportunities in `simplifyICmpWithZero`,
such as folding icmp sgt X, -1 when X is known to be non-negative.
---
- IR Impact: https://github.com/dtcxzyw/llvm-opt-benchmark/pull/3414
We reassociate ((x && y) && z) -> (x && (y && z)) if x has more than
use, in order to allow simplifying the header mask further. However this
is somewhat unreliable as there are times when it doesn't have more than
one use, e.g. see the case we run into in
https://github.com/llvm/llvm-project/pull/173265/changes#r2769759907.
This moves it into a separate transformation that always reassociates
the header mask regardless of the number of uses, which prevents some
fragile test changes in #173265.
We need to run it before both calls to simplifyRecipes in optimize. I
considered putting it in simplifyRecipes itself but simplifyRecipes is
also called after unrolling and when the loop region is dissolved which
causes vputils::findHeaderMask to assert.
There isn't really any benefit to reassociating masks that aren't the
header mask so the existing simplification was removed.
Add a new VPlan subdirectory as common place for tests checking VPlan
printing. It contains a lit.local.cfg that only runs the tests when
assertions are enabled.
This removes the need to add explicit REQUIRES: asserts to VPlan tests.
PR: https://github.com/llvm/llvm-project/pull/180611
The legacy cost model tries to scalarize loads that are used as
pointers. Skip if the load would need predicating when scalarized,
because that would incur very high costs, see useEmulatedMaskMemRefHack.
Fixes https://github.com/llvm/llvm-project/issues/180780.
The checks created by LAA only compute a pointer difference and do not
need to capture provenance. Use SCEVPtrToAddr instead of SCEVPtrToInt
for computations.
To avoid regressions while parts of SCEV are migrated to use PtrToAddr
this adds logic to rewrite all PtrToInt to PtrToAddr if possible in the
created expressions. This is needed to avoid regressions.
Similarly, if in the original IR we have a PtrToInt, SCEVExpander tries
to re-use it if possible when expanding PtrToAddr.
Depends on https://github.com/llvm/llvm-project/pull/178727.
Fixes https://github.com/llvm/llvm-project/issues/156978.
PR: https://github.com/llvm/llvm-project/pull/178861
Check if costs for partial reductions are valid up-front in
getScaledReductions instead when transforming each link in the chain in
transformToPartialReduction. This ensures that we either transform all
entries in the chain together, or none via the existing invalidation
logic.
This fixes a crash when a link in the chain would have invalid cost, as
in the added test cases.
Fixes https://github.com/llvm/llvm-project/issues/180340.
PR: https://github.com/llvm/llvm-project/pull/180438
This fixes a miscompile in #180005 where we didn't check that the first
incoming value isn't poison.
We should use the first non-poison incoming value if it exists, or just
poison if all the incoming values are poison.
Sub-reductions can be implemented in two ways:
(1) negate the operand in the vector loop (the default way).
(2) subtract the reduced value from the init value in the middle block.
Note that both ways keep the reduction itself as an 'add' reduction,
which is necessary because only llvm.vector.partial.reduce.add exists.
The ISD nodes for partial reductions don't support folding the
sub/negation into its operands because the following is not a valid
transformation:
```
sub(0, mul(ext(a), ext(b)))
-> mul(ext(a), ext(sub(0, b)))
```
It can therefore be better to choose option (2) such that the partial
reduction is always positive (starting at '0') and to do a final
subtract in the middle block.
For AArch64 there are no dot-product instructions that can
do a `partial.reduce.sub(acc, mul(ext(a), ext(b)))` operation.
I'm not sure if such instructions exist for other targets.
(If so then we may want to make this decision a target option)
This PR also increases the AArch64 cost of a partial sub-reduction
when this exists in an 'add-sub' reduction chain.
Fixes https://github.com/llvm/llvm-project/issues/178703
This patch extends the support added in #158088 to loops where the
assignment is non-speculatable (e.g. a conditional load or divide).
For example, the following loop can now be vectorized:
```
int simple_csa_int_load(
int* a, int* b, int default_val, int N, int threshold)
{
int result = default_val;
for (int i = 0; i < N; ++i)
if (a[i] > threshold)
result = b[i];
return result;
}
```
It does this by extending the recurrence matching from only looking for
selects, to include phis where all operands are the header phi, except
for one which can be an arbitrary value outside the recurrence.
---
Reverts llvm/llvm-project#180275 (original PR: #178862)
Additional type legalization for `ISD::VECTOR_FIND_LAST_ACTIVE` was
added in #180290, which should resolve the backend crashes on x86.
Start trying to use SimplifyDemandedFPClass on instructions, starting
with fmul. This subsumes the old transform on multiply of 0. The
main change is the introduction of nnan/ninf. I do not think anywhere
was systematically trying to introduce fast math flags before, though
a few odd transforms would set them.
Previously we only called SimplifyDemandedFPClass on function returns
with nofpclass annotations. Start following the pattern of
SimplifyDemandedBits, where this will be called from relevant root
instructions.
I was wondering if this should go into InstCombineAggressive, but that
apparently does not make use of InstCombineInternal's worklist.
