This patch fixes the update of the DAGNode UnscheduledSucc counter when
a use edge is modified. This is the result of a setOperand() or a RAUW
(and friends) operation.
Before this patch we would not check if the User (i.e., the consumer of
the use-def edge) is scheduled and we would update the definition's
UnscheduledSucc counter, resulting in counting errors.
For example, consider the following IR:
```
%A = ...
%B = ...
%U = %A ; scheduled
```
Note that %U's DAGNode is marked as "scheduled" while %A and %B are not.
If we change %U's operand from %A to %B then we should not attempt to
update %A's or %B's UnscheduledSuccs because %U is scheduled so it
should not get counted as an "unscheduled" successor.
Need to update matching between the original reduced values and their
vectorized matches after ordered reduction vectorization to avoid
a compiler crash
Patch models ordered reductions as a series of extractelements for the
cases which cannot be modeled as unordered reductions.
Fixes#50590
Reviewers: RKSimon, hiraditya
Pull Request: https://github.com/llvm/llvm-project/pull/182644
If the instructions state is alternate and/or contains non-directly
matching instructions, need to check if it is better to represent such
operations as non-alternate with copyables.
To do this, we need to compare operands between the instructions in their
different representations and choose the best one for optimal
vectorization.
Reviewers: RKSimon, hiraditya
Pull Request: https://github.com/llvm/llvm-project/pull/183777
When narrowing interleave groups, the main vector loop processes IC
iterations instead of VF * IC. Update selectEpilogueVectorizationFactor
to use the effective VF, checking if the canonical IV controlling the
loop now steps by UF instead of VFxUF.
This avoids epilogue vectorization with dead epilogue vector loops and
also prevents crashes in cases where we can prove both the epilogue and
scalar loop are dead.
Fixes https://github.com/llvm/llvm-project/issues/186846
PR: https://github.com/llvm/llvm-project/pull/187016
Simplify exactly as InstCombine does. A follow-up would include
simplifying add x, (sub 0, y) -> sub x, y.
Alive2 proof: https://alive2.llvm.org/ce/z/Af7QiD
This updates `matchExtendedReductionOperand` so the simple case of
`UpdateR(PrevValue, ext(...))` is matched first as an early exit. The
binop matching is then flattened to remove the extra layer of the
`MatchExtends` lambda.
I was very puzzled the other day when it showed that VF 8 had a cost of
X and VF 16 had a cost of X/2, yet it still choose VF 8. This PR adds
some extra debug output to explain why this happens.
Instead of checking dereferenceability early during
LoopVectorizationLegality, defer the check to VPlan construction via
areAllLoadsDereferenceable.
This in preparation for supporting early exit vectorization of
non-dereferencable loads, e.g. via speculative loads
(https://discourse.llvm.org/t/rfc-provide-intrinsics-for-speculative-loads/89692)
or first-faulting loads. Detection in VPlan allows easily replacing
potentially non-deref loads with other loads as needed.
PR: https://github.com/llvm/llvm-project/pull/185323
shl-based reduced values in many cases serve as a bitcast/bswap-based
transfromation root, but need to improve analysis for better matching.
This patch merges reduction candidates into a single reduced value
array, if there are only 2 different candidate arrays, one of them has
only single element, the second is a list of shl instructions. Also,
sorts these shl instructions by their shift amount and merges with the
single candidate, if it is profitable to have a copyable reduction.
This patch implements the unconditional and conditional branch
instructions mirroring the newly added LLVM IR instructions.
So now we have two new classes UncondBrInst and CondBrInst inheriting
from BranchInst.
The original Br opcode has been removed in favor of UncondBr and CondBr.
When two GEPs A & B with variable indices have source element types of
different sizes (e.g. i8 vs i32), their offsets cannot be constants.
Currently the `getConstantOffsetComplexAddrs` does not check this and
simply use the stride of GEP A assuming they are equal, resulting in
miscompile.
Add a check that the source element types are the same, otherwise bail
out.
Fixes#186327
The original support for copyables leads to a regression in x264 in
RISCV, this patch improves detection of the copyable candidates by more
precise checking of the profitability and adds and extra check for
splitnode reduction, if it is profitable.
Fixes#184313
Reviewers: hiraditya, RKSimon
Pull Request: https://github.com/llvm/llvm-project/pull/185697
Recursively splitting out some work from #183318; this covers
the enums for early exit loop type (none, readonly, readwrite)
and the style used (just readonly and
masked-handle-ee-in-scalar-tail for now) and refactoring for
basic use of those enums.
This reverts commit 91b928f919364b29e241821fc639b9ef56dab1a5.
This complicates some analysis that need the happen on the scalar VPlan,
before regions have been created, e.g.
https://github.com/llvm/llvm-project/pull/185323/.
If current buildvector node is part of the combined nodes of the
matching candidate node, this matching candidate must be considered as
non-matching to prevent wrong def-use chain
Reviewers:
Pull Request: https://github.com/llvm/llvm-project/pull/187491
When matching scalar steps of the canonical IV, also match a derived IV
of the canonical IV if the derivation is essentially a no-op. Fixes a
failure in the mve-reg-pressure-spills.ll test when expensive checks are
enabled.
This patch implements the auxiliary pass argument, a secondary argument
that can be passed to Sandbox IR passes.
It also updates the SeedCollection pass to accept the auxiliary argument
"enable-diff-types", which enables collection of seeds of different
types. This will be used in a follow-up patch.
This patch replace the remaining LogicalAnd to vp.merge in the second
pass to not break the `m_RemoveMask` pattern in the optimizeMaskToEVL.
Also skip cost model comparison when the plan contains `vp_merge` which
won't be calculated by the legacy model.
This can help to remove header mask for FindLast reduction (CSA) loops.
Original PR: https://github.com/llvm/llvm-project/pull/184068
Original built-bot failure:
https://lab.llvm.org/buildbot/#/builders/213/builds/2497
Currently, SLP vectorizer do not care about loops and their trip count.
It may lead to inefficient vectorization in some cases. Patch adds loop
nest-aware tree building and cost estimation.
When it comes to tree building, it now checks that tree do not span
across different loop nests. The nodes from other loop nests are
immediate buildvector nodes.
The cost model adds the knowledge about loop trip count. If it is
unknown, the default value is used, controlled by the
-slp-cost-loop-min-trip-count=<value> option. The cost of the vector
nodes in the loop is multiplied by the number of iteration (trip count),
because each vector node will be executed the trip count number of
times. This allows better cost estimation.
Original Reviewers:
jdenny-ornl, vporpo, hiraditya, RKSimon
Original PR: https://github.com/llvm/llvm-project/pull/150450
Recommit after revert in c7bd3062f1dac975cf9b706f457b3c55b4bf57ff and in 4e500bd0015042b0cd4b7c87b81caeea06072d24
Reviewers:
Pull Request: https://github.com/llvm/llvm-project/pull/187391
masked_cond is used to combine early-exit conditions with masks from
predicate. The early-exit condition should only be evaluated if the mask
is true. Emit the mask first, to avoid incorrect poison propagation.
Fixes https://github.com/llvm/llvm-project/issues/187061.
Now that we can vectorize loops with multiple early exits, we emit
dispatch blocks after the middle block to go to a specific exit or
continue in the dispatch chain.
With that, we need to be a bit more careful when it comes to picking the
loop the dispatch block belongs to. The dispatch block will belong to
the innermost loop of all exit blocks reachable from the current block.
Fixes https://github.com/llvm/llvm-project/issues/185362
PR: https://github.com/llvm/llvm-project/pull/185618
Update VPReplicateRecipe::computeCost to compute the cost for stores to
invariant addresses only masked by the header mask.
This matches the legacy cost model logic, but it is slightly odd that
the legacy cost model only seems to do this for stores predicated by the
header mask (i.e. tail-folding and not executed conditionally
otherwise). This is probably something we want to re-evaluate
eventually.
PR: https://github.com/llvm/llvm-project/pull/181572
Currently when considering register pressure is enabled, we reject any
VF that has higher pressure than the number of registers. However this
can result in failing to vectorize in cases where it's beneficial, as
the cost of the extra spills is less than the benefit we get from
vectorizing.
Deal with this by instead calculating the cost of spills and adding that
to the rest of the cost, so we can detect this kind of situation and
still vectorize while avoiding vectorizing in cases where the extra cost
makes it not with it.
The function instructionsWithoutDebug serves two uses: skipping debug
intrinsics and skipping pseudo instructions. Nonetheless, these
functions are expensive due to out-of-line filtering using
std::function. Ideally, the filter should be inlined, but that would
require including IntrinsicInst.h in BasicBlock.h.
We no longer use debug intrinsics, so the first use (parameter false) is
no longer needed. The second use is sometimes needed, but the
distinction between PseudoProbe instructions can be made at the call
sites more easily in many cases.
Therefore, remove instructionsWithoutDebug/sizeWithoutDebug.
c-t-t stage2-O3 -0.21%.
Fixes#186005
On RV32 with zve32x, i.e. no legal 64 bit types either scalar or vector,
@llvm.cttz.elts.i64 cannot be lowered and so returns an illegal cost for
scalable VFs. However VPInstruction::FirstActiveLane and
VPInstruction::LastActiveLane always use a hardcoded i64 type.
This causes a legacy/VPlan cost model mismatch in the live-out.ll test,
and in early-exit-live-out.ll prevents the scalable VF from being
chosen.
This PR teaches the two VPInstructions to use the target's index type,
i.e. the width of a pointer in the default address space, so it will
generate a 32 bit cttz.elts on RV32. This should be large enough to hold
the maximum number of elements in a vector, as if the vector was any
bigger it would imply it isn't accessible by memory.
I considered using the canonical IV type but I don't think that will
work since the canonical IV can be i64 on RV32, and it causes
regressions due to extra zexting on 64-bit targets with a 32-bit IV.
Currently, SLP vectorizer do not care about loops and their trip count.
It may lead to inefficient vectorization in some cases. Patch adds loop
nest-aware tree building and cost estimation.
When it comes to tree building, it now checks that tree do not span
across different loop nests. The nodes from other loop nests are
immediate buildvector nodes.
The cost model adds the knowledge about loop trip count. If it is
unknown, the default value is used, controlled by the
-slp-cost-loop-min-trip-count=<value> option. The cost of the vector
nodes in the loop is multiplied by the number of iteration (trip count),
because each vector node will be executed the trip count number of
times. This allows better cost estimation.
Reviewers: jdenny-ornl, vporpo, hiraditya, RKSimon
Pull Request: https://github.com/llvm/llvm-project/pull/150450
Recommit after revert in c7bd3062f1dac975cf9b706f457b3c55b4bf57ff
This patch replace the remaining LogicalAnd to vp.merge in the second
pass to not break the `m_RemoveMask` pattern in the optimizeMaskToEVL.
This can help to remove header mask for FindLast reduction (CSA) loops.
PR: https://github.com/llvm/llvm-project/pull/184068
This patch disables vectorizing Cmps with different operand types
because we can't form a legal vector.
This used to cause an assertion check crash once we attempted to pack
the bundle formed by Cmp's operands.
I've added the Instruction::Select opcode to the existing list of
opcodes that call getCostForRecipeWithOpcode. There are currently 5
tests that ask for the cost of the select:
Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll
Transforms/LoopVectorize/first-order-recurrence-with-uniform-ops.ll
Transforms/LoopVectorize/narrow-to-single-scalar.ll
Transforms/LoopVectorize/replicate_fneg.ll
Transforms/LoopVectorize/single-scalar-cast-minbw.ll
The fact they all pass with this change is hopefully proof enough that
the costs are correct.
Since 1b29ac1d1857ea42273fc7862ea019a74a55195d, regions are constructed
as part of the scalar transforms; moving header phi creation after
region creation slightly simplifies the code.
This patch removes the extra logical-and in `x && (x && y)` and `x && (y && x)` to `x && y`.
This helps to simplify mask calculation in the FindLast reduction and
exposes more opportunities to replace to EVL.
PR link: https://github.com/llvm/llvm-project/pull/185806
This patch tries to drastically simplify resume value handling for the
scalar loop when vectorizing the epilogue.
It uses a simpler, uniform approach for updating all resume values in
the scalar loop:
1. Create ResumeForEpilogue recipes for all scalar resume phis in the
main loop (the epilogue plan will have exactly the same scalar resume
phis, in exactly the same order)
2. Update ::execute for ResumeForEpilogue to set the underlying value
when executing. This is not super clean, but allows easy lookup of the
generated IR value when we update the resume phis in the epilogue. Once
we connect the 2 plans together explicitly, this can be removed.
3. Use the list of ResumeForEpilogue VPInstructions from the main loop
to update the resume/bypass values from the epilogue.
This simplifies the code quite a bit, makes it more robust (should fix
https://github.com/llvm/llvm-project/issues/179407) and also fixes a
mis-compile in the existing tests (see change in
llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub-epilogue-vec.ll,
where previously we would incorrectly resume using the start value when
the epilogue iteration check failed)
In some cases, we get simpler code, due to additional CSE, in some cases
the induction end value computations get moved from the epilogue
iteration check to the vector preheader. We could try to sink the
instructions as cleanup, but it is probably not worth the trouble.
Fixes https://github.com/llvm/llvm-project/issues/179407.
Fix the checks for the non-power-of-2 base bswaps by checking the
power-of-2 of the source type, not the target scalar type. Plus, add
cost estimation for zext, if the source type does not match the scalar type and fixes final bitcasting for the reduced values.
Fixes https://github.com/llvm/llvm-project/pull/184018#issuecomment-4053477562
