ConstantRange::getNonEmpty was excluding MAX and MAX - 1 in FindFirstIV
vectorization, and this was discovered in an i1 miscompile, where it
returns the full range: fix it to exclude MAX only. The change has also
necessitated fixing a test that's not supposed to be vectorized.
Fixes#173459.
Co-authored-by: Nikita Popov <npopov@redhat.com>
i16 -> i32 dot-product operations are natively supported with SVE2p1 and
SME2. This updates the cost-model so that those operations are
recognized as cheap by the LoopVectorizer.
Fixes#175028
We have a VPlanVerifier assertion that a VPInstruction that uses EVL
only has one use. This used to hold until we implemented CSE, but now we
can run into the case where e.g. a multiply from an expanded
VPWidenPointerInductionRecipe gets cse'd, causing it to have multiple
uses:
EMIT ir<%0> = WIDEN-POINTER-INDUCTION ir<%.pre3>, ir<6>, vp<%5>
EMIT ir<%1> = WIDEN-POINTER-INDUCTION ir<%.pre>, ir<6>, vp<%5>
EMIT-SCALAR vp<%5> = EXPLICIT-VECTOR-LENGTH vp<%avl>
-->
EMIT-SCALAR vp<%10> = EXPLICIT-VECTOR-LENGTH vp<%avl>
EMIT vp<%11> = mul ir<6>, vp<%10>
EMIT vp<%ptr.ind> = ptradd vp<%pointer.phi>, vp<%11>
EMIT vp<%12> = mul ir<6>, vp<%10>
EMIT vp<%ptr.ind>.1 = ptradd vp<%pointer.phi>.1, vp<%12>
-->
EMIT-SCALAR vp<%5> = EXPLICIT-VECTOR-LENGTH vp<%avl>
EMIT vp<%6> = mul ir<6>, vp<%5>
EMIT vp<%ptr.ind> = ptradd vp<%pointer.phi>, vp<%6>
EMIT vp<%ptr.ind>.1 = ptradd vp<%pointer.phi>.1, vp<%6>
This removes the check, as I'm not sure it's that useful anymore now
that we have CSE. Coincidentally, this crash only happened on RV32
because RV64 requires zexting the EVL, which sidesteps a lot of the
checks to begin with.
We've already optimised these, so update the cost model to reflect it.
And skip the isBeforeLegalize check when lowering i8 muls, because it
then misses the cases where, say v32i8, has been type legalised into 2x
v16i8.
Also explicitly disable memory interleaving for any factor other than
two or four.
This patch simplifies extract-lane(%lane_num, %X) to %X when %X is a
scalar value. Extracting from a scalar is redundant since there is only
one value to extract.
SCEV has a manual fold when doing SCEV construction from IR, that is not
integrated in the regular SCEV construction functions. Mirror the
behavior in getSCEVExprForVPValue, to match results when constructing
SCEVs from IR.
Fixes https://github.com/llvm/llvm-project/issues/174622.
Skip live-ins in findRecipe to prevent a crash for cases with degenerate
reductions (where the backedge value is a live-in). Such reductions
should be removed, but this requires further changes.
Fixes https://github.com/llvm/llvm-project/issues/175229.
VPScalarIVStepsRecipe relies on APInt truncation in order to vectorize
blocks with a width greater than the maximum value the types of some of
their (changing) operands are able to hold (e.g., an i1 input with a
vector width of 4). Simply reenable implicit truncation in
ConstantInt::get() to cover this case.
Remove the helper function given it is only called in one place to
prevent accidentally using it elsewhere where we probably do not want
implicit truncation turned on.
This fixes another case that we saw after
acb78bde6fb613a9af2a604bc69fa744a8cee850 did not fix that issue, which
had the same stack trace. We still want to keep lane constants as
unsigned.
Somewhat similar to 6d1e7d4982fabc9e245897056a5425496df6a7a3.
This test case comes from a tensorflow/XLA compilation from a test case
in https://github.com/google-research/spherical-cnn.
a83c89495ba6fe0134dcaa02372c320cc7ff0dbf caused assertion failures here
as if we have a single bit induction variable and two lanes (0 and 1),
then the second lane index (1) will be out of bounds of what a signed
1-bit integer can hold. Lane indices are always >0 according to
VPlanHelpers.h:125, and the lane representation in this code is also
unsigned.
The test case come from tensorflow/XLA.
Following the improvements introduced in #109833 and the most recent
development of the libamath library (used by `-fveclib=ArmPL`), this
patch adds the missing mappings for the functions that return literal
struct values.
In VPlanPatternMatch.h I have changed the int_pred_ty code to look
through broadcasts in order to catch more cases, i.e. multiplying by a
splat of one, etc.
Conservatively predicate sdiv/srem:
- RHS may carry poison in masked‑off lanes.
- RHS could be −1 while LHS has masked‑off lanes (risking INT_MIN/−1
overflow).
We’ll relax this once we can prove non‑wrap/non‑poison conditions.
Fixes#170775.
The original patch, landed as a2db31b0 ([VPlan] Simplify pow-of-2
(mul|udiv) -> (shl|lshr), #172477) had a critical commutative matcher
bug, which has now been fixed. An assert has also been strengthened,
following a post-commit review.
Follow-up to https://github.com/llvm/llvm-project/pull/171204 and
1f331e453f to only rely on isAddressSCEVForCost in legacy isAddressSCEVForCost,
completely aligning the decisions of VPlan and legacy cost model.
Restore the all-operands-invariant handling in WidenGEP::execute prior
to 37f7b31 (Reland [VPlan] Handle WidenGEP in narrowToSingleScalars), as
crashes have been reported.
Fixes#173761.
The stride can be negative here, so we should use getSigned().
This avoids an assertion failure with
https://github.com/llvm/llvm-project/pull/171456. It also avoids a
miscompile if the index is >64-bit, but I don't think that can happen in
practice.
This PR implements the first change outlined in
https://discourse.llvm.org/t/rfc-allow-non-constant-offsets-in-llvm-vector-splice/88974?u=lukel
In order to allow non-immediate offsets in the llvm.vector.splice
intrinsic, we need to separate out the "shift left" and "shift right"
modes into two separate intrinsics, which were previously determined by
whether or not the offset is positive or negative.
The description in the LangRef has also been reworded in terms of
sliding elements left or right and extracting either the upper or lower
half as opposed to extracting from a certain index, which brings it
inline with the definition of `llvm.fshr.*`/`llvm.fshl.*`.
This patch teaches AutoUpgrade.cpp to upgrade the old intrinsics into
their new equivalent one based on their offset, so existing uses of
vector.splice should still work.
Uses of llvm.vector.splice in `llvm/test/CodeGen` haven't been replaced
in this PR to keep the diff small and kick the tyres on the AutoUpgrader
a bit. I planned to do this in a follow up NFC but can include it in
this PR if reviewers prefer.
Similarly the shuffle costing kind `SK_Splice` has just been kept the
same for now, to be split into `SK_SpliceLeft` and `SK_SpliceRight`
later.
All extra state has been removed from VPWidenSelectRecipe at this point.
There's no benefit of having a separate recipe and Select can easily be
handled by the existing VPWidenRecipe.
PR: https://github.com/llvm/llvm-project/pull/174234
Reland #171963, #172639 and #173444, they are reverted in
86b9f90b9574b3a7d15d28a91f6316459dcfa046 because of introducing
non-determinism in compiles.
The non-determinism has been fixed in
9b8addffa70cee5b2acc5454712d9cf78ce45710.
The default value for MaxInterleaveFactor is 2, but some CPUs prefer a
wider factor of 4. This adds a subtarget feature so that cpus can
override the default in their tuning features, keeping more of the
options together in one place.
Also handle missing PtrToAddrs and AddrSpaceCast in
getCostForRecipeWithOpcode.
This makes sure all cast opcodes are handled, fixing a crash on loops
replicating addrspacecast and ptrtoaddrs.
getLiveIns returns an iterator to members of a dense map. The loop may
create new live-ins, which can trigger re-allocation of the underlying
dense map, causing use-after-free accesses for the iterator.
Make sure we iterate over a copy of the live-ins to avoid
use-after-free.
Fixes https://github.com/llvm/llvm-project/issues/173222.
Currently we need to precompute costs for exit conditions, to match the
legacy cost, as they will get replaced by a compare against the
canonical IV (or others, like active-lane-mask or EVL based) and the
original compare will get removed.
This is not true for instructions with users other than the exit
condition. Those will remain, and we can just use the VPlan-based cost
model to get more accurate results.
This improves results in some cases, like
@test_value_in_exit_compare_chain_used_outside because the IV increment
user outside the loop is replaced by computing the final value outside
the loop.
It also fixes a crash introduced by f196b1d66ff (#146525).
PR: https://github.com/llvm/llvm-project/pull/174029
This fixes a crash after introducing BranchOnTwoConds (524b1788,
https://github.com/llvm/llvm-project/pull/172750) when trying to
replace BranchOnTwoConds with a VPBranchOnCond, without dissolving the
region.
In that case, we need to update the appropriate condition operand.
This causes non-determinism in compiles.
From nikic: "FYI the non-determinism is also visible on
llvm-opt-benchmark. Maybe repeatedly running test cases from
299446d99f
could reproduce the issue..."
Also revert dependent 796fafeff92fe5d2d20594859e92607116e30a16 and
e135447bda617125688b71d33480d131d1076a72.
This PR introduces a new BranchOnTwoConds VPInstruction, that takes 2
boolean operands and must be placed in a block with 3 successors.
If condition I is true, branches to successor I, otherwise falls through
to check the next condition. If both conditions are false, branch to the
third successor.
This new branch recipe is used for early-exit loops, to simplify the
representation in VPlan initially, by avoid the need for splitting the
middle block early on, in a way that preserves the single-exit block
property of regions. All exits still go through the latch block, but
they can go to more than 2 successors.
This idea was part of one of the original proposals for how to model
early exits in VPlan, but at that point in time, there was no good way
to handle this during code-gen, and we went with the early split-middle
block approach initially.
Now that we dissolve regions before ::execute, the new recipe can be
lowered nicely after regions have been removed, to a set of VPBBs and
BranchOnCond recipes. The initial lowering preserves the original
structure with the split middle blocks. Follow-ups will improve the
lowering to avoid this splitting, providing performance gains.
PR: https://github.com/llvm/llvm-project/pull/172750
This patch enables the vectorization of the llvm.frexp intrinsic.
Following the suggestion in #112408, frexp is moved from
isTriviallyScalarizable to isTriviallyVectorizable.
Fixes#112408
LV can create step vectors that wrap around, e.g. `step-vector i1` with
VF>2. Allow truncation when creating the vector constant to avoid an
assertion failure with https://github.com/llvm/llvm-project/pull/171456.
After https://github.com/llvm/llvm-project/pull/173494 the definition of
the llvm.stepvector intrinsic has been changed to make it have wrapping
semantics, so the semantics for the fixed and scalable case match now.
Improve known non-nan sign bit tracking. Handle cases with
a known 0 or inf input of indeterminate sign.
The tails of some library functions have sign management
for special cases.
This reverts commit f42af14073228 and re-applies
https://github.com/llvm/llvm-project/pull/172915.
It has an additional check if the condition is a live-in,
which makes sure we preserve the original behavior in that case.
This should fix the crash that caused the revert.
Original commit message:
Instead of looking up the predicate from the VPValue condition instead
of the underlying IR.
This improves cost modeling in some cases, e.g. when we can fold
operations like negations in compares. On AArch64, this leads to
additional vectorization in a few cases in practice.
Example lowering for the modified test case:
https://llvm.godbolt.org/z/6nc6jo5eG
Instead of looking up the predicate from the VPValue condition instead
of the underlying IR.
This improves cost modeling in some cases, e.g. when we can fold
operations like negations in compares. On AArch64, this leads to
additional vectorization in a few cases in practice.
Example lowering for the modified test case:
https://llvm.godbolt.org/z/6nc6jo5eG
PR: https://github.com/llvm/llvm-project/pull/172915
getAddressAccessSCEV previously had some restrictive checks that limited
pointer SCEV expressions passed to TTI to GEPs with operands that must
either be invariant or marked as inductions.
As a consequence, the check rejected things like `GEP %base, (%iv + 1)`,
while the SCEV for the GEP should be as easily analyzeable as for `GEP
%base, %v`, with the only difference being the of the AddRec start
adjusted by 1.
This patch changes the code to use a SCEV-based check, limiting the
address SCEV to be loop invariant, an affine AddRec (i.e. induction ),
or an add expression of such operands or a sign-extended AddRec.
This catches all existing cases getAddressAccessSCEV caught, plus
additional ones like the cases mentioned above.
This means we pass address SCEVs in more cases, giving the backends a
better change to make informed decisions. It also unifies the decision
when to use an address SCEV between the legacy and VPlan-based cost
model.
An illustrative example of showing the impact are the gather-cost.ll
tests. Previously they were considered not profitable to vectorize
because we failed to determine that
%gep.src_data = getelementptr inbounds [1536 x float], ptr @src_data,
i64 0, i64 %mul
has a relatively small constant stride.
There may be some rough edges in the cost models, where not passing
pointer SCEVs hid some incorrect modeling, but those issues should be
fixed in the target cost models if they surface.
PR: https://github.com/llvm/llvm-project/pull/171204
This patch introduces VPInstruction::Reverse and extracts the reverse
operations of loaded/stored values from reverse memory accesses. This
extraction facilitates future support for permutation elimination within
VPlan.
