When creating new nodes with illegal types after type legalization, we
should try to use promoted type to avoid creating nodes with illegal
types.
Fixes: https://github.com/llvm/llvm-project/issues/177155
This patch adds a case in getIntrinsicInstrCost and
getTypeBasedIntrinsicInstrCost in
llvm/include/llvm/CodeGen/BasicTTIImpl.h for Intrinsic::clmul. This
patch uses TLI->isOperationLegalOrCustom to check if the instruction is
cheap. If not cheap, it sums up the cost of the arithmetic operations
(AND, SHIFT, XOR) multiplied by the bit width.
Fixes#176354
This change improves memset code generation for non-zero values on
AArch64 by using NEON's DUP instruction instead of
the less efficient multiplication with 0x01010101 pattern.
For small sizes, the value is extracted from a larger DUP. For
non-power-of-two sizes, overlapping stores are used in some cases.
TargetLowering::findOptimalMemOpLowering is modified to allow explicitly
specifying the size of the constant in cases where the constant is
larger than the store operations.
Fixes#165949
This restriction was originally added in
https://reviews.llvm.org/D143256, with the given justification:
> Currently, in TargetLowering, if the target does not support fminnum,
we lower to fminimum if neither operand could be a NaN. But this isn't
quite correct because fminnum and fminimum treat +/-0 differently; so,
we need to prove that one of the operands isn't a zero.
As far as I can tell, this was never correct. Before
https://github.com/llvm/llvm-project/pull/172012, `minnum` and `maxnum`
were nondeterministic with regards to signed zero, so it's always been
perfectly legal to lower them to operations that order signed zeroes.
Following on from #170796, this PR implements the second part of
https://discourse.llvm.org/t/rfc-allow-non-constant-offsets-in-llvm-vector-splice/88974
by allowing non-constant offsets in the vector splice intrinsics.
Previously @llvm.vector.splice had a restriction enforced by the
verifier that the offset had to be known to be within the range of the
vector at compile time. Because we can't enforce this with non-constant
offsets, it's been relaxed so that offsets that would slide the vector
out of bounds return a poison value, similar to
insertelement/extractelement.
@llvm.vector.splice.left also previously only allowed offsets within the
range 0 <= Offset < N, but this has been relaxed to 0 <= Offset <= N so
that it's consistent with @llvm.vector.splice.right.
In lieu of the verifier checks that were removed, InstSimplify has been
taught to fold splices to poison when the offset is out of bounds.
The cost model isn't implemented in this PR, and just returns invalid
for any non-constant offsets for now. I think the correct way to cost
these non-constant offets isn't through getShuffleCost because they
can't handle variable masks, but instead just through
getIntrinsicInstCost.
When comparisons produce all-zeros or all-ones in scalars or per lane in
vectors, comparing results of such comparisons against 0 is an identity
operation. This change eliminates redundant comparison instructions
after another comparison operation.
On some targets, BoolVT may have been widened earlier. In those cases,
choosing StepVT to be smaller can cause crashes when widening the
mis-matched select. Without the fix, the new test
@extract_last_active_v4i32_penryn crashes when trying to widen.
It also improves codegen for other cases.
PR: https://github.com/llvm/llvm-project/pull/175971
Fold splat_vector(fneg(X)) -> splat_vector(-X)
Call the getCheaperNegatedExpression function, and ISD::SPLAT_VECTOR
return NegatibleCost::Cheaper.
This optimization is applied only to the fneg instruction.
This PR implements the first change outlined in
https://discourse.llvm.org/t/rfc-allow-non-constant-offsets-in-llvm-vector-splice/88974?u=lukel
In order to allow non-immediate offsets in the llvm.vector.splice
intrinsic, we need to separate out the "shift left" and "shift right"
modes into two separate intrinsics, which were previously determined by
whether or not the offset is positive or negative.
The description in the LangRef has also been reworded in terms of
sliding elements left or right and extracting either the upper or lower
half as opposed to extracting from a certain index, which brings it
inline with the definition of `llvm.fshr.*`/`llvm.fshl.*`.
This patch teaches AutoUpgrade.cpp to upgrade the old intrinsics into
their new equivalent one based on their offset, so existing uses of
vector.splice should still work.
Uses of llvm.vector.splice in `llvm/test/CodeGen` haven't been replaced
in this PR to keep the diff small and kick the tyres on the AutoUpgrader
a bit. I planned to do this in a follow up NFC but can include it in
this PR if reviewers prefer.
Similarly the shuffle costing kind `SK_Splice` has just been kept the
same for now, to be split into `SK_SpliceLeft` and `SK_SpliceRight`
later.
In line with a std proposal to introduce the llvm.clmul family of
intrinsics corresponding to carry-less multiply operations. This work
builds upon 727ee7e ([APInt] Introduce carry-less multiply primitives),
and follow-up patches will introduce custom-lowering on supported
targets, replacing target-specific clmul intrinsics.
Testing is done on the RISC-V target, which should be sufficient to
prove that the intrinsics work, since no RISC-V specific lowering has
been added.
Ref: https://isocpp.org/files/papers/P3642R3.html
Co-authored-by: Craig Topper <craig.topper@sifive.com>
We currently use the generic expansions to custom lower integer min/max
instructions, but if we have sufficient leading bits, SSE/AVX is always
better off handling it directly with smaller types.
vXi64 cmp/min/max is particularly weak, and as we narrow the types the
better legality we have - this approach seems to work well for x86, but
I'm not sure if its valid enough to try generically in this manner.
However, I added the signed/unsigned generic flip fold to
expandIntMINMAX to further improve SSE2 codegen, similar to what we
already attempt in DAGCombiner (which with a bit more work we might be
able to remove now).
All thats missing is better ComputeNumSignBits handling for vXi64 ashr
expansion, which still misses a lot of cases when split across vXi32
types and shuffles.
Fixes#174169
Treat these like other shift operations by allowing the shift amount to
be a different type than the result.
The PromoteIntOp_Shift and LegalizeDAG code are not tested due to lack
of target support.
I'm looking at adding SSHLSAT for the RISC-V P extension. I don't need
this support for that since RISC-V only has one legal type. I just thought it
was odd that they weren't like other shifts.
I recently observed that LLVM generates the following code:
```
addi a1, a0, -1
sltu a0, a0, a1
addi a0, a0, -1
and a0, a0, a1
ret
```
This could be optimized using the snez instruction instead.
Builds up on the solution proposed for #169491 and #169924 and applies
it for SDIV exact as well. Almost a carbon copy of UDIV exact solution
from #169949.
1. On GPR32 platform, expandIS_FPCLASS may fail due to ISD::BITCAST
double to int64 may fail. Let's FP_ROUND double to float first.
Since we use it if MinMax is zero only, so the flushing won't
break anything.
2. Only one IS_FPCLASS is needed. MinMax will always be RHS if equal.
So we can select between LHS and MinMax.
It will even safe if FP_ROUND flush a small LHS, as if LHS is not zero
then, MinMax won't be Zero, so we will always use MinMax.
---------
Co-authored-by: Nikita Popov <github@npopov.com>
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Similar to how getElementCount avoids the need to reason about fixed and
scalable ElementCounts separately, this patch adds getTypeSize to do the
same for TypeSize.
It also goes through and replaces some of the manual uses of getVScale
with getTypeSize/getElementCount where possible.
This patch adds SVE support for the `masked.compressstore` intrinsic via
the existing `VECTOR_COMPRESS` lowering and compressing the store mask
via `VECREDUCE_ADD`.
Currently, only `nxv4[i32|f32]` and `nxv2[i64|f64]` are directly
supported, with other types promoted to these, where possible.
This is done in preparation for LV support of this intrinsic, which is
currently being worked on in #140723.
Currently LibcallLoweringInfo is defined inside of TargetLowering,
which is owned by the subtarget. Pass in the subtarget so we can
construct LibcallLoweringInfo with the subtarget. This is a temporary
step that should be revertable in the future, after LibcallLoweringInfo
is moved out of TargetLowering.
When a load or store accesses N bytes starting from a pointer P, and we want to
compute an offset pointer within these N bytes after P, we know that the
arithmetic to add the offset must be inbounds. This is for example relevant
when legalizing too-wide memory accesses, when lowering memcpy&Co., or when
optimizing "vector-load -> extractelement" into an offset load.
For SWDEV-516125.
Reland #161355, after fixing up the cross-projects-tests for the wasm
simd intrinsics.
Original commit message:
Lower v4f32 and v2f64 fmuladd calls to relaxed_madd instructions.
If we have FP16, then lower v8f16 fmuladds to FMA.
I've introduced an ISD node for fmuladd to maintain the rounding
ambiguity through legalization / combine / isel.
Lower v4f32 and v2f64 fmuladd calls to relaxed_madd instructions.
If we have FP16, then lower v8f16 fmuladds to FMA.
I've introduced an ISD node for fmuladd to maintain the rounding
ambiguity through legalization / combine / isel.
On targets where f32 maximumnum is legal, but maximumnum on vectors of
smaller types is not legal (e.g. v2f16), try unrolling the vector first
as part of the expansion.
Only fall back to expanding the full maximumnum computation into
compares + selects if maximumnum on the scalar element type cannot be
supported.
Same deal we use for determining ucmp vs scmp.
Using selects on platforms that like selects is better than using usubo.
Rename function to be more general fitting this new description.
There are more places in SIISelLowering.cpp and AMDGPUISelDAGToDAG.cpp
that check for ISD::ADD in a pointer context, but as far as I can tell
those are only relevant for 32-bit pointer arithmetic (like frame
indices/scratch addresses and LDS), for which we don't enable PTRADD
generation yet.
For SWDEV-516125.
As reported in https://github.com/llvm/llvm-project/issues/141034
SelectionDAG::getNode had some unexpected
behaviors when trying to create vectors with UNDEF elements. Since
we treat both UNDEF and POISON as undefined (when using isUndef())
we can't just fold away INSERT_VECTOR_ELT/INSERT_SUBVECTOR based on
isUndef(), as that could make the resulting vector more poisonous.
Same kind of bug existed in DAGCombiner::visitINSERT_SUBVECTOR.
Here are some examples:
This fold was done even if vec[idx] was POISON:
INSERT_VECTOR_ELT vec, UNDEF, idx -> vec
This fold was done even if any of vec[idx..idx+size] was POISON:
INSERT_SUBVECTOR vec, UNDEF, idx -> vec
This fold was done even if the elements not extracted from vec could
be POISON:
sub = EXTRACT_SUBVECTOR vec, idx
INSERT_SUBVECTOR UNDEF, sub, idx -> vec
With this patch we avoid such folds unless we can prove that the
result isn't more poisonous when eliminating the insert.
Fixes https://github.com/llvm/llvm-project/issues/141034
This patch reverts changes from commit 585e65d3307f5f0
(https://reviews.llvm.org/D104250), as it doesn't seem to be needed
nowadays.
The removed code was doing a recursive call to
SimplifyDemandedVectorElts trying to simplify the vector %vec when
finding things like
(SCALAR_TO_VECTOR (EXTRACT_VECTOR_ELT %vec, 0))
I figure that (EXTRACT_VECTOR_ELT %vec, 0) would be simplified based on
only demanding element zero regardless of being used in a
SCALAR_TO_VECTOR operation or not.
It had been different if the code tried to simplify the whole expression
as %vec. That could also have motivate why to make element zero a
special case. But it only simplified %vec without folding away the
SCALAR_TO_VECTOR.
Instead of std::optional<uint64_t>. Shift amounts must be less than or
equal to our maximum supported bit widths which fit in unsigned. Most of
the callers already assumed it fit in unsigned.
