Fixes#181651
Added DemandedElts argument to isConstOrConstSplat and to
isKnowTobePowerOfTwo calls and OrZero || isKnownNeverZero(Val, Depth) is
checked before isKnowTobePowerOfTwo. Also added unit tests.
Refactor the SelectionDAG::getMemcmp,
SelectionDAG::getStrcpy,SelectionDAG::getStrlen with a existing helper
function SelectionDAG::getRuntimeCallSDValueHelper.
This reverts commit ea3fdc5972db7f2d459e543307af05c357f2be26.
Re-enable const-folding for maxnum/minnum in the middle-end, GlobalISel,
and SelectionDAG.
Re-enable optimizations that depend on maxnum/minnum sNaN semantics in
InstCombine and DAGCombiner.
Now that maxnum(x, sNaN) is specified to non-deterministically produce
either NaN or x, these constant-foldings and optimizations are now valid
again according to the newly clarified semantics in #172012 .
This patch updates `SelectionDAG::isKnownNeverZero` to support `ISD::OR`
by forwarding the `DemandedElts` mask to its operands.
Previously, `ISD::OR` dropped the mask, causing the compiler to be
overly conservative if any lane in the vector was zero, even if that
lane wasn't demanded. This change allows the compiler to prove a vector
result is non-zero even if ignored lanes are zero.
Fixes#183037
**Tests:**
- Moved tests from the C++ file to the IR assembly file
(`known-never-zero.ll`) as requested.
- Confirmed the code now correctly tracks which parts of a vector are
actually needed for `ISD::OR`.
- This allows the compiler to prove a result is "never zero" even if
some unused lanes contain zeros.
Fixes#183038
Adds `isKnownNeverZero` support for `UADDSAT`, `UMAX`, and `UMIN`. This
allows the compiler to prove a vector result is _non-zero_ by analyzing
only the demanded lanes of its operands.
Fixes#183042
This patch updates `SelectionDAG::isKnownNeverZero` to forward the
`DemandedElts` mask when analyzing `ISD::ZERO_EXTEND` and
`ISD::SIGN_EXTEND` nodes.
This patch updates `SelectionDAG::isKnownNeverZero` to support
`ISD::SRA`/`ISD::SRL` by forwarding the `DemandedElts` mask to its
operands.
Fixes#183046
AIX has "millicode" routines, which are functions loaded at boot time
into fixed addresses in kernel memory. This allows them to be customized
for the processor. The __memccpy routine is a millicode implementation;
we use millicode for the memccpy function instead of a library call to
improve performance
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
After https://github.com/llvm/llvm-project/pull/183080 this is no longer
a configurable property.
NOTE: No test changes expected beyond
llvm/test/Transforms/LoopVectorize/scalable-predication.ll which has
been removed because it only existed to verfiy the now unsupported
functionality.
Following changes were made for isKnownNeverZero :
- Added BUILDVECTOR and SPLATVECTOR cases.
- Added support for DemandedElts arguments for SELECT/VSELECT cases.
- Added tests for constants and SELECT/VSELECT.
Closes#181656
Pass DemandedElts and OrZero arguments through SELECT/VSELECT nodes when
checking if values are power-of-two.
Add unit tests for scalar SELECT (OrZero) and vector VSELECT
(DemandedElts + OrZero).
Resolves#181645
Update the signature for isKnownToBeAPowerOfTwo methods to take
DemandedElts and OrZero arguments.
So far all I've done is add OrZero support for constants
(scalars/buildvector/splat), with the intention to incrementally extend
support in the future (or spread the work as beginner GFI patches).
Similarly the DemandedElts argument is currently only used for constant
build vector tests but support can be extended in future patches.
Fixes#178938
DAGCombiner can fold a chain of INSERT_VECTOR_ELT into a vector AND/OR
operation. This patch adds protection to avoid that we end up making the
vector more poisonous by freezing the source vector when the elements
that should be set to 0/-1 may be poison in the source vector.
The patch also fixes a bug in SimplifyDemandedVectorElts for
MUL/MULHU/MULHS/AND that could result in making the vector more
poisonous. Problem was that we skipped demanding elements from Op0 that
were known to be zero in Op1. But that could result in elements being
simplified into poison when simplifying Op0, and then the result would
be poison and not zero after the MUL/MULHU/MULHS/AND. The solution is to
defensively make sure that we demand all the elements originally
demanded also when simplifying Op0.
This bugs were found when analysing the miscompiles in
https://github.com/llvm/llvm-project/issues/179448
Main culprit in #179448 seems to have been the bug in DAGCombiner. The
bug in SimplifyDemandedVectorElts surfaced when fixing the DAGCombiner,
as that fix typically introduce the (AND (FREEZE x), y) pattern that
wasn't handled correctly in SimplifyDemandedVectorElts.
Also fixes#180409.
Also fixes#176682.
There are target intrinsics that logically require two MMOs, such as
llvm.amdgcn.global.load.lds, which is a copy from global memory to LDS,
so there's both a load and a store to different addresses.
Add an overload of getTgtMemIntrinsic that produces intrinsic info in a
vector, and implement it in terms of the existing (now protected)
overload.
GlobalISel and SelectionDAG paths are updated to support multiple MMOs.
The main part of this change is supporting multiple MMOs in
MemIntrinsicNodes.
Converting the backends to using the new overload is a fairly mechanical step
that is done in a separate change in the hope that that allows reducing merging
pains during review and for downstreams. A later change will then enable
using multiple MMOs in AMDGPU.
1. AIX has "millicode" routines, which are functions loaded at boot time
into fixed addresses in kernel memory. This allows them to be customized
for the processor. The __strcmp routine is a millicode implementation;
we use millicode for the strcmp function instead of a library call to
improve performance.
TLI.isBinOp recognises some opcodes that have multiple results,
including UADDO etc.
In most cases we currently just bail if a binop has multiple results,
but shuffle combining was missing the check and its pretty trivial to
add handling in this case.
I've added add/sub-overflow opcodes to verifyNode to help catch these
cases in the future - IIRC there was a plan to autogen these, but there
isn't anything at the moment.
Fixes#179112
This patch extends `BuildVectorSDNode::isConstantSequence` to recognize
constant sequences that contain undef elements at any position.
The new implementation finds the first two non-undef constant elements,
computes the stride from their difference, then verifies all other
defined elements match the sequence. This enables SVE's INDEX
instruction to be used in more cases.
This change particularly benefits ZIP1/ZIP2 patterns where one operand
is a constant sequence. When a smaller constant vector like `<0, 1, 2,
3>` is used in a ZIP1 shuffle producing a wider result, it gets expanded
with trailing undefs. Similarly, for ZIP2 patterns, the DAG combiner
transforms the constant to have leading undefs since ZIP2 only uses the
upper half of its operands.
In particular, these patterns arise naturally from `VectorCombine`'s
`compactShuffleOperands` optimization (see #176074) that I am suggesting
as a fix for #137447.
This change improves memset code generation for non-zero values on
AArch64 by using NEON's DUP instruction instead of
the less efficient multiplication with 0x01010101 pattern.
For small sizes, the value is extracted from a larger DUP. For
non-power-of-two sizes, overlapping stores are used in some cases.
TargetLowering::findOptimalMemOpLowering is modified to allow explicitly
specifying the size of the constant in cases where the constant is
larger than the store operations.
Fixes#165949
The previous getMemcpyLoadsAndStores implementation would chain
load/store instructions from "NumLdStInMemcpy - GlueIter -
GluedLdStLimit" to "NumLdStInMemcpy - GlueIter". This approach caused
issues when copying non-power-of-two sizes, as it would chain leading
load/stores with subsequent instructions at non-power-of-two aligned
offsets.
This chaining pattern prevented optimal optimizations in
aarch64-ldst-opt pass for these load/store instructions.
This commit modifies the chaining range to be from GlueIter to GlueIter
+ GluedLdStLimit, enabling proper optimization of load/store
instructions in aarch64-ldst-opt.
Closes https://github.com/llvm/llvm-project/issues/165947
This assert should not have existed, because just below it the code
bails out for that same condition. The case of the vector being a
scalable vector also shouldn't cause the compiler to crash with an
assertion failure, and instead it should just avoid analysing the
expression.
1. Implement `SelectionDAG::computeKnownBits` for TRUNCATE_SSAT_S/U and
TRUNCATE_USAT_U
2. Saturating truncation operations are well-defined for all inputs and
cannot create poison or undef values. This allows the optimizer to
eliminate unnecessary freeze instructions after these operations.
Fixes#152143
Extend the existing DAGCombine logic in visitIMINMAX so that signed and
unsigned MIN/MAX can be flipped not only when both operands are known
non-negative but also when both operands are known negative. This
replaces the old SignBitIsZero checks with computeKnownBits and explicit
tests for non-negative or negative operands while keeping all existing
legality and saturation gating in place. Add regression tests to cover
both the known-negative case and the known-non-negative case.
Fixes#174325
Libcall lowering decisions should come from the LibcallLoweringInfo
analysis. Query this through the DAG, so eventually the source
can be the analysis. For the moment this is just a wrapper around
the TargetLowering information.
AIX has "millicode" routines, which are functions loaded at boot time
into fixed addresses in kernel memory. This allows them to be customized
for the processor. The __strstr routine is a millicode implementation;
we use millicode for the strstr function instead of a library call to
improve performance.
I add a helper function `getRuntimeCallSDValueHelper` in the patch. I
will refactor the function `SelectionDAG::getStrlen`
`SelectionDAG::getStrcpy` etc later in another patch.
Add handling for CTLS using the same method as in
https://github.com/llvm/llvm-project/pull/174636.
Added tests to AArch64 and RISCV, but it seems that ARM is actually
resolving `llvm.arm.cls` to `clz`, so not tests added there.
