#175194, #177159, and #173069 introduced the code calling
`TTI.getMinMaxReductionCost` with unexpected `Intrinsic::ID` causing
RISC-V to fail with `llvm_unreachable` panic.
Functionally, this is a small fix that also ports tests for the
aforementioned folds to RISCV.
Resolves#170500.
Implemented mergeInfo static helper to return common
TTI::OperandValueInfo data .
Added common OperandValueInfo `Op0Info` && `Op1Info` to NewCost
calculation.
This patch removes the single-use restriction of selects in
foldShuffleOfSelects, allowing the fold to trigger for multi-use
instructions as well if the cost model finds it cheaper.
Fixes#173036
Fixes#173037
Remove the `m_OneUse` restriction in `foldShuffleOfIntrinsics` and
update the cost model to account for additional uses of the original intrinsics.
Fixes#173033
This patch extends VectorCombine to fold binary operations through
shuffles in scenarios involving multiple uses of both the binary
operator and its operands.
Previously, the transformation was restricted to single-use cases to
prevent instruction duplication. This change implements a cost-based
evaluation that allows the fold even when:
1. The binary operator has multiple users (requiring duplication of the
arithmetic instruction).
2. The operands of the binary operator (the shuffles) have multiple
users (requiring the original shuffles to be preserved).
The optimization is performed if the TTI cost of the new instruction
sequence—including any duplicated arithmetic—is lower than the cost of
the shuffle sequence it replaces. This is particularly beneficial on X86
targets for expensive cross-lane shuffles.
If we're shuffling/concatenating the same operands then ensure we don't
duplicate the total cost, ensure we reuse the final shuffle and
recognise that we reduce the total instruction count (so fold even when
NewCost == OldCost, not just NewCost < OldCost).
Keeping the extracted element in a natural position in the narrowed
vector has two beneficial effects:
1. It makes the narrowing shuffles cheaper (at least on AMDGPU), which
allows the insert/extract fold to trigger.
2. It makes the narrowing shuffles in a chain of extract/insert
compatible, which allows foldLengthChangingShuffles to successfully
recognize a chain that can be folded.
There are minor X86 test changes that look reasonable to me. The IR
change for AVX2 in
llvm/test/Transforms/VectorCombine/X86/extract-insert-poison.ll
doesn't change the assembly generated by `llc -mtriple=x86_64--
-mattr=AVX2`
at all.
This pr resolves [#170867](https://github.com/llvm/llvm-project/issues/170867)
Existing code recomputes the cost for creating a shuffle instruction even for the
repeating Intrinsic operand pairs. This will result in higher newCost.
Hence the runtime will decide not to fold.
The change proposed in this pr will address this issue. When calculating
the newCost we are skipping the cost calculation of an operand pair if
it was already considered. And when creating the transformed code, we
are reusing the already created shuffle instruction for repeated operand
pair.
[VectorCombine] Fold permute of intrinsics into intrinsic of permutes
Add foldPermuteOfIntrinsic to transform:
shuffle(intrinsic(args), poison) -> intrinsic(shuffle(args))
when the shuffle is a permute (operates on single vector) and the cost
model determines the transformation is profitable.
This optimization is particularly beneficial for subvector extractions
where we can avoid computing unused elements.
For example:
%fma = call <8 x float> @llvm.fma.v8f32(<8 x float> %a, %b, %c)
%result = shufflevector <8 x float> %fma, poison, <4 x i32> <0,1,2,3>
transforms to:
%a_low = shufflevector <8 x float> %a, poison, <4 x i32> <0,1,2,3>
%b_low = shufflevector <8 x float> %b, poison, <4 x i32> <0,1,2,3>
%c_low = shufflevector <8 x float> %c, poison, <4 x i32> <0,1,2,3>
%result = call <4 x float> @llvm.fma.v4f32(%a_low, %b_low, %c_low)
The transformation creates one shuffle per vector argument and calls the
intrinsic with smaller vector types, reducing computation when only a
subset of elements is needed.
The existing foldShuffleOfIntrinsics handled the blend case (two
intrinsic inputs), this adds support for the permute case (single
intrinsic input).
Fixes#170002
Ensure the arguments are passed to the getShuffleCost calls to improve
cost analysis, in particular if these are constant the costs will be
recognised as free
Noticed while reviewing #170052
This change aims to convert vector loads to scalar loads, if they are
only converted to scalars after anyway.
alive2 proof: https://alive2.llvm.org/ce/z/U_rvht
insertelt DestVec, (fneg (extractelt SrcVec, Index)), Index
-> shuffle DestVec, (shuffle (fneg SrcVec), poison, SrcMask), Mask
In previous, the above transform was only possible if the Extract/Insert
Index was the same; this patch makes the above transform possible
even if the two indexes are different.
Proof: https://alive2.llvm.org/ce/z/aDfdyG
Fixes: https://github.com/llvm/llvm-project/issues/125675
This change aims to avoid inserting a freeze instruction between the
load and bitcast when scalarizing extend-extract. This is particularly
useful in combination with
https://github.com/llvm/llvm-project/pull/164682, which can then
potentially further scalarize, provided there is no freeze.
alive2 proof: https://alive2.llvm.org/ce/z/W-GD88
The scalarizeExtExtract transform assumed little-endian lane ordering,
causing miscompiles on big-endian targets such as AIX/PowerPC under -O3
-flto.
This patch updates the shift calculation to handle endianness correctly
for big-endian targets. No functional change
for little-endian targets.
Fixes#158197.
---------
Co-authored-by: Simon Pilgrim <llvm-dev@redking.me.uk>
This patch addresses
https://github.com/llvm/llvm-project/pull/155216#discussion_r2297724663.
This patch adds a helper function to put the inverse cast on constants,
with cast flags preserved(optional).
Follow-up patches will add trunc/ext handling on VectorCombine and flags
preservation on InstCombine.
Technically this could happen with vector units that can't handle all legal scalar widths - but its good enough to use a generic crash test without a suitable target
Fixes#157335
Resolves#154797.
This patch adds the fold `bitop(bitcast(x), C) -> bitop(bitcast(x),
cast(InvC)) -> bitcast(bitop(x, InvC))`.
The helper function `getLosslessInvCast` tries to calculate the constant
`InvC`, satisfying `castop(InvC) == C`, and will try its best to keep
the poison-generated flags of the cast operation.
Consider the following pattern:
```
C = op A B
D = op C
E = op D, C
```
As `E` is dead, we call `eraseInstruction(E)` and see if its operands
become dead. `RecursivelyDeleteTriviallyDeadInstructions(D)` also erases
`C`, which causes a UAF crash in the subsequent call
`RecursivelyDeleteTriviallyDeadInstructions(C)`.
This patch also adds deleted ops into the visit list to avoid double
deletion.
Closes https://github.com/llvm/llvm-project/issues/155543.
`RecursivelyDeleteTriviallyDeadInstructions` is introduced by
https://github.com/llvm/llvm-project/pull/149047 to immediately drop
dead instructions. However, it may invalidate the next iterator in
`make_early_inc_range` in some edge cases, which leads to a crash. This
patch manually maintains the next iterator and updates it when the next
instruction is about to be deleted.
Closes https://github.com/llvm/llvm-project/issues/155110.
The vector combiner will process all instructions as it first loops
through the function, adding any newly added and deleted instructions to
a worklist which is then processed when all nodes are done. These leaves
extra uses in the graph as the initial processing is performed, leading
to sub-optimal decisions being made for other combines. This changes it
so that trivially dead instructions are removed immediately. The main
changes that this requires is to make sure iterator invalidation does not
occur.
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
Fixes#153012
As we tolerate unfoldable constant expressions in `scalarizeOpOrCmp`, we
may fold
```llvm
define void @bug(ptr %ptr1, ptr %ptr2, i64 %idx) #0 {
entry:
%158 = insertelement <2 x i64> <i64 5, i64 ptrtoint (ptr @val to i64)>, i64 %idx, i32 0
%159 = or disjoint <2 x i64> splat (i64 2), %158
store <2 x i64> %159, ptr %ptr2
ret void
}
```
to
```llvm
define void @bug(ptr %ptr1, ptr %ptr2, i64 %idx) {
entry:
%.scalar = or disjoint i64 2, %idx
%0 = or <2 x i64> splat (i64 2), <i64 5, i64 ptrtoint (ptr @val to i64)>
%1 = insertelement <2 x i64> %0, i64 %.scalar, i64 0
store <2 x i64> %1, ptr %ptr2, align 16
ret void
}
```
And it would be folded back in `foldInsExtBinop`, resulting in an
infinite loop.
This patch forces scalarization iff InstSimplify can fold the constant
expression.
When matching integers, `m_ConstantInt` is a convenient alternative to
`m_APInt` for matching unsigned 64-bit integers, allowing one to
simplify
```cpp
const APInt *IntC;
if (match(V, m_APInt(IntC))) {
if (IntC->ule(UINT64_MAX)) {
uint64_t Int = IntC->getZExtValue();
// ...
}
}
```
to
```cpp
uint64_t Int;
if (match(V, m_ConstantInt(Int))) {
// ...
}
```
However, this simplification is only true if `V` is a scalar type.
Specifically, `m_APInt` also matches integer splats, but `m_ConstantInt`
does not.
This patch ensures that the matching behaviour of `m_ConstantInt`
parallels that of `m_APInt`, and also incorporates it in some obvious
places.
This patch add cost kind to `getAddressComputationCost()` for #149955.
Note that this patch also remove all the default value in `getAddressComputationCost()`.
Attempt to narrow a phi of shufflevector instructions where the two
incoming values have the same operands but different masks.
Related to #128938.
---------
Co-authored-by: Leon Clark <leoclark@amd.com>
Reopen#128938.
Attempt to shrink the size of vector loads where only some of the
incoming lanes are used for rebroadcasts in shufflevector instructions.
---------
Co-authored-by: Leon Clark <leoclark@amd.com>
Co-authored-by: Simon Pilgrim <llvm-dev@redking.me.uk>
In some places we were passing the type of value being accessed, in
other cases we were passing the type of the pointer for the access.
The most "involved" user is
LoopVectorizationCostModel::getMemInstScalarizationCost, which is the
only call site that passes in the SCEV, and it passes along the pointer
type.
This changes call sites to consistently pass the pointer type, and
renames the arguments to clarify this.
No target actually checks the contents of the type passed, only to see
if it's a vector or not, so this shouldn't have an effect.
