Use the probe loop structure to allocate vector code in the stack as
well. We add the pseudo instruction RISCV::PROBED_STACKALLOC_RVV to
differentiate from the normal loop.
This takes inspiration from AArch64 which does the same thing to assist
with zip/trn/etc.. Doing this recursion unconditionally when the mask
allows is slightly questionable, but seems to work out okay in practice.
As a bit of context, it's helpful to realize that we have existing logic
in both DAGCombine and InstCombine which mutates the element width of in
an analogous manner. However, that code has two restriction which
prevent it from handling the motivating cases here. First, it only
triggers if there is a bitcast involving a different element type.
Second, the matcher used considers a partially undef wide element to be
a non-match. I considered trying to relax those assumptions, but the
information loss for undef in mid-level opt seemed more likely to open a
can of worms than I wanted.
`RegisterClassInfo::getRegPressureSetLimit` is a wrapper of
`TargetRegisterInfo::getRegPressureSetLimit` with some logics to
adjust the limit by removing reserved registers.
It seems that we shouldn't use
`TargetRegisterInfo::getRegPressureSetLimit`
directly, just like the comment "This limit must be adjusted
dynamically for reserved registers" said.
Separate from https://github.com/llvm/llvm-project/pull/118787
Some masked pseudos like PseudoVCPOP_M_B8_MASK don't have a passthru,
but in the masked->unmasked peephole we assumed the masked pseudo always
had one.
This checks for a passthru first and fixes#122245.
When these instructions are marked nofpexcept, we can optimize them.
There are some added toggles in the output, likley because other
noexcept fp instructions are not part of isSupportedInstr yet. We may
want to avoid marking an instruction as isSupported in the future if any
of its FP users are missing nofpexcept to avoid added toggles. However,
we seem to get some GPRs back as a result of this change, which may
outweigh the cost of avoiding extra toggles.
The plan is to follow this patch up with added support for more FP
instructions in the same way. The instructions in this patch are a
natural starting point because they allow us to test with integer
instructions which have good support already.
Reductions are weird because for some operands, they are vector
registers but only read the first lane. For these operands, we do not
need to check to make sure the EEW and EMUL ratios match. The EEWs,
however, do need to match.
Although we cannot reduce the VL of these instructions (i.e. add to
isSupported) we can add them to getOperandInfo to enable optimization
where the FP vector instruction are users. Most of the instructions are
covered by existing tests, and I added tests for the narrowing
conversions because I was a little unsure whether the dest or the source
was 2*SEW and 2*LMUL.
This pattern was originally spotted in 429.mcf by @topperc.
We already have a DAGCombiner pattern to turn `(neg (abs x))` into `(min
x, (neg x))`. But in some cases `(neg (max x, (neg x)))` is formed by an
expanded `abs` followed by a `neg` that is generated only after the
`abs` expansion. This patch adds a separate pattern to match cases like
this, as well as its inverse pattern: `(neg (min X, (neg X))) --> (max
X, (neg X))`.
This pattern is applicable to both signed and unsigned min/max.
This reverts commit b8952d4b1b0c73bf39d6440ad3166a088ced563f.
spec x264 fails to build in all VLS configurations, with the assertion
failure: clang: ../llvm-project/llvm/lib/Target/RISCV/RISCVISelLowering.cpp:5246: llvm::SDValue lowerShuffleViaVRegSplitting(llvm::ShuffleVectorSDNode*, llvm::SelectionDAG&, const llvm::RISCVSubtarget&): Assertion `RegCnt == NumOfDestRegs && "Whole vector must be processed"' failed.
I can reduce a failing piece of IR, but the failure appears pretty
broad, so I suspect any reasonable vls build will hit it.
Patch adds usage of processShuffleMasks in in codegen
in lowerShuffleViaVRegSplitting. This function is already used for X86
shuffles estimations and in DAGTypeLegalizer::SplitVecRes_VECTOR_SHUFFLE
functions, unifies the code.
Reviewers: preames, topperc, lukel97, wangpc-pp
Reviewed By: wangpc-pp
Pull Request: https://github.com/llvm/llvm-project/pull/120803
SEWGreaterThanOrEqualAndLessThan64 is a stricter constraint so it should
have a higher value than SEWGreaterThanOrEqual.
Found by our random test generator.
Prior to this patch, we required that all users had the same VL in order
to optimize. But as the FIXME said, we can use the largest VL to
optimize, as long as we can determine what the largest is. This patch
implements the FIXME.
The default legalization uses vmslt with a vector of XLen to compute a
mask. This doesn't work if the type isn't legal. For fixed vectors it
will scalarize. For scalable vectors it crashes the compiler.
This patch uses an alternate strategy that promotes the i1 vector to an
i8 vector and does the merge. I don't claim this to be the best
lowering. I wrote it quickly almost 3 years ago when a crash was
reported in our downstream.
Fixes#120405.
Scalarize vector FPOWI instead of promoting the type. This allows the
scalar FPOWIs to be visited and converted to libcalls before promoting
the type.
FIXME: This should be done in LegalizeVectorOps/LegalizeDAG, but call
lowering needs the unpromoted EVT.
Without this patch, in some backends, such as RISCV64 and LoongArch64,
the i32 type is illegal and will be promoted. This causes exponent type
check to fail when ISD::FPOWI node generates a libcall.
Fix https://github.com/llvm/llvm-project/issues/118079
Use `MO.getOperandNo() == 0` instead of `IsMODef` so naming is clear for the store, since the store should treat its operand 0 like that even though it is not a def.The load should treat its operand 0 def in the same way.
Now that we have testing of all instructions in the isSupportedInstr
switch, and better coverage of getOperandInfo, I think it is a good time
to enable this by default.
This is a follow up to 2af2634 to use vmv.v.x of i8 constants instead of
the prior vid/vand/vmsne sequence. The advantage of the vmv.v.x sequence
is that it's always m1 (so cheaper at high LMUL), and can be
rematerialized by the register allocator if needed to locally reduce
register pressure.
I found this crash when trying to enable the VLOptimizer pass. We need
this patch before we can enable by default. The old assert was not
checking that USE and DEF were vector registers. The correct condition
is guarded at the callsite of tryReduceVL.
Mask only instructions like vmand and vmsbf should always have 0 for
their Log2SEW operand. Non-mask instructions should only have
3, 4, 5, or 6 for their Log2SEW operand.
Split the operand type so we can verify these cases separately.
I had to fix the SEW for whole register move to vmv.v.v copy
optimization and update an mir test. The vmv.v.v change isn't
functional since we have already done vsetvli insertion before and
nothing else uses the field after copy expansion. I can split these
changes off if desired.
These instructions should always be created with Log2SEW=0 and an LMUL
based on SEW=8. This is used by the vsetvli pass to know these
instructions only care about the ratio and not the specific value.
Looks like I fixed riscv_vmsge(u)_mask intrinsics years ago, but forgot
the unmasked intrinsics.
I'm working on an enhancement to our MachineVerifier checks that will
require VMNAND and VMSET to have Log2SEW=0.
Replace LMUL suffixes with _B1, _B2, etc. This matches what we do
for other mask only instructions like VCPOP_M, VFIRST_M, VMSBF_M,
VLM, VSM, etc.
Now all pseudoinstructions that use Log2SEW=0 will be consistently
named.
These are suffixed with B1, B2, B4, B8, B16, B32, or B64 which I think
these were supposed to match the naming of the vbool types from C where
the number should be SEW/LMUL. So the smallest mask is 64 and the
largest is 1. This provides a compact syntax for describing the 7
possible ratios between LMUL and SEW.
We had the instruction names in the opposite order.
We need to adjust getEMULEqualsEEWDivSEWTimesLMUL to account for the
fact that Log2EEW for mask instructions is 0 but their EMUL is
calculated using Log2EEW=3.
Masking out most significant bits can be done with shl followed by srl
with same shift amount. If this is followed by a shl, we could instead
srl by a smaller amount of bits.
This transform is already implemented in tablegen for masking out
32 most significant bits.
Emits better code for e.g.
float *index(float *p, int i)
{
return p + (i & (1 << 30) - 1);
}
In x264, there's a few kernels with shuffles like this:
%41 = add nsw <16 x i32> %39, %40
%42 = sub nsw <16 x i32> %39, %40
%43 = shufflevector <16 x i32> %41, <16 x i32> %42, <16 x i32> <i32 11,
i32 15, i32 7, i32 3, i32 26, i32 30, i32 22, i32 18, i32 9, i32 13, i32
5, i32 1, i32 24, i32 28, i32 20, i32 16>
Because this is a complex two-source shuffle, this will get lowered as
two vrgather.vvs that are blended together.
vadd.vv v20, v16, v12
vsub.vv v12, v16, v12
vrgatherei16.vv v24, v20, v10
vrgatherei16.vv v24, v12, v16, v0.t
However the indices coming from each source are disjoint, so we can
blend the two together and perform a single source shuffle instead:
%41 = add nsw <16 x i32> %39, %40
%42 = sub nsw <16 x i32> %39, %40
%43 = select <0,0,0,0,1,1,1,1,0,0,0,0,1,1,1,1> %41, %42
%44 = shufflevector <16 x i32> %43, <16 x i32> poison, <16 x i32> <i32
11, i32 15, i32 7, i32 3, i32 10, i32 14, i32 6, i32 2, i32 9, i32 13,
i32 5, i32 1, i32 8, i32 12, i32 4, i32 0>
The select will likely get merged into the preceding instruction, and
then we only have to do one vrgather.vv:
vadd.vv v20, v16, v12
vsub.vv v20, v16, v12, v0.t
vrgatherei16.vv v24, v20, v10
This patch bails if either of the sources are a broadcast/splat/identity
shuffle, since that will usually already have some sort of cheaper
lowering.
This improves performance on 525.x264_r by 4.12% with -O3 -flto
-march=rva22u64_v on the spacemit-x60.
