ISD::SRA and ISD::MUL are already registered unconditionally in the
constructor, so remove the redundant registrations from the
hasVInstructions() block. Fold the standalone SRA call into the existing
brace-initializer list.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Currently llvm.experimental.cttz.elts are directly lowered from the
intrinsic.
If the type isn't legal then the target tells SelectionDAGBuilder to
expand it into a reduction, but this means we can't split the operation.
E.g. it's possible to split a cttz.elts nxv32i1 into two nxv16i1,
instead of expanding it into a nxv32i64 reduction.
vp.cttz.elts can be split because it has a dedicated SelectionDAG node.
This adds CTTZ_ELTS and CTTZ_ELTS[_ZERO_POISON] nodes and just enough
legalization to get tests passing. A follow up patch will add splitting
and move the expansion into LegalizeDAG.
combinePExtTruncate is called from performTRUNCATECombine when the
P-extension is enabled. It attempts to match patterns like
truncate(srl(mul/sub(...), shamt)) and combine them into P-extension
narrowing shift instructions (e.g. PNSRLI, PNSRAI).
However, after extracting the shift input operand `Op` from the SRL
node, the function unconditionally accessed Op.getOperand(0) and
Op.getOperand(1) without first verifying that Op has at least two
operands. For example, when combining:
```
truncate(v2i16
srl(v2i32
bitcast(v2i32 i64), <-- Op = bitcast, a unary op with 1 operand
BUILD_VECTOR <8, 8>))
```
Op is a BITCAST node (unary, only 1 operand), so accessing
Op.getOperand(1) triggers an out-of-bounds assertion:
```
Assertion `Num < NumOperands && "Invalid child # of SDNode!"' failed.
```
Add an early return when Op has fewer than two operands.
Hoist the common ANY_EXTEND, DAG.getNode, and convertFromScalableVector
calls out of the duplicated if/else branches. Use a single IsScalar bool
to select between VMV_S_X_VL/VFMV_S_F_VL and VMV_V_X_VL/VFMV_V_F_VL.
On RV32 with <N x i64> vectors, inserting a value that is a
sign_extend of an i32 only uses the lower 32 bits, so it can be
lowered without scalar legalization, same as i32 constants.
Fixes cannot select errors for other types of shift amounts.
I've made a new RISCVISD node that only allows an immediate operand.
It's assumed that the lowering code will only allow valid immediates so
I'm not using a TImmLeaf in the match.
If the shift is created during LegalizeVectorOps, the shift amount
will be created as a build_vector. Splat_vector is formed by a later
DAGCombine. LegalizeVectorOps will visit the new shift before the
splat_vector can be created. Handle this case too
We only support splat shift amounts. Previously we checked if the shift
amount was a splat_vector and considered it legal.
I don't think there is a guarantee that the splat_vector will stick
around as a splat_vector. It's safer if we capture the splat and create
a dedicated node with a scalar shift amount.
All known crashes have been fixed.
We do still need to work out how fixed length vectors are handled when V
and P are both enabled, but I don't think this option is the solution
for that.
Fold (and X, Mask) ==/!= C -> X ==/!= sext(C, countr_one(Mask))if the
Mask is only clearing redundant sign bits.
The AND may have come from type legalization of a narrower setcc.
Type legalization will check computeNumSignBits before inserting
the AND, but sometimes the sign bits don't appear until after
op legalization. In the motivating examples, the inputs isn't known
to be sign extended until EXTRACT_VECTOR_ELT is converted to vmv.x.s.
Spotted in the before code from #182684.
After https://github.com/llvm/llvm-project/pull/183080 this is no longer
a configurable property.
NOTE: No test changes expected beyond
llvm/test/Transforms/LoopVectorize/scalable-predication.ll which has
been removed because it only existed to verfiy the now unsupported
functionality.
Given the test case:
struct CBase {
virtual void foo();
};
void bar(CBase *Base) {
Base->foo();
}
and using '-emit-call-site-info' with llc, the following DWARF
is produced for the indirect call 'Base->foo()':
1$: DW_TAG_structure_type "CBase"
...
2$: DW_TAG_subprogram "foo"
...
3$: DW_TAG_subprogram "bar"
...
4$: DW_TAG_call_site
...
We add DW_AT_LLVM_virtual_call_origin to existing call-site
information, linking indirect calls to the function-declaration
they correspond to.
4$: DW_TAG_call_site
...
DW_AT_LLVM_virtual_call_origin (2$ "_ZN5CBase3fooEv")
The new attribute DW_AT_LLVM_virtual_call_origin helps to
address the ambiguity to any consumer due to the usage of
DW_AT_call_origin.
The functionality is available to all supported debuggers and
it is generated only for DWARF version 5 or greater.
(setne (and X, 1 << C), 0) is canonicalized to (srl (and X, (1 << C)),
C).
If this is later used as a czero.eqz/nez condition, we can remove
the srl if the and can be represented as an ANDI.
We can't do the NSRLI trick on RV64, but we can use srx similar to what
we do in LowerShiftLeftParts. We need an additional fixup step for the
FSHR result that NSRLI doesn't need.
Assisted-by: claude
…n in DWARF. (#167666)"
This reverts commit 418ba6e8ae2cde7924388142b8ab90c636d2c21f.
The commit caused an ICE due to hitting unreachable in
llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.cpp:1307
Fixes#182337
Given the test case:
struct CBase {
virtual void foo();
};
void bar(CBase *Base) {
Base->foo();
}
and using '-emit-call-site-info' with llc, the following DWARF
is produced for the indirect call 'Base->foo()':
1$: DW_TAG_structure_type "CBase"
...
2$: DW_TAG_subprogram "foo"
...
3$: DW_TAG_subprogram "bar"
...
4$: DW_TAG_call_site
...
We add DW_AT_LLVM_virtual_call_origin to existing call-site
information, linking indirect calls to the function-declaration
they correspond to.
4$: DW_TAG_call_site
...
DW_AT_LLVM_virtual_call_origin (2$ "_ZN5CBase3fooEv")
The new attribute DW_AT_LLVM_virtual_call_origin helps to
address the ambiguity to any consumer due to the usage of
DW_AT_call_origin.
The functionality is available to all supported debuggers.
I want to enable the frame pointer when the call frame size is too large
to access emergency spill slots. To do that I need to know the call
frame size early enough to reserve FP.
The code here is copied from AArch64. ARM does the same. I did not check
other targets.
Splitting this off separately because it stops us from unnecessarily
reserving the base pointer in the some RVV tests. That appears to due to
this check
(!hasReservedCallFrame(MF) && (!MFI.isMaxCallFrameSizeComputed() ||
MFI.getMaxCallFrameSize() != 0))) &&
By calculating early !MFI.isMaxCallFrameSizeComputed() is no longer true
and the size is zero.
WADDAU is rd += zext(rs1) + zext(rs2)
If we only have 1 32-bit input can force rs2 to avoid zeroing the upper
part of a register pair to use ADDD.
Unfortunately, WADDAU clobbers rd so it might need a GPRPair copy
if we need the old value of rd. We might need to look into that in
the future. Maybe we could have convertToThreeAddress could turn
it back into ADDD+WADDU or ADDD+LI.
Assisted-by: claude
I think we may want to be able to fold ADDD nodes independent of the MUL
in some cases. For example turning NSRAI into NSRARI.
If we fold ADDD into WMACC we would need to be able to extract it again.
Keep the nodes separate avoids this.
Code change was assisted by AI.
For an i64 shift by a constant < 32 on RV32, we can use NSRLI
with 32-ShAmt to calculate the high half of the result.
For non-constant shifts, we can use SLX and some bit tricks to
avoid branches. I wanted to use the target independent code from
TargetLowering, but it currently produces worse code.
Assisted-by: claude
If the shift amount might be in the range [0, 31], we can use
NSRL/NSRA to shift the i64 value to compute the lower 32 bits of
the result.
If the shift amount is >= 32, the high half of the result is all
zeros or sign bits. Otherwise it is a srl/sra of the high bits.
I've handled the constant case in ReplaceNodeResults but deferred
the non-constant case to lowerShiftRightParts. This function is
not called for constants. This gives the opportunity for DAGCombine to
optimize the SRL_PARTS/SRA_PARTS if the shift amount can be proven
to be >= 32 or < 32.
Sequences were also discussed on the P extension mailing list here
https://lists.riscv.org/g/tech-p-ext/message/861
Assisted-by: claude
Basically https://github.com/llvm/llvm-project/pull/168506 but for
riscv, so to be clear the hard work here is @heiher 's. I figured we may
as well get some extra eyeballs on this from riscv too.
Previously the riscv backend could not handle `musttail` calls with more
arguments than fit in registers, or any explicit `byval` or `sret`
parameters/return values. Those have now been implemented.
This is part of my push to get more LLVM backends to support `byval` and
`sret` parameters so that rust can stabilize guaranteed tail call
support. See also:
- https://github.com/llvm/llvm-project/pull/168956
- https://github.com/rust-lang/rust/issues/148748
---------
Co-authored-by: WANG Rui <wangrui@loongson.cn>
Similar to #180706, the masked off lanes in vp.reverse are poison so can
be replaced with anything. Because of this, we should be able to fold a
masked vp.reverse(vp.load) into a vp.strided.load stride=-1 even when
the mask isn't all ones.
We have combines for vp.reverse(vp.load) -> vp.strided.load stride=-1
and vp.store(vp.reverse) -> vp.strided.store stride=-1.
If the load or store is masked, the mask needs to be also a vp.reverse
with the same EVL. However we also have the requirement that the mask's
vp.reverse is unmasked (has an all-ones mask).
vp.reverse's mask only sets masked off lanes to poison, and doesn't
affect the permutation of elements. So given those lanes are poison, I
believe the combine is valid for any mask, not just all ones.
This is split off from another patch I plan on posting to generalize
those combines to vector.splice+vector.reverse patterns, as part of
#172961
These are 3 variations of the same operation with a different operand
tied to the destination register. We need to pick the one that
minimizes the number of mvs.
To do this we take the approach used by AArch64 to select between
BIT, BIF, and BSL which the same operations. We define a pseudo
with no tied constraint and expand it after register allocation based
on where the destination register ended up. If the destination
register is none of the operands, we'll insert a mv.
I've replaced RISCVISD::MVM with RISCVISD::MERGE and updated the operand
order accordingly. I find the MERGE name easier to read so I've made it
the canonical name.
Ideally we could use commuteInstructionImpl and the
TwoAddressInstructionPass
to select the opcode before register allocation. That only works if
you can commute exactly 2 operands and maybe change the opcode in the MI
representation of any of the forms to get to the either of the other 2
forms.
That is not possible. We'd need to define 3 more pseudoinstructions
with different permutations.
With the current approach it might be possible that we insert a mv
not because all of the operand registers we needed by later
instructions,
but because the register allocator needed to put the result in a
different register. It's possible a different allocation for other
instructions might have avoided the mv.
I wrote the patch based on the AArch64, but the tests were generated
by AI.
When comparing multi-word integers with Zicond, we generate:
(or (czero_eqz (lo1 < lo2), (hi1 == hi2)),
(czero_nez (hi1 < hi2), (hi1 == hi2)))
The czero_nez is redundant because when hi1 == hi2 is true, hi1 < hi2 is
already 0. This patch adds a DAG combine to recognize:
czero_nez (setcc X, Y, CC), (setcc X, Y, eq) -> (setcc X, Y, CC)
when CC is a strict inequality (lt, gt, ult, ugt).
This saves one instruction in 128-bit comparisons on RV64 with Zicond.
Note the czero_nez becomes a czero.eqz in the final assembly because the
seteq is replaced by an xor that produces 0 when the values are equal.
Part of #179584
Assisted-by: claude
Compressing to a single shuffle doesn't remove any information and the backend can better apply specific optimizations to a single shuffle.
Addresses #176218.
---------
Co-authored-by: Luke Lau <luke_lau@igalia.com>
Extend the existing combineADDDToWMACC DAG combine to also match
RISCVISD::WMULSU and produce RISCVISD::WMACCSU. This is similar to
how ADDD+UMUL_LOHI is combined to WMACCU and ADDD+SMUL_LOHI is
combined to WMACC.
This patch was generated by AI, but I reviewed it.
We add pseudos/patterns for `vabs.v` instruction and handle the
lowering in `RISCVTargetLowering::lowerABS`.
Reviewers: topperc, 4vtomat, mshockwave, preames, lukel97, tclin914
Reviewed By: mshockwave
Pull Request: https://github.com/llvm/llvm-project/pull/180142
We directly lower `ISD::ABDS`/`ISD::ABDU` to `Zvabd` instructions.
Note that we only support SEW=8/16 for `vabd.vv`/`vabdu.vv`.
Reviewers: mshockwave, lukel97, topperc, preames, tclin914, 4vtomat
Reviewed By: lukel97, topperc
Pull Request: https://github.com/llvm/llvm-project/pull/180141
Combine the pattern:
ADDD(addlo, addhi, UMUL_LOHI(x, y).0, UMUL_LOHI(x, y).1)
into:
WMACCU(x, y, addlo, addhi)
And similarly for SMUL_LOHI -> WMACC.
This patch was written with AI, but I reviewed it carefully.
We already do all the checks necessary in order to prioritize
MULHU/MULHS/UMUL_LOHI/SMUL_LOHI over MULHSU/WMULSU. We might as
well just emit the nodes instead of letting generic type legalization
redo the checks.
This is slightly different than the default legalization because we
don't have access to ExpandInteger so we have to emit TRUNCATES and
BUILD_PAIR. Not sure if this will result in any differences in practice.
