Previously we returned i32 on RV32 and i64 on RV64. The instructions
only consume 32 bits and only produce 32 bits. For RV64, the result
is sign extended to 64 bits like *W instructions.
This patch removes this detail from the interface to improve
portability and consistency. This matches the proposal for scalar
intrinsics here https://github.com/riscv-non-isa/riscv-c-api-doc/pull/44
I've included IR autoupgrade support as well.
I'll be doing this for other builtins/intrinsics that currently use
'long' in other patches.
Reviewed By: VincentWu
Differential Revision: https://reviews.llvm.org/D154647
This follows the pattern of lowering VP nodes to equivalent
RISCVISD::*_VL nodes. The nodes are modelled after the VP ISD nodes rather
than the actual zvbb instructions, and I've included a merge operand to be
consistent with the underlying pseudos that were recently refactored.
I've defined the nodes in RISCVInstrInfoVVLpatterns.td as the nodes aren't Zvk
specific, but the patterns are in RISCVInstrInfoZvk.td.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D155229
This allows us to remove some curly braces around the if body.
The code wasn't consistent about it anyway. Comments before is
used in other places in this file already.
Reviewed By: wangpc, MaskRay
Differential Revision: https://reviews.llvm.org/D155390
We can use a null SDValue for the 'false' case. This avoids the
need for an output parameter. This is consistent with other
SelectionDAG code.
Reviewed By: wangpc
Differential Revision: https://reviews.llvm.org/D155388
This allows us to remove some curly braces around the if body.
The code wasn't consistent about it anyway. Comments before is
used in other places in this file already.
Differential Revision: https://reviews.llvm.org/D155390
This fixes some bugs in the original commit:
(1) Operands are passed in correct order when creating new constant
and the binary operator. New tests were added to cover these cases.
(2) Check was added to see if it is safe to commute the select and the binary operator.
Reviewed By: Craig Topper
Differential Revision: https://reviews.llvm.org/D152147
We can use an i64 clmul to emulate i32 clmul.
For clmulh and clmulr we need to zero extend the 32 bit input
to 64 bits then extract either bits [63:32] or [62:31].
Unfortunately, without Zba we need to use 2 shifts for the
zero extends. These can be optimized out later if the producing
instruction already zeroed the upper bits or if we can use lwu.
There are alternative sequences we can use for clmulh/clmulr
when the zero extend isn't free, but those are best handled by
a DAG combine to give the best opportunity for removing the extend.
This allows us to implement i32 clmul C intrinsics proposed in
https://github.com/riscv-non-isa/riscv-c-api-doc/pull/44.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D154729
This patch is a step towards altering how we handle the emission of
condops. Marking ISD::SELECT as legal is a major change in the codegen
path, and gives few options for maintaining the old codegen path when
it is believed to be better (e.g. a better branchless sequence is
possible using non-zicond instructions, or the branch-based sequence is
preferable).
This removes the existing SelectionDAG patterns and moves the logic into
lowerSELECT. Along some small codegen changes you'll note a few minor
regressions in the generated code quality - this are due to the fact
that by lowering the SELECT node early we miss out on combines that
would kick in later when setcc condcodes that aren't natively supported
have been expanded (thus exposing opportunities for optimisation by
performing logical negation and swapping truev/falsev). I've opted to
split out work that addresses these into follow-on patches (especially
as zicond is still 'experimental').
matchSetCC is a straight-forward translation from the version in
RISCVISelDAGToDAG. Ideally, in the future it can be converted to a
helper shared between both files.
Differential Revision: https://reviews.llvm.org/D155083
(shl (zext to iXLenVec), C) is a possible pattern in auto-vectorized code for
indexed loads/stores. But extending to iXLen might be too aggressive, RVV
indexed load/store instructions zero extend their indexed operand to XLEN.
The patch tries to narrow the type of the zero extension. It's benefit to
decrease register pressure.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D154687
This patch adds pseudos and SDNode patterns for vbrev.v, vrev8.v, vclz.v,
vctz.v and vcpop.v.
I've only added them for integer element types so far since we're lacking tests
for floats.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D155216
Depends on D154634
For the cover letter of the patch-set, please checkout D154628.
This is the 7th patch of the patch-set. This patch includes change to
vfcvt_x_f, vfcvt_xu_f, vfwcvt_x_f, vfwcvt_xu_f, vfncvt_x_f, vfncvt_xu_f
vfcvt_f_x, vfcvt_f_xu, vfncvt_f_x vfncvt_f_xu, vfncvt_f_f
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D154635
Depends on D152879.
Specification PR: riscv-non-isa/rvv-intrinsic-doc#226
This patch adds variant of `vfadd` that models the rounding mode control.
The added variant has suffix `_rm` appended to differentiate from the
existing ones that does not alternate `frm` and uses whatever is inside.
The value `7` is used to indicate no rounding mode change. Reusing the
semantic from the rounding mode encoding for scalar floating-point
instructions.
Additional data member `HasFRMRoundModeOp` is added so we can append
`_rm` suffix for the fadd variants that models rounding mode control.
Additional data member `IsRVVFixedPoint` is added so we can define
pseudo instructions with rounding mode operand and distinguish the
instructions between fixed-point and floating-point.
Reviewed By: craig.topper, kito-cheng
Differential Revision: https://reviews.llvm.org/D152996
Some instructions treat x0 as a special encoding rather than as a
value of 0. Since we don't parse the inline assembly to know what
the instruction is, chooser the safest option of never using x0.
Fixes#63747.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D154744
The orc_b and brev8 intrinsics are type overloaded, but only
i32 and XLen are supported types. The type legalization code in
ReplaceNodeResults only handles the i32 case on RV64. Add some
checks so we will fail type legalization for other types.
vwcvt.f.x doesn't use rounding mode. The integer value fits in
the mantissa of a 2x larger FP type so no rounding is required.
I've remove the Uses = [FRM] that is also not needed.
I deleted the isel patterns. Alternatively, we could keep them and
drop the rounding mode immediate. The patterns are currently untested
so I chose to delete them. If they become needed in the future, we
can decide then if we should have the patterns or teach the node
creation to use the non-RM form for widening.
This reverts part of D142102.
Reviewed By: luke
Differential Revision: https://reviews.llvm.org/D154653
As noted by @reames, we should be checking that the memory access is aligned to
the element size (or the unaligned vector memory access feature is enabled)
before lowering vlseg/vsseg intrinsics via the interleaved access pass.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D154536
This constructs a proper memory operand for riscv_vsoxei_mask and riscv_vsuxei_mask.
I think they are missed in D147119.
Reviewed By: kito-cheng
Differential Revision: https://reviews.llvm.org/D154694
Following from D153864, this patch implements the lowerDeinterleaveIntrinsic
hook to lower deinterleaves of loads into vlseg2 intrinsics.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D153876
This patch teaches the RISCV TargetLowering class to lower interleave
intrinsics to vsseg2, so it can lower interleaved stores for scalable vectors.
Previously, we could only lower stores of interleaves for fixed length vectors
with vector shuffles.
This uses the lowerInterleaveIntrinsic interface for the interleaved
access pass that was added in D146218, and subsumes the DAG combine
approach taken in D144175
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D153864
When emitting a vfcvt with a rounding mode, we end up generating an unnecessary
vmset because the only rounding mode pseudos have a mask operand. This patch
adds a pseudo without a mask, and marks the masked variant with the
MaskedPseudo class so the doPeepholeMergeVMV optimisation knows to remove the
redundant vmset.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D154266
Some signed opcodes were being lowered to their unsigned counterparts and
vice-versa.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D154234
As introduced in D99148, RISC-V uses the softPromoteHalf legalisation
for fp16 values without zfh, with logic ensuring that f16 values are
passed in lower bits of FPRs (see D98670) when F or D support is
present. This legalisation produces ISD::FP_TO_FP16 and ISD::FP16_TO_FP
nodes which (as described in ISDOpcodes.h) provide a "semi-softened
interface for dealing with f16 (as an i16)". i.e. the return type of the
FP_TO_FP16 is an integer rather than a float (and the arg of FP16_TO_FP
is an integer). The remainder of the description focuses primarily on
FP_TO_FP16 for ease of explanation.
FP_TO_FP16 is lowered to a libcall to `__truncsfhf2 (float)` or
`__truncdfhf2 (double)`. As of D92241, `_Float16` is used as the return
type of these libcalls if the host compiler accepts `_Float16` in a test
input (i.e. dst_t is set to `_Float16`). `_Float16` is enabled for the
RISC-V target as of D105001 and so the return value should be passed in
an FPR on hard float ABIs.
This patch fixes the ABI issue in what appears to be a minimally
invasive way - leaving the softPromoteHalf logic undisturbed, and
lowering FP_TO_FP16 to an f32-returning libcall, converting its result
to an XLen integer value.
As can be seen in the test changes, the custom lowering for FP16_TO_FP
means the libcall is no longer tail-callable.
Although this patch fixes the issue, there are two open items:
* Redundant fmv.x.w and fmv.w.x pairs are now somtimes produced during
lowering (not a correctness issue).
* Now coverage for STRICT variants of FP16 conversion opcodes.
Differential Revision: https://reviews.llvm.org/D151284
Without the changes from D153598.
Original commit message:
For the same reasons as D151284, this requires custom lowering of the
truncate libcall on hard float ABIs (the normal libcall code path is
used on soft ABIs).
The extend operation is implemented by a shift just as in the standard
legalisation, but needs to be custom lowered because i32 isn't a legal
type on RV64.
This patch aims to make the minimal changes that result in correct
codegen for the bfloat.ll tests.
Differential Revision: https://reviews.llvm.org/D151663
This was committed with D153598 merged into it. Reverting to recommit as separate patches.
This reverts commit 690b1c847f0b188202a86dc25a0a76fd8c4618f4.
With `-fsanitize=kcfi` (Kernel Control-Flow Integrity), Clang emits
"kcfi" operand bundles to indirect call instructions. Similarly to
the target-specific lowering added in D119296, implement KCFI operand
bundle lowering for RISC-V.
This patch disables the generic KCFI pass for RISC-V in Clang, and
adds the KCFI machine function pass in `RISCVPassConfig::addPreSched`
to emit target-specific `KCFI_CHECK` pseudo instructions before calls
that have KCFI operand bundles. The machine function pass also bundles
the instructions to ensure we emit the checks immediately before the
calls, which is not possible with the generic pass.
`KCFI_CHECK` instructions are lowered in `RISCVAsmPrinter` to a
contiguous code sequence that traps if the expected hash in the
operand bundle doesn't match the hash before the target function
address. This patch emits an `ebreak` instruction for error handling
to match the Linux kernel's `BUG()` implementation. Just like for X86,
we also emit trap locations to a `.kcfi_traps` section to support
error handling, as we cannot embed additional information to the trap
instruction itself.
Relands commit 62fa708ceb027713b386c7e0efda994f8bdc27e2 with fixed
tests.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D148385
With `-fsanitize=kcfi` (Kernel Control-Flow Integrity), Clang emits
"kcfi" operand bundles to indirect call instructions. Similarly to
the target-specific lowering added in D119296, implement KCFI operand
bundle lowering for RISC-V.
This patch disables the generic KCFI pass for RISC-V in Clang, and
adds the KCFI machine function pass in `RISCVPassConfig::addPreSched`
to emit target-specific `KCFI_CHECK` pseudo instructions before calls
that have KCFI operand bundles. The machine function pass also bundles
the instructions to ensure we emit the checks immediately before the
calls, which is not possible with the generic pass.
`KCFI_CHECK` instructions are lowered in `RISCVAsmPrinter` to a
contiguous code sequence that traps if the expected hash in the
operand bundle doesn't match the hash before the target function
address. This patch emits an `ebreak` instruction for error handling
to match the Linux kernel's `BUG()` implementation. Just like for X86,
we also emit trap locations to a `.kcfi_traps` section to support
error handling, as we cannot embed additional information to the trap
instruction itself.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D148385
For the same reasons as D151284, this requires custom lowering of the
truncate libcall on hard float ABIs (the normal libcall code path is
used on soft ABIs).
The extend operation is implemented by a shift just as in the standard
legalisation, but needs to be custom lowered because i32 isn't a legal
type on RV64.
This patch aims to make the minimal changes that result in correct
codegen for the bfloat.ll tests.
Differential Revision: https://reviews.llvm.org/D151663
The GPRF64 has the same spill size as GPR and is only used for RV64.
There's no real reason to have it as a separate class other than
for type inference for isel patterns in tablegen.
This patch adds f64 to the GPR register class when XLen=64. I use
f32 when XLen=32 even though we don't make use of it just to avoid
the oddity.
isel patterns have been updated to fix the lack of type infererence.
I might do similar for GPRF16 and GPRF32 or I might change them to
use an optimized spill size instead of always using XLen.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D153110
This is a similar change to one proposed for GCC:
https://inbox.sourceware.org/gcc-patches/20230414170942.1695672-1-patrick@rivosinc.com/
The changes in this patch are based on the proposal by Hans Boehm to more
closely match the intended semantics for sequentially consistent stores
and to allow some platforms to avoid an ABI break when switching to more
performant atomic instructions. Platforms that have already compiled
code using the existing mappings will also have more time to gradually
replace that code in preparation of the switch.
Further details can be found in the psABI proposal:
https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/378.
This patch implements a mapping that is stronger than the one outlined in table
A.6 of the RISC-V unprivileged spec to be future compatible with table A.7 of
the same document. The related discussion can be found at
https://lists.riscv.org/g/tech-unprivileged/topic/risc_v_memory_model_topics/92916241
The major change to RISC-V code generation is that we will now emit a trailing
fence for sequentially consistent stores.
The new code sequence should have the following form:
```
fence rw,w; s{b|h|w|d}; fence rw,rw;
```
Other changes and optimizations like using amoswap will be handled separately.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D149486
This avoids undefs from being expanded to a build vector of zeroes.
As noted by @craig.topper in D153399
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D153411
A build_vector is the canonical representation rather than multiple
insert_vector_elts.
Unfortunately, this regresses quite a few tests now primarily due to not
having a vmv.s.x special case, but I hope we can improve this with future
patches.
Stress testing in our downstream found an infinite loop in DAG combine.
This patch breaks the infinite loop.
The insert_vector_element chain starts with a fixed vector undef.
Fixed vector undef is currently expanded to a build_vector of 0s
which gets lowered to a vmv.v.i. The insert chain overwrites all
elements so SimplifyDemandedVectorElts turns the vmv.v.i back into
undef and the cycle repeats.
We probably should custom lower fixed vector undef to scalable
vector undef. I think that would also fix the infinite loop, but
I didn't test that.
Reviewed By: luke
Differential Revision: https://reviews.llvm.org/D153399
The definition for ISD::EXTRACT_SUBVECTOR says the index must be
aligned to the known minimum elements of the extracted type. We mostly
got away with this but it turns out there are places that depend on this.
For example, this code in getNode for ISD::EXTRACT_SUBVECTOR
```
// EXTRACT_SUBVECTOR of CONCAT_VECTOR can be simplified if the pieces of
// the concat have the same type as the extract.
if (N1.getOpcode() == ISD::CONCAT_VECTORS && N1.getNumOperands() > 0 &&
VT == N1.getOperand(0).getValueType()) {
unsigned Factor = VT.getVectorMinNumElements();
return N1.getOperand(N2C->getZExtValue() / Factor);
}
```
This depends on N2C->getZExtValue() being evenly divisible by Factor.
Reviewed By: luke
Differential Revision: https://reviews.llvm.org/D153380
Unless I'm missing something we need to update the whole vector
not just where OddMask is true.
Reviewed By: luke
Differential Revision: https://reviews.llvm.org/D153087
