Previously we returned i32 on RV32 and i64 on RV64. The instructions
only consume 32 bits and only produce 32 bits. For RV64, the result
is sign extended to 64 bits like *W instructions.
This patch removes this detail from the interface to improve
portability and consistency. This matches the proposal for scalar
intrinsics here https://github.com/riscv-non-isa/riscv-c-api-doc/pull/44
I've included IR autoupgrade support as well.
I'll be doing this for other builtins/intrinsics that currently use
'long' in other patches.
Reviewed By: VincentWu
Differential Revision: https://reviews.llvm.org/D154647
Inspired by some of the cases from D145468
Let SimplifyDemandedBits handle the narrowing of lshr to half-width if we don't require the upper bits, the narrowed shift is profitable and the zext/trunc are free.
A future patch will propose the equivalent shl narrowing combine.
Differential Revision: https://reviews.llvm.org/D146121
If somehow a vXi64 bool sign_extend_inreg pattern has been lowered to vector shifts (without PSRAQ support), then try to canonicalize to vXi32 shifts to improve likelihood of value tracking being able to fold them away.
Using a PSLLQ and bitcasted PSRAD node make it very difficult for later fold to recover from this.
When doing a masked load of an illegal unpacked type and then
zero-extending to some illegal wider types we sometimes end up
with pointless 'and' instructions that are trying to zero bits
that we already know are zero. This patch fixes that by adding
more cases to performSVEAndCombine.
Differential Revision: https://reviews.llvm.org/D155281
This follows the pattern of lowering VP nodes to equivalent
RISCVISD::*_VL nodes. The nodes are modelled after the VP ISD nodes rather
than the actual zvbb instructions, and I've included a merge operand to be
consistent with the underlying pseudos that were recently refactored.
I've defined the nodes in RISCVInstrInfoVVLpatterns.td as the nodes aren't Zvk
specific, but the patterns are in RISCVInstrInfoZvk.td.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D155229
Set setMaxAtomicSizeInBitsSupported for Mips. Set the value as appropriate for 64-bit MIPS vs 32-bit.
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D141189
When generating XCOFF, the compiler generates a csect with an internal
name. Each function results in a label within the csect. This patch
replaces the internal name ".text" with an empty string "". This avoids
adding special code to handle a function text() in the source file, and
works better with some XCOFF tools that are confused when the csect and
the first function have the same address.
Reviewed By: hubert.reinterpretcast
Differential Revision: https://reviews.llvm.org/D154854
For rv32, we test the legalization of i8, i16 and i32. For rv64, we additionally test the legalization of i64.
This is the first of a series of commits aiming to legalize arithmetic instructions for RISCV.
Reviewed By: craig.topper, arsenm
Differential Revision: https://reviews.llvm.org/D154978
- Switch to generated checks
- Use a different run line per denormal mode to reduce test duplication
- Add test coverage for rsqrt cases
- Add test coverage for repeated arcp denominator
- Fix the optnone test
In change https://reviews.llvm.org/D152790, it was discovered that the
alignment requirement calculation for LDRD/STRD codegen was suboptimal
and the calculation for volatile loads and stores was adjusted.
This change here adopts the calculation for the remaining non-volatile
occurances.
Recommitting after undefined behavior fix in D155093.
Differential Revision: https://reviews.llvm.org/D153800
This fixes some bugs in the original commit:
(1) Operands are passed in correct order when creating new constant
and the binary operator. New tests were added to cover these cases.
(2) Check was added to see if it is safe to commute the select and the binary operator.
Reviewed By: Craig Topper
Differential Revision: https://reviews.llvm.org/D152147
Future cpu instructions dmxxinstdmr512 and dmxxextfdmr512 insert and extract
quad vectors from the new wide accumulator(wacc) register class.
The introduction of these new instructions renders the p10 instructions
xxmtacc and xxmfacc obsolete since the new wacc register class is a better
choice for handing quad vector operations. This patch ensures that, for
future cpu, instructions dmxxinstdmr512 and dmxxextfdmr512 are generated
by custom lowering the intrinsics for xxm[t|f]acc to produce no instructions.
Reviewed By: amyk, lei
Differential Revision: https://reviews.llvm.org/D153034
We can use an i64 clmul to emulate i32 clmul.
For clmulh and clmulr we need to zero extend the 32 bit input
to 64 bits then extract either bits [63:32] or [62:31].
Unfortunately, without Zba we need to use 2 shifts for the
zero extends. These can be optimized out later if the producing
instruction already zeroed the upper bits or if we can use lwu.
There are alternative sequences we can use for clmulh/clmulr
when the zero extend isn't free, but those are best handled by
a DAG combine to give the best opportunity for removing the extend.
This allows us to implement i32 clmul C intrinsics proposed in
https://github.com/riscv-non-isa/riscv-c-api-doc/pull/44.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D154729
This is a mostly NFC change cleaning up and clarifying components of the
in-tree CORE-V (xcv*) extensions following discussions on the remaining
extensions.
This makes the following changes to the xcbitmanip and xcvmac support:
1. Add missing extensions from RISCVISAInfo, such that they can be
supported in clang's -march option.
2. Clarify the extension version number is 1.0.0 in documentation.
3. Clarify the extensions are by OpenHW Group, and the capitilization
of the CORE-V extension family.
4. Add CORE-V to extension name in RISCVFeatures, both to be consistent
with other vendors, and also better distinguish e.g. CORE-V bit
manipulation vs RISC-V's standard Zb extensions.
Differential Revision: https://reviews.llvm.org/D155283
This patch is a step towards altering how we handle the emission of
condops. Marking ISD::SELECT as legal is a major change in the codegen
path, and gives few options for maintaining the old codegen path when
it is believed to be better (e.g. a better branchless sequence is
possible using non-zicond instructions, or the branch-based sequence is
preferable).
This removes the existing SelectionDAG patterns and moves the logic into
lowerSELECT. Along some small codegen changes you'll note a few minor
regressions in the generated code quality - this are due to the fact
that by lowering the SELECT node early we miss out on combines that
would kick in later when setcc condcodes that aren't natively supported
have been expanded (thus exposing opportunities for optimisation by
performing logical negation and swapping truev/falsev). I've opted to
split out work that addresses these into follow-on patches (especially
as zicond is still 'experimental').
matchSetCC is a straight-forward translation from the version in
RISCVISelDAGToDAG. Ideally, in the future it can be converted to a
helper shared between both files.
Differential Revision: https://reviews.llvm.org/D155083
This is similar to D153632, but for mul nodes instead of add/sub. They get
recognised in LowerMUL in order to detect the mul(ext, ext), in a way that will
work for i64 nodes as well as i16/i32. This extends it to look for
mul(subvector_extract(ext(x), 0), subvector_extract(ext(y), 0)), generating a
subvector_extract(mull(x,y)) if it matches.
Differential Revision: https://reviews.llvm.org/D154063
(shl (zext to iXLenVec), C) is a possible pattern in auto-vectorized code for
indexed loads/stores. But extending to iXLen might be too aggressive, RVV
indexed load/store instructions zero extend their indexed operand to XLEN.
The patch tries to narrow the type of the zero extension. It's benefit to
decrease register pressure.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D154687
Previously we remove a pattern like:
%reg = and32ri %in_reg, 5
... // EFLAGS not changed.
%src_reg = subreg_to_reg 0, %reg, %subreg.sub_index
test64rr %src_reg, %src_reg, implicit-def $eflags
We can remove test64rr since it has same functionality as and subreg_to_reg avoid the opt in previous code, so we handle this case specially.
And this case is also can be opted for the same reason, like:
%reg = and32ri %in_reg, 5
... // EFLAGS not changed.
%src_reg = copy %reg.sub_16bit:gr32
test16rr %src_reg, %src_reg, implicit-def $eflags
The COPY from gr32 to gr16 prevent the opt in previous code too, just handle it specially as what we did for test64rr.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D154193
The previous heuristic rejected a PHI if one of its user was an unbreakable PHI, no matter what the other users were.
This worked well in most cases, but there's one case in rocRAND where
it doesn't work. In that case, a PHI node has 2 PHI users where one is
breakable but not the other. When that PHI node isn't broken performance falls by 35%.
Relaxing the restriction to "require that half of the PHI node users are breakable" fixes the issue, and seems like a sensible change.
Solves SWDEV-409648, SWDEV-398393
Reviewed By: #amdgpu, arsenm
Differential Revision: https://reviews.llvm.org/D155184
Followup to D101178 - peephole optimization that converts a
load address instruction and a consuming load/store into just the
load/store when its safe to do so.
eg: converts the 2 instruction code sequence
la 4, i[TD](2)
stw 3, 0(4)
to
stw 3, i[TD](2)
Differential Revision: https://reviews.llvm.org/D101470
Do the LDS frame calculation once, in the IR pass, instead of repeating the work in the backend.
Prior to this patch:
The IR lowering pass sets up a per-kernel LDS frame and annotates the variables with absolute_symbol
metadata so that the assembler can build lookup tables out of it. There is a fragile association between
kernel functions and named structs which is used to recompute the frame layout in the backend, with
fatal_errors catching inconsistencies in the second calculation.
After this patch:
The IR lowering pass additionally sets a frame size attribute on kernels. The backend uses the same
absolute_symbol metadata that the assembler uses to place objects within that frame size.
Deleted the now dead allocation code from the backend. Left for a later cleanup:
- enabling lowering for anonymous functions
- removing the elide-module-lds attribute (test churn, it's not used by llc any more)
- adjusting the dynamic alignment check to not use symbol names
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D155190
According to the spec, Zce is an alias for Zca, Zcb, Zcmp, and Zcmt.
If F is enabled on RV32 it also includes Zcf.
This patch adds the Zce and the implication rule which unfortunately
requires custom handling for adding Zcf.
I've also made all the Zc* extensions imply Zca.
I've also added an error for Zcf without RV32.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D153742
This adds the IGLP strategy for single-wave gemms. The SchedGroup pipeline is laid out in multiple phases, with each phase corresponding to a distinct pattern present in gemm kernels. The resilience of the optimization is dependent upon IR (as seen by pre-RA scheduling) continuing to have these patterns (as defined by instruction class and dependencies) in their current relative ordering.
The kernels of interest have these specific phases:
NT: 1, 2a, 2c
NN: 1, 2a, 2b
TT: 1, 2b, 2c
TN: 1, 2b
The general approach taken was to have a long SchedGroup pipeline. In this way the scheduler will have less capability of doing the wrong thing. In order to resolve the challenge of correctly fitting these long pipelines, we leverage the rules infrastructure to help the solver.
Differential Revision: https://reviews.llvm.org/D149773
Change-Id: I1a35962a95b4bdf740602b8f110d3297c6fb9d96
This patch adds pseudos and SDNode patterns for vbrev.v, vrev8.v, vclz.v,
vctz.v and vcpop.v.
I've only added them for integer element types so far since we're lacking tests
for floats.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D155216
GCC and existing codebases allow the use of integral values to be used
with this constraint. A recent change D133914 in this area started causing asserts.
Removing the assert is enough as the rest of the code works fine.
rdar://109675485
Differential Revision: https://reviews.llvm.org/D155023
This time without the extra `->dump()`
A recent addition to the device libs, `__ockl_dm_trim`, caused a series of
failures at O0 due to a i64 ballot intrinsic being inlined into a wave32 function.
The quick fix for this is to support codegen for this rare case.
A proper long-term fix for this type of issue is still being discussed.
Fixes SWDEV-408929, SWDEV-408957, SWDEV-409885, SWDEV-410193
Reviewed By: #amdgpu, arsenm
Differential Revision: https://reviews.llvm.org/D155050
