This changes the register class to GPRPair and adds the destination
register as a source with a tied operand constraint.
Parsing for the paired register is done with a custom parser that
checks for even register and converts it to its pair version. A
bit of care needs to be taken so that we only parse as a pair register
based on which instruction we're parsing and the mode in the subtarget.
This allows amocas.w to be parsed correcty in both modes.
I've added a FIXME to note that we should be creating pair registers
for Zdinx on RV32 to match the instructions CodeGen generates.
This patch changes the following intrinsic
```svst1uwq[_{d}] replaced by svst1wq[_{d}]
svst1uwq_vnum[_{d}] replaced by svst1wq_vnum[_{d}]
svst1udq[_{d}] replaced by svst1dq[_{d}]
svst1udq_vnum[_{d}] replaced by svst1dq_vnum[_{d}]
```
Drops 'u' from the quadword stores because it is simply truncating the
quadwords to 32 bits
```
svextq_lane[_{d}] replaced by svextq[_{d}]
```
EXTQ follows the previous defined EXT intrinsics
```
svdot[_{d}_{2}_{3}] replaced by svdot[_{d}_{2}]
```
Introduced with the latest SME2 ACLE change
[1]https://github.com/ARM-software/acle/pull/257
When i128 is a legal type, SelectionDAG now attempts to use
SRL_PARTS etc. with type i128, which is not implemented. Fix
by marking those as Expand, just like we do for i64.
Fixes https://github.com/llvm/llvm-project/issues/77132
As not all fake instructions have their real counterparts implemented
yet, we specify no AssemblerPredicate for UseFakeTrue16Insts to allow
both fake and real True16 instructions in assembler and disassembler
tests in the -mattr=+real-true16 mode during the transition period.
Source DPP and desitnation VOPDstOperand_t16 operands are still not
supported and will be addressed separately.
The intrinsics get_fpenv, set_fpenv and reset_fpenv in this change are
implemented as calls to math library functions. Target specific lowering
will be implemented later on.
This patch fixes WebAssembly's FastISel pass to correctly consider
signext/zeroext parameter flags at function declaration.
Previously, the flags at call sites were only considered during code
generation, which caused an interesting bug report #63388 .
This is problematic especially because in WebAssembly's ABI, either
signext or zeroext can be tagged to a function argument, and it must be
correctly reflected in the generated code. Unit test
https://github.com/llvm/llvm-project/blob/main/llvm/test/CodeGen/WebAssembly/signext-zeroext.ll
shows that `i8 zeroext %t` and `i8 signext %t`'s code gen are different.
* Add IRTranslate tests for ADD, SUB, AND, OR, and XOR with scalable
vector types to show that they work as expected.
* Legalize G_ADD, G_SUB, G_AND, G_OR, and G_XOR of scalable vector
type for the RISC-V vector extension.
Move classes out of `let Predicates` scopes. The instantiation of the
class should be responsible for providing the Predicates.
Put the RV64 pseudoinstructions and patterns next to the RV32 version of
the same category. The categories are AMOs, pseudo AMOs, and compare
exchange. The main reason for this commit is that the compare exchange
patterns need to be disabled when Zacas is enabled so we can directly
select Zacas instructions with isel patterns. This necessitates compare
exchange having a different `let Predicates=` from the others anyway.
Similar to #76550, but for `ISD::AVGCEILU`.
Specifically, this patch aims to use `vaaddu` with rounding mode rnu
(i.e `vxrm[1:0] = 0b00`) for `ISD::AVGCEILU`.
### Source code
```
define <vscale x 8 x i8> @vaaddu_vv_nxv8i8_ceil(<vscale x 8 x i8> %x, <vscale x 8 x i8> %y) {
%xzv = zext <vscale x 8 x i8> %x to <vscale x 8 x i16>
%yzv = zext <vscale x 8 x i8> %y to <vscale x 8 x i16>
%add = add nuw nsw <vscale x 8 x i16> %xzv, %yzv
%one = insertelement <vscale x 8 x i16> poison, i16 1, i32 0
%splat = shufflevector <vscale x 8 x i16> %one, <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer
%add1 = add nuw nsw <vscale x 8 x i16> %add, %splat
%div = lshr <vscale x 8 x i16> %add1, %splat
%ret = trunc <vscale x 8 x i16> %div to <vscale x 8 x i8>
ret <vscale x 8 x i8> %ret
}
```
### Before this patch
```
vaaddu_vv_nxv8i8_ceil:
vsetvli a0, zero, e8, m1, ta, ma
vwaddu.vv v10, v8, v9
vsetvli zero, zero, e16, m2, ta, ma
vadd.vi v10, v10, 1
vsetvli zero, zero, e8, m1, ta, ma
vnsrl.wi v8, v10, 1
ret
```
### After this patch
```
vaaddu_vv_nxv8i8_ceil:
vsetvli a0, zero, e8, m1, ta, ma
csrwi vxrm, 0
vaaddu.vv v8, v8, v9
ret
```
`llvm.trap` is lowered to `PPC::TRAP` and `PPC::TRAP` is set as
terminator. Verifier complains about terminator should not lie in the
middle of an MBB. See #77095.
Fix it by removing `isTerminator` and `isBarrier` and then set `isTrap`
which was introduced by https://reviews.llvm.org/D48836# and is being
used by X86 and AArch64.
`PPC::TRAP` is not a hardware memory barrier and `llvm.trap` doesn't
indicate a memory barrier either.
This patch adds an intrinsic for setmaxnreg PTX instruction.
* PTX Doc link for this instruction:
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-setmaxnreg
* The i32 argument, an immediate value, specifies the actual
absolute register count for the instruction.
* The `setmaxnreg` instruction is available in SM90a.
So, this patch adds 'hasSM90a' predicate to use in
the NVPTX backend.
* lit tests are added to verify the lowering of the intrinsic.
* Verifier logic (and tests) are added to test the register
count range and divisibility-by-8 requirements.
Signed-off-by: Durgadoss R <durgadossr@nvidia.com>
Regarding
```
.option norelax
j label
.option relax
// relaxable instructions
// For assembly input, RISCVAsmParser::ParseInstruction will set ForceRelocs (https://reviews.llvm.org/D46423).
// For direct object emission, ForceRelocs is not set after https://github.com/llvm/llvm-project/pull/73721
label:
```
The J instruction needs a relocation to ensure the target is correct
after linker relaxation. This is related a limitation in the assembler:
RISCVAsmBackend::shouldForceRelocation decides upfront whether a
relocation is needed, instead of checking more information (whether
there are relaxable fragments in between).
Despite the limitation, `j label` produces a relocation in direct object
emission mode, but was broken by #73721 due to the shouldForceRelocation
limitation.
Add a workaround to RISCVTargetELFStreamer to emulate the previous
behavior.
Link: https://github.com/ClangBuiltLinux/linux/issues/1965
X86 doesn't have a BITREVERSE instruction, so if we're working with a casted boolean vector, we're better off shuffling the vector instead if we have PSHUFB (SSSE3 or later)
Fixes#77459
-Rename to GPRPair.
-Rename registers to be named like X10_X11 instead of X10_PD. Except X0
which is now X0_Pair since it is not paired with X1.
-Use unknown size and offset for the subreg indices. This might
be a functional change, but does not affect any lit tests.
This is in preparation for moving the run of AMDGPUAttributor earlier.
Currently it infers the lack of the corresponding intrinsic calls,
so if we introduce new ones we need to remove the attribute from any
possible transitive callers. This is more conservative than necessary,
we could try to identify specific subgraphs where LDS globals are not
used.
Other options include teaching the attributor to avoid adding it in
cases
where the lowering may choose the table, but this seems more complex.
Alternatively could add a second run which doesn't seem worth it.
Depends #71349
Add the vop3p op_sel fields in getInsVOP3P instead of getInsVOP3Base.
Also start using defvar for some of the intermediate fields. let
overrides of all the visible fields are really difficult to follow.
This is the logical equivalent for #76710 for APInt and uses the same
naming scheme.
Converted existing users through:
`git grep -l "cast<ConstantSDNode>\(.*\).*getAPIntValueValue" | xargs
sed -E -i
's/cast<ConstantSDNode>\((.*)\)->getAPIntValue/\1->getAsAPIntVal/'`
This patch fixes the crash issue in the test:
CodeGen/LoongArch/can-not-realign-stack.ll
Register allocator may spill virtual registers to the stack, which
introduces stack alignment requirements (when the size of spilled
registers exceeds the default alignment size of the stack). If a
function does not have stack alignment requirements before register
allocation, registers used for stack alignment will not be preserved.
Therefore, we should implement `canRealignStack()` to inform the
register allocator whether it is allowed to perform stack realignment
operations.
This follows on from #76708, allowing
`cast<ConstantSDNode>(N)->getZExtValue()` to be replaced with just
`N->getAsZextVal();`
Introduced via `git grep -l "cast<ConstantSDNode>\(.*\).*getZExtValue" |
xargs sed -E -i
's/cast<ConstantSDNode>\((.*)\)->getZExtValue/\1->getAsZExtVal/'` and
then using `git clang-format` on the result.
We have added a new pass that looks for loops such as the following:
```
while (i != max_len)
if (a[i] != b[i])
break;
... use index i ...
```
Although similar to a memcmp, this is slightly different because instead
of returning the difference between the values of the first non-matching
pair of bytes, it returns the index of the first mismatch. As such, we
are not able to lower this to a memcmp call.
The new pass can now spot such idioms and transform them into a
specialised predicated loop that gives a significant performance
improvement for AArch64. It is intended as a stop-gap solution until
this can be handled by the vectoriser, which doesn't currently deal with
early exits.
This specialised loop makes use of a generic intrinsic that counts the
trailing zero elements in a predicate vector. This was added in
https://reviews.llvm.org/D159283 and for SVE we end up with brkb & incp
instructions.
Although we have added this pass only for AArch64, it was written in a
generic way so that in theory it could be used by other targets.
Currently the pass requires scalable vector support and needs to know
the minimum page size for the target, however it's possible to make it
work for fixed-width vectors too. Also, the llvm.experimental.cttz.elts
intrinsic used by the pass has generic lowering, but can be made
efficient for targets with instructions similar to SVE's brkb, cntp and
incp.
Original version of patch was posted on Phabricator:
https://reviews.llvm.org/D158291
Patch co-authored by Kerry McLaughlin (@kmclaughlin-arm) and David
Sherwood (@david-arm)
See the original discussion on Discourse:
https://discourse.llvm.org/t/aarch64-target-specific-loop-idiom-recognition/72383
In practice maybeAtomic = 0 is used to prevent SIMemoryLegalizer from
interfering with instructions that are mayLoad or mayStore but lack
MachineMemOperands. These instructions should be the exception not the
rule, so this patch sets maybeAtomic = 1 by default and only overrides
it to 0 where necessary.
GFX12 has separate wait instructions per counter e.g. S_WAIT_LOADCNT.
S_WAITCNT still exists but is deprecated and codegen should stop using
it. S_WAITCNT_* (e.g. S_WAITCNT_VSCNT) are removed.
This patch adds/removes MC layer support for these instructions.
The library implementation is just a wrapper around a call to the
intrinsic, but loses metadata. Swap out the call site to the intrinsic
so that the lowering can see the !fpmath metadata and fast math flags.
Since d56e0d07cc5ee8e334fd1ad403eef0b1a771384f, clang started placing
!fpmath on OpenCL library sqrt calls. Also don't bother emitting
native_sqrt anymore, it's just another wrapper around llvm.sqrt.
1, Follow RISCV 1df5ea29 to support generates relocs for .uleb128 which
can not be folded. Unlike RISCV, the located content of LoongArch should
be zero. LoongArch fixup uleb128 value by in-place addition and
subtraction reloc types named R_LARCH_{ADD,SUB}_ULEB128. The located
content can affect the result and R_LARCH_ADD_ULEB128 has enough info to
represent the first symbol value, so it needs to be set to zero.
2, Force relocs if sym is not in section so that it can emit relocs for
external symbol.
Fixes:
https://github.com/llvm/llvm-project/pull/72960#issuecomment-1866844679
This patch aims to use `vaaddu` with rounding mode rdn (i.e `vxrm[1:0] =
0b10`) for `ISD::AVGFLOORU`.
### Source code
```
define <8 x i8> @vaaddu_auto(ptr %x, ptr %y, ptr %z) {
%xv = load <8 x i8>, ptr %x, align 2
%yv = load <8 x i8>, ptr %y, align 2
%xzv = zext <8 x i8> %xv to <8 x i16>
%yzv = zext <8 x i8> %yv to <8 x i16>
%add = add nuw nsw <8 x i16> %xzv, %yzv
%div = lshr <8 x i16> %add, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
%ret = trunc <8 x i16> %div to <8 x i8>
ret <8 x i8> %ret
}
```
### Before this patch
```
vaaddu_auto:
vsetivli zero, 8, e8, mf2, ta, ma
vle8.v v8, (a0)
vle8.v v9, (a1)
vwaddu.vv v10, v8, v9
vnsrl.wi v8, v10, 1
ret
```
### After this patch
```
vaaddu_auto:
vsetivli zero, 8, e8, mf2, ta, ma
vle8.v v8, (a0)
vle8.v v9, (a1)
csrwi vxrm, 2
vaaddu.vv v8, v8, v9
ret
```
### Note on signed averaging addition
Based on the rvv spec, there is also a variant for signed averaging
addition called `vaadd`.
But AFAIU, no matter in which rounding mode, we cannot achieve the
semantic of signed averaging addition through `vaadd`.
Thus this patch only introduces `vaaddu`.
This makes it possible to pass null_frag to the MAIInst multiclass.
null_frag does not work as you may hope if used as the input to a
PatFrag, which is what happens when it's passed through to *MAIFrag.
Avoid this by checking for null_frag. It might be possible to hack up
tablegen to allow consuming PatFrag inputs.
Targets affected:
- NVPTX and BPF: set to 64 bits.
- ARC, Lanai, and MSP430: set to 0 (they don't implement atomics).
Those which didn't yet add AtomicExpandPass to their pass pipeline now
do so.
This will result in larger atomic operations getting expanded to
`__atomic_*` libcalls via AtomicExpandPass. On all these targets, this
now matches what Clang already does in the frontend.
The only targets which do not configure AtomicExpandPass now are:
- DirectX and SPIRV: they aren't normal backends.
- AVR: a single-cpu architecture with no privileged/user divide, which
could implement all atomics by disabling/enabling interrupts, regardless
of size/alignment. Will be addressed by future work.
sifive-p450 supports a very restricted version of the short forward
branch optimization from the sifive-7-series.
For sifive-p450, a branch over a single c.mv can be macrofused as a
conditional move operation. Due to encoding restrictions on c.mv, we
can't conditionally move from X0. That would require c.li instead.
`nvvm_fabs_f`
`nvvm_fabs_ftz_f`
Unfortunately, llvm fabs is not equivalent to these intrinsics since
llvm fabs is defined to only set the sign bit to zero while these can
also flush subnormal inputs and modify NaNs.
`nvvm_round_d`
`nvvm_round_f`
`nvvm_round_ftz_f`
llvm.nvvm.round uses RNI, while llvm.round codegens to RZI. LLVM defines
llvm.round to use the same rounding as libm
`round[f]()`, which is not necessary the same as how we define
llvm.nvvm.round.
`nvvm_sqrt_rn_f`
`nvvm_sqrt_rn_ftz_f`
sqrt may be lowered to a less precise version of sqrt, such as
sqrt.approx in NVPTX depending on factors such as the value of
-nvptx-prec-sqrtf32. These intrinsics should always become the
corresponding NVPTX instructions.
`nvvm_add_rn_d`
`nvvm_add_rn_f`
`nvvm_add_rn_ftz_f`
`nvvm_mul_rn_d`
`nvvm_mul_rn_f`
`nvvm_mul_rn_ftz_f`
These nvvm intrinsics have an explicitly specified rounding mode (.rn).
They should always be lowered to a PTX instruction with the same
explicit rounding mode. Converting to fmul and fadd instructions result
in the PTX instructions without rounding modes specified. This can cause
issue because:
> An add [or mul] instruction with no rounding modifier defaults to
round-to-nearest-even and may be optimized aggressively by the code
optimizer. In particular, mul/add sequences with no rounding modifiers
may be optimized to use fused-multiply-add instructions on the target
device.
`nvvm_div_rn_f`
`nvvm_div_rn_ftz_f`
`nvvm_rcp_rn_f`
`nvvm_rcp_rn_ftz_f`
fdiv may be lowered to a less precise version of div, such as div.full
in NVPTX depending on factors such as the value of -nvptx-prec-divf32.
These intrinsics should always become the corresponding NVPTX
instructions.
Since their values are small enough ([-1, 65535] & [0, 65535],
respectively) to fit into signed 32 bits, any sext (or downcasting +
sext) will be redundnat. Hence marking them as SignExtendingOpW.
We can already support selection of these through imported patterns, we were
just missing the legalizer rule to allow these to be formed.
Nano size benefit overall.
After d6bb96e677759375b2bea00115918b2cb6552f5b, calling
getPrefLoopAlignment without passing in a pointer to a MachineLoop
causes a segmentation fault. This conflicts with the API in
TargetLoweringBase where the default MachineLoop pointer passed is
nullptr. This patch fixes this by checking if the pointer points to
something before enabling the optional functionality.
