Basically https://github.com/llvm/llvm-project/pull/168506 but for
riscv, so to be clear the hard work here is @heiher 's. I figured we may
as well get some extra eyeballs on this from riscv too.
Previously the riscv backend could not handle `musttail` calls with more
arguments than fit in registers, or any explicit `byval` or `sret`
parameters/return values. Those have now been implemented.
This is part of my push to get more LLVM backends to support `byval` and
`sret` parameters so that rust can stabilize guaranteed tail call
support. See also:
- https://github.com/llvm/llvm-project/pull/168956
- https://github.com/rust-lang/rust/issues/148748
---------
Co-authored-by: WANG Rui <wangrui@loongson.cn>
This patch prioritizes lowering to `stnp` over `st2` store instructions
marked !nontemporal.
From performance perspective, we should conservatively prioritize STNP
lowering for non-temporal stores, because currently NT stores requires
explicit usage of `__builtin_nontemporal_store()` intrinsic, so I think
its reasonable to assume the developer explicitly intends to optimize
D-cache usage of some hot non-temporal execution. He can rollback if it
doesnt help.
The cost here is it adds a few instructions for code size (thus we
predicate when not optimizing for code size), few extra fast
instructions to execute, few extra short dep chains - should be commonly
handled by OOO execution, I-cache alignment effects, few extra
registers. In the future we can may be able to approximate a cost model
to select by.
The patch implements an AArch64 specific static function to model what
NT stores are directly legal on the backend currently, and
`AArch64TargetLowering::lowerInterleavedStore` to conditionally skip st2
lowering.
The idea behind this PR is to propose a type that we can deploy
gradually to add some guardrails and enforce invariants in the SPIRV
backend.
The PR has 3 commits:
* A first commit where the `SPIRVTypeInst` type is proposed. It's just a
wrapper around `MachineInstr` that adds an assert to check that a
`SPIRVTypeInst` defines a register with the type register class.
* A second commit that shows how the migration could look like for a
single function.
* A third commit that motivates why: we have a `SPIRVType *TypeInst`
that never defines a type in a function whose intention looks very
confusing.
Rotations that implement reverse operations, for example:
```c
uint64x2_t foo(uint64x2_t r) {
return (r >> 32) | (r << 32);
}
```
Are currently lowered as XAR (when available):
```gas
foo:
movi v1.2d, #0000000000000000
xar v0.2d, v0.2d, v1.2d, #32
ret
```
This is suboptimal as REV* instructions typically have higher throughput
than XAR and do not require the zero operand.
This patch combines half-rotations to Neon or SVE REVs so that they're
no longer selected as XAR.
We didn't explicitly add spilled registers as liveins for CM_PUSH as
well. The liveins of the basic block will be updated by updateLiveness
in PrologEpilogInserter.
Similar to #180706, the masked off lanes in vp.reverse are poison so can
be replaced with anything. Because of this, we should be able to fold a
masked vp.reverse(vp.load) into a vp.strided.load stride=-1 even when
the mask isn't all ones.
We have combines for vp.reverse(vp.load) -> vp.strided.load stride=-1
and vp.store(vp.reverse) -> vp.strided.store stride=-1.
If the load or store is masked, the mask needs to be also a vp.reverse
with the same EVL. However we also have the requirement that the mask's
vp.reverse is unmasked (has an all-ones mask).
vp.reverse's mask only sets masked off lanes to poison, and doesn't
affect the permutation of elements. So given those lanes are poison, I
believe the combine is valid for any mask, not just all ones.
This is split off from another patch I plan on posting to generalize
those combines to vector.splice+vector.reverse patterns, as part of
#172961
MIPS16 cannot handle constant pools created by CTTZ table lookup
expansion. This causes "Cannot select" errors when trying to select
MipsISD::Lo nodes for constant pool addresses.
Modify the table lookup conditions to check ConstantPool operation
status, and only set ConstantPool to Custom in non-MIPS16 mode in MIPS
backend.
This ensures MIPS16 uses the ISD::CTPOP instead of attempting
unsupported constant pool operations.
Fix#61055.
Unlike the base add/sub opcodes which will just overflow, these will
produce incorrect results, because the carry operates on the full
64-bits. Trying to use these with i32 operands on PPC64 should result in
a selection failure instead of a silent miscompile, like the one seen in
https://github.com/llvm/llvm-project/pull/178979.
Asynchronous operations are memory transfers (usually between the global
memory and LDS) that are completed independently at an unspecified
scope. A thread that requests one or more asynchronous transfers can use
async marks to track their completion. The thread waits for each mark to
be completed, which indicates that requests initiated in program order
before this mark have also completed.
For now, we implement asyncmark/wait operations on pre-GFX12
architectures that support "LDS DMA" operations. Future work will extend
support to GFX12Plus architectures that support "true" async operations.
This is part of a stack split out from #173259
- #180467
- #180466
Co-authored-by: Ryan Mitchell ryan.mitchell@amd.com
Fixes: SWDEV-521121
Similar to other portings created by @aidenboom154. No specific test
coverage as there are no MIR->MIR tests that exercise this pass. Going
with other naming conventions, I renamed WaitInsert to
X86InsertX87WaitLegacy
The existing "LDS DMA" builtins/intrinsics copy data from global/buffer
pointer to LDS. These are now augmented with their ".async" version,
where the compiler does not automatically track completion. The
completion is now tracked using explicit mark/wait intrinsics, which
must be inserted by the user. This makes it possible to write programs
with efficient waits in software pipeline loops. The program can now
wait for only the oldest outstanding operations to finish, while
launching more operations for later use.
This change only contains the new names of the builtins/intrinsics,
which continue to behave exactly like their non-async counterparts. A
later change will implement the actual mark/wait semantics in
SIInsertWaitcnts.
This is part of a stack split out from #173259:
- #180467
- #180466
Fixes: SWDEV-521121
On RISC-V, the interrupt attribute relates only to the prolog and epilog
of the attributed function (and has specific restrictions on the
function's signature). It does not change how that function calls other
functions, and when outlining, the outlined function must not have this
attribute.
This adds a target-independent hook to TTI so targets can choose which
attributes to propagate (by default all are propagated).
Fixes#149969
`updateAsyncLDSAddress`, introduces by
https://github.com/llvm/llvm-project/pull/176816, previously only
handled async loads , where the LDS address is in the `vdst` operand.
Therefore Async stores produced a nullptr dereference since the LDS
address is in `vdata` for those instructions.
---------
Co-authored-by: Jay Foad <jay.foad@gmail.com>
These are 3 variations of the same operation with a different operand
tied to the destination register. We need to pick the one that
minimizes the number of mvs.
To do this we take the approach used by AArch64 to select between
BIT, BIF, and BSL which the same operations. We define a pseudo
with no tied constraint and expand it after register allocation based
on where the destination register ended up. If the destination
register is none of the operands, we'll insert a mv.
I've replaced RISCVISD::MVM with RISCVISD::MERGE and updated the operand
order accordingly. I find the MERGE name easier to read so I've made it
the canonical name.
Ideally we could use commuteInstructionImpl and the
TwoAddressInstructionPass
to select the opcode before register allocation. That only works if
you can commute exactly 2 operands and maybe change the opcode in the MI
representation of any of the forms to get to the either of the other 2
forms.
That is not possible. We'd need to define 3 more pseudoinstructions
with different permutations.
With the current approach it might be possible that we insert a mv
not because all of the operand registers we needed by later
instructions,
but because the register allocator needed to put the result in a
different register. It's possible a different allocation for other
instructions might have avoided the mv.
I wrote the patch based on the AArch64, but the tests were generated
by AI.
Currently, legalization fails for integer (lower than 8 bit) comparison
with extensions, for example, SPV_INTEL_int4. This PR extends integers
for supported extensions.
---------
Co-authored-by: Michal Paszkowski <michal@michalpaszkowski.com>
My initial goal was to name bits {19-15} as either rs1, vs1, or imm as
appropriate.
The VCIX instructions effectively have 3 properties that are combined:
the type of bits 19-15 (determined by funct3), how many vector registers
are sources (determined by funct6), and whether vd is a written or not
(determined by vm).
I found the VCIXInfo class very hard to read and understand. This patch
breaks it up into simpler steps and moves some properties to flags in
the existing VCIXType classs. A new VCIXRs1Info class is added to
contain the properties for bits 19-15. The VCIXInfo now combines the
properties from these 2 classes and the HaveOutputDst flag to create the
various things needed to build the final instruction. To reduce the
number of template arguments, the VCIXInfo is passed all the way down to
the base class.
Much of the refactoring in this patch was accomplished with AI. I have
looked at the diffs in the output tablegen files to verify it works as
expected. RISCVGenInstrInfo.inc only changes line number comments.
RISCVGenDisassembler.inc doesn't change at all as expected.
RISCVGenMCCodeEmitter.inc changes due to the different field names and
orders in the record.
optimizeCondBranch is looking for immediates that are 1 apart to rewrite
the branch by sharing a constant. BSETI x0, 11 can be used to produce
2048 which is one more than the largest positive constant produced by
addi.
This change adds initial support to libObject for reading compact
imports and support for writing compact imports in the linker.
There is minimal testing here since to tools like lllvm-readobj, and
obj2yaml don't currently report compact imports any differently.
See https://github.com/WebAssembly/compact-import-section
Previous commit message:
>Previous commit message:
>
>> Original commit message:
>>
>>>When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>>>
>>>When no SM or PTX version is specified, we now use PTX 3.2 (the
minimum for the default SM 3.0) instead of PTX 6.0.
>>
>>The following commits should fix the failures that arose when I
previously tried to land this commit:
>>
>>
>>9fc5fd0ad6
should address the llvm-nvptx*-nvidia-* build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
>>
>>
>>600514a637
should address the MLIR failures
>
>The previous commit was reverted with
d23cb79ba4
because the
[mlir-nvidia](https://lab.llvm.org/buildbot/#/builders/138/builds/24797)
and
[mlir-nvidia-gcc7](https://lab.llvm.org/buildbot/#/builders/116/builds/23929)
Buildbots were failing.
>
>Those tests failed because MLIR's default SM was 5.0, which caused
NVPTX
to target PTX ISA v4.0, which did not support the intrinsics used in the
failing tests.
>
>243f011577
should address this by bumping MLIR's default SM to 7.5. Now, using
MLIR's new default SM, NVPTX
targets the PTX ISA v6.3, which supports the intrinsics used in the
failing tests.
---
The previous commit was reverted with
e9b578a4d77025e18318efedd0f3f3764338d859
[because](https://github.com/llvm/llvm-project/pull/179304#issuecomment-3856301333)
the clang driver set the default PTX ISA version to v4.2 when no CUDA
installation is found. However, given our patch, we should not set a
default; instead, let the LLVM backend select the appropriate PTX ISA
version for the target SM.
Sub-reductions can be implemented in two ways:
(1) negate the operand in the vector loop (the default way).
(2) subtract the reduced value from the init value in the middle block.
Note that both ways keep the reduction itself as an 'add' reduction,
which is necessary because only llvm.vector.partial.reduce.add exists.
The ISD nodes for partial reductions don't support folding the
sub/negation into its operands because the following is not a valid
transformation:
```
sub(0, mul(ext(a), ext(b)))
-> mul(ext(a), ext(sub(0, b)))
```
It can therefore be better to choose option (2) such that the partial
reduction is always positive (starting at '0') and to do a final
subtract in the middle block.
For AArch64 there are no dot-product instructions that can
do a `partial.reduce.sub(acc, mul(ext(a), ext(b)))` operation.
I'm not sure if such instructions exist for other targets.
(If so then we may want to make this decision a target option)
This PR also increases the AArch64 cost of a partial sub-reduction
when this exists in an 'add-sub' reduction chain.
Fixes https://github.com/llvm/llvm-project/issues/178703
Adds rules for G_ATOMICRMW_{MAX, MIN, UMAX, UMIN, UINC_WRAP, UDEC_WRAP}.
Each of these generic opcode are supported for S32 and S64 types
on flat, global and local address spaces.
This patch adds support in Clang for the PRFM IR instruction, by adding
the following builtin:
void __pldir(void const *addr);
This builtin is described in the following ACLE proposal:
https://github.com/ARM-software/acle/pull/406
Make it possible to use `s_alloc_vgpr` at the IR level. This is a huge
footgun and use for anything other than compiler internal purposes is
heavily discouraged. The calling code must make sure that it does not
allocate fewer VGPRs than necessary - the intrinsic is NOT a request to
the backend to limit the number of VGPRs it uses (in essence it's not so
different from what we do with the dynamic VGPR flags of the
`amdgcn.cs.chain` intrinsic, it just makes it possible to use this
functionality in other scenarios).
When comparing multi-word integers with Zicond, we generate:
(or (czero_eqz (lo1 < lo2), (hi1 == hi2)),
(czero_nez (hi1 < hi2), (hi1 == hi2)))
The czero_nez is redundant because when hi1 == hi2 is true, hi1 < hi2 is
already 0. This patch adds a DAG combine to recognize:
czero_nez (setcc X, Y, CC), (setcc X, Y, eq) -> (setcc X, Y, CC)
when CC is a strict inequality (lt, gt, ult, ugt).
This saves one instruction in 128-bit comparisons on RV64 with Zicond.
Note the czero_nez becomes a czero.eqz in the final assembly because the
seteq is replaced by an xor that produces 0 when the values are equal.
Part of #179584
Assisted-by: claude
This patch introduces `get(T)` and `set(T, Val)` functions for Waitcnt
and removes getCounterRef() and getWait(). For this to work we also need
to move InstrCounterType to AMDGPUBaseInfo.h.
Please note that the member variables are still public to keep this
patch small.
They will be replaced in the follow-up patch.
This adds atomicrmw `uinc_wrap` and `udec_wrap` operations support for
SPIR-V. Since SPIR-V doesn't provide dedicated instructions for those
two operations, we have to use the `AtomicExpand` pass to expand the
operations into CAS forms.
Closes#177204.
