Address space information may be encoded anywhere along the use-def
chain. Take advantage of this by traversing the chain until we find a
non-generic addrspace.
When [lowering return
values](99a10f1fe8/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp (L3422))
from LLVM IR to SelectionDAG, we check that [the number of values
`SelectionDAG` tells us to return is equal to the number of values that
`ComputePTXValueVTs()` tells us to
return](99a10f1fe8/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp (L3441)).
However, this check can fail on valid IR. For example:
```
define <6 x half> @foo() {
ret <6 x half> zeroinitializer
}
```
`ComputePTXValueVTs()` tells us to return ***3*** `v2f16` values, while
`SelectionDAG` tells us to return ***6*** `f16` values. Thus, the
compiler will crash.
`ComputePTXValueVTs()` [supports all `half` element vectors with an even
number of
elements](99a10f1fe8/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp (L213)).
Whereas `SelectionDAG` [only supports power-of-2 sized
vectors](4e078e3797/llvm/lib/CodeGen/TargetLoweringBase.cpp (L1580)).
This is the root of the discrepancy.
Assuming that the developers who added the code to
`ComputePTXValueVTs()` overlooked this, I've restricted
`ComputePTXValueVTs()` to compute the same number of return values as
`SelectionDAG`, instead of extending `SelectionDAG` to support
non-power-of-2 sized vectors.
Fix bug introduced in #105730
The bug is in how the batch RAUW is implemented. If we have
```
%0 = mov %src
%1 = mov %0
use %0
use %1
```
The use of `%1` is rewritten to `%0`, not `%src`. This PR just looks for
a replacement when it maps to the src register, which should
transitively propagate the replacements.
Intrinsics are defined with a bfloat type as of commit
250f2bb2c6a9c288faeb821585e9394697c561d8, not i16 and i32 storage types.
As such declarations are no longer needed once the correct types are
used.
Add custom lowering for `BR_JT` DAG nodes to the `brx.idx` PTX
instruction ([PTX ISA 9.7.13.4. Control Flow Instructions: brx.idx]
(https://docs.nvidia.com/cuda/parallel-thread-execution/#control-flow-instructions-brx-idx)).
Depending on the heuristics in DAG selection, `switch` statements may
now be lowered using `brx.idx`.
Note: this fixes the previous issue in #102400 by adding the isBarrier
attribute to BRX_END
Generate nuw GEPs for struct member accesses, as inbounds + non-negative
implies nuw.
Regression tests are updated using update scripts where possible, and by
find + replace where not.
This commit adds LLVM Intrinsics and NVPTX codegen support for
`fence.proxy.tensormap` with lit tests under fence-proxy-tensormap.ll.
Also, added Intrinsics documentation in NVPTXUsage.rst
---------
Co-authored-by: gonzalobg <65027571+gonzalobg@users.noreply.github.com>
`LowerDYNAMIC_STACKALLOC()` emits the `dynamic_stackalloc` chain operand
instead of the chain produced by the `NVPTXISD::DYNAMIC_STACKALLOC`. Fix
this behavior and don't produce an unnecessary `ISD::MERGE_VALUES`.
We want to use newer instructions if we are targeting sufficiently new
SM and PTX versions. If we cannot use those newer instructions, let LLVM
synthesize the sequence from more fundamental instructions.
So far, the IR-level lowering of llvm.memmove intrinsics generates loops
that copy each byte individually. This can be wasteful for targets that
provide wider memory access operations.
This patch makes the memmove lowering more similar to the lowering of
memcpy with unknown length.
TargetTransformInfo::getMemcpyLoopLoweringType() is queried for an
adequate type for the memory accesses, and if it is wider than a single
byte, the greatest multiple of the type's size that is less than or
equal to the length is copied with corresponding wide memory accesses. A
residual loop with byte-wise accesses (or a sequence of suitable memory
accesses in case the length is statically known) is introduced for the
remaining bytes.
For memmove, this construct is required in two variants: one for copying
forward and one for copying backwards, to handle overlapping memory
ranges. For the backwards case, the residual code still covers the bytes
at the end of the copied region and is therefore executed before the
wide main loop. This implementation choice is based on the assumption
that we are more likely to encounter memory ranges whose start aligns
with the access width than ones whose end does.
In microbenchmarks on gfx1030 (AMDGPU), this change yields speedups up
to 16x for memmoves with variable or large constant lengths.
Part of SWDEV-455845.
Followup to https://github.com/llvm/llvm-project/pull/98022 which broke
support for LLVM `unordered` atomic ordering.
This PR lowers `atomic unordered` to PTX volatile operations to preserve
atomicity, and adds a bunch of tests for this ordering to the NVPTX
backend (we had none).
This patch preserves `undef` SDNodes that are `volatile` qualified.
Previously, these nodes would be discarded. The motivation behind this
change is to adhere to the
[LangRef](https://llvm.org/docs/LangRef.html#volatile-memory-accesses),
even though that doc is mostly in terms of LLVM-IR, it seems reasonable
to imply that the volatile constraints also imply to SDNodes.
> Certain memory accesses, such as
[load](https://llvm.org/docs/LangRef.html#i-load)’s,
[store](https://llvm.org/docs/LangRef.html#i-store)’s, and
[llvm.memcpy](https://llvm.org/docs/LangRef.html#int-memcpy)’s may be
marked volatile. The optimizers must not change the number of volatile
operations or change their order of execution relative to other volatile
operations. The optimizers may change the order of volatile operations
relative to non-volatile operations. This is not Java’s “volatile” and
has no cross-thread synchronization behavior.
Source: https://llvm.org/docs/LangRef.html#volatile-memory-accesses
Summary:
The NVPTX backend optimizes the ABI for functions that are internal,
however, this is not legal for indirect call prototypes. Previously, we
would modify the ABI on an aggregate byval type passed to an indirect
call prototype, which would make PTXAS error. This patch just passes the
function as a nullptr to force strict ABI compliance without
modification in the helper function.
Fixes https://github.com/llvm/llvm-project/issues/100055
Just like for regular IR we need to treat SELECT as conditionally
blocking poison in SelectionDAG. So (unless the condition itself is
poison) the result is only poison if the selected true/false value is
poison.
Thus, when doing DAG combines that turn SELECT into arithmetic/logical
operations (e.g. AND/OR) we need to make sure that the new operations
aren't more poisonous. One way to do that is to use FREEZE to make
sure the operands aren't posion.
This patch aims at fixing the kind of miscompiles reported in
https://github.com/llvm/llvm-project/issues/84653
and
https://github.com/llvm/llvm-project/issues/85190
Solution is to make sure that we insert FREEZE, if needed to make
the fold sound, when using the foldBoolSelectToLogic and
foldVSelectToSignBitSplatMask DAG combines.
The NVPTX ISA states that an immOff must fit in a signed 32-bit integer
(https://docs.nvidia.com/cuda/parallel-thread-execution/#addresses-as-operands):
> `[reg+immOff]`
>
> a sum of register `reg` containing a byte address plus a constant
> integer byte offset (signed, 32-bit).
>
> `[var+immOff]`
>
> a sum of address of addressable variable `var` containing a byte
> address plus a constant integer byte offset (signed, 32-bit).
Currently we do not consider this constraint, meaning that in some edge
cases we generate invalid PTX when a value is offset by a very large
immediate.
Currently, an illegal 2-byte load from a 1-byte global variable is being
generated. This change instead generates a 1-byte load and zero-extends
it to i16 register. This was always the intended behavior of the
function.
In addition, an i1 ext load of any kind needs to be promoted. A missing
setLoadExtAction for ISD::EXTLOAD was causing an "Unhandled source type"
unreachable due to an illegal i1 ext load during ISelDAGtoDAG (see below
bug).
Bug https://github.com/llvm/llvm-project/issues/98033.
The NVPTX backend lowers 1/x to rcp.rn.f64 instruction instead of slower
fdiv instruction. However, in the case of -1/x, it uses the slower fdiv
instruction. After this change, -1/x will be lowered into neg.f64
(rcp.rn.f64).
This PR adds initial support for some of Volta's (sm_70) load/store
atomic and volatile/MMIO operations, hopefully without breaking any
preexisting code. Only relaxed, acquire, and release operations w/
volatile are handled.
This PR does not aim to add support for any of the following:
- syncscope support
- read atomic ops to const, param, grid param
- local memory atomics
- sequentially consistent atomics
- atomicrmw
- ...
Summary:
This patch implements support for variadic functions for NVPTX targets.
The implementation here mainly follows what was done to implement it for
AMDGPU in https://github.com/llvm/llvm-project/pull/93362.
We change the NVPTX codegen to lower all variadic arguments to functions
by-value. This creates a flattened set of arguments that the IR lowering
pass converts into a struct with the proper alignment.
The behavior of this function was determined by iteratively checking
what the NVCC copmiler generates for its output. See examples like
https://godbolt.org/z/KavfTGY93. I have noted the main methods that
NVIDIA uses to lower variadic functions.
1. All arguments are passed in a pointer to aggregate.
2. The minimum alignment for a plain argument is 4 bytes.
3. Alignment is dictated by the underlying type
4. Structs are flattened and do not have their alignment changed.
5. NVPTX never passes any arguments indirectly, even very large ones.
This patch passes the tests in the `libc` project currently, including
support for `sprintf`.
Summary:
The target information needs to configure that the platform has a
maximum integer size of 64 in order for it to enable i128 support. The
motivation behind this patch is that the i128 libcalls seem to be the
only ones used by the NVPTX backend and it would be ideal to disable
those completely. That would allow LTO to optimize libcalls properly
after https://github.com/llvm/llvm-project/pull/98512.
Summary:
These used the now-removed `nvptx` target. Because they used
`ptxas-verify` they would then fail since it's been removed. Update
these to 64-bit.
This change supports the 128-bit operands for inline ptx asm, both input
and output.\
\
The major changes are:
- Tablegen:\
Define Int128Regs in NVPTXRegisterInfo.td. But this register does
not set as general register type in NVPTX backend so that this change
will not influence the codegen without inline asm.\
Define three NVPTX intrinsics, IMOV128rr, V2I64toI128 and
I128toV2I64. The first one moves a register, the second one moves two
64-bit registers into one 128-bit register, and the third one just does
the opposite.
- NVPTXISelLowering & NVPTXISelDAGToDAG:\
Custom lowering CopyToReg and CopyFromReg with 128-bit operands.
CopyToReg deals with the inputs of the inline asm and the CopyFromReg
deals with the outputs.\
CopyToReg is custom lowered into a V2I64toI128, which takes in the
expanded values(Lo and Hi) of the input, and moves into a 128-bit reg.\
CopyFromReg is custom lowered by adding a I128toV2I64, which breaks
down the 128-bit outputs of inline asm into the expanded values.
- Supports escaped grid_constant pointers less conservatively. Casts
uses inside Calls, PtrToInts, Stores where the pointer is a _value
operand_ to generic address space, immediately before the escape, while
keeping other uses in the param address space
- Related to: https://github.com/llvm/llvm-project/pull/96125
Previously, if a ptr had align N, then the NVPTX lowering was taking
this align N to refer to the alignment of the pointer type itself, as
opposed to the alignment of the memory that it points to.
As such, if a kernel with signature
```
define void @foo(ptr align 4 %_arg_ptr)
```
takes align 4 to be the alignment of the parameter, this would result in
breaking the ld.param into two separate loads like so:
```
ld.param.u32 %rd1, [foo_param_0+4];
shl.b64 %rd2, %rd1, 32;
ld.param.u32 %rd3, [foo_param_0];
or.b64 %rd4, %rd2, %rd3;
```
It isn't necessary as far as I can tell from the PTX ISA documents to
specify the alignment of params, nor to break the loading of params into
smaller loads based on their alignment. So this patch changes the
codegen to the better:
```
ld.param.u64 %rd1, [foo_param_0];
```
Add folding for `(add (select cond, 0, (mul a, b)), c)` to `(select
cond, c, (mad a, b, c))`. Also, refactor the DAG folding implementation
to separate out the `ADD` and `FADD` folding cases.
We are running into NVPTX backend generating wrong code for an input:
```
%0 = llvm.nvvm.mma.m?n?k?.row.col.??? (...)
if laneid == 0:
ret
else:
store %0
```
The backend reorder the instruction (as an effect of `MachineSink` pass)
to
```
if laneid == 0:
ret
else:
%0 = llvm.nvvm.mma.m?n?k?.row.col.??? (...)
store %0
```
This is incorrect because `mma` is a warp instruction which needs all
threads to sync before performing the operation instead of being guarded
by a specific thread id. It should be similar as the shuffle instruction
`shfl` in terms of warp level sync, and `shfl` is marked as
`isConvergent = true`.
Apply `isConvergent = true` to `mma` instructions.
Revamp the NVVMIntrRange pass making the following updates:
- Use range attributes over range metadata. This is what instcombine has
move to for ranges on intrinsics in
https://github.com/llvm/llvm-project/pull/88776 and it seems a bit
cleaner.
- Consider the `!"maxntid{x,y,z}"` and `!"reqntid{x,y,z}"` function
metadata when adding ranges for `tid` srge instrinsics. This can allow
for smaller ranges and more optimization.
- When range attributes are already present, use the intersection of the
old and new range. This complements the metadata change by allowing
ranges to be shrunk when an intrinsic is in a function which is inlined
into a kernel with metadata. While we don't call this more then once
yet, we should consider adding a second call after inlining, once this
has had a chance to soak for a while and no issues have arisen.
I've also re-enabled this pass in the TM, it was disabled years ago due
to "numerical discrepancies" https://reviews.llvm.org/D96166. In our
testing we haven't seen any issues with adding ranges to intrinsics, and
I cannot find any further info about what issues were encountered.
Extend NVPTX DAG combining logic to distribute a mul instruction across
an add of 1 into a mad where possible. In addition, add support for
transposing a mul through a select with an option of 1, if that would
allow further mul folding.
