SITargetLowering::adjustWritemask calls SelectionDAG::UpdateNodeOperands
to update an EXTRACT_SUBREG node in-place to refer to a new IMAGE_LOAD
instruction, before we delete the old IMAGE_LOAD instruction. But in
UpdateNodeOperands can do CSE on the fly and return a different
EXTRACT_SUBREG node, so the original EXTRACT_SUBREG node would still
exist and would refer to the old deleted IMAGE_LOAD instruction. This
caused errors like:
t31: v3i32,ch = <<Deleted Node!>> # D:1
This target-independent node should have been selected!
UNREACHABLE executed at lib/CodeGen/SelectionDAG/InstrEmitter.cpp:1209!
Fix it by detecting the CSE case and replacing all uses of the original
EXTRACT_SUBREG node with the CSE'd one.
This was requiring all fast math flags, which is practically
useless. This wouldn't fire using all the standard OpenCL fast math
flags. This only needs afn nnan and ninf.
https://reviews.llvm.org/D158904
There are really two rounding modes, so only return the standard
values if both modes are the same. Otherwise, return a bitmask
representing the two modes.
Annoyingly the register doesn't use the same values as FLT_ROUNDS. Use
a simple integer table we can shift into to convert.
https://reviews.llvm.org/D153158
Expand-Atomic pass emits the CAS loop for FP operations
which limits the optimizations offered by atomic optimizer.
Moving atomic optimizer before expand-atomics allows
better codegen.
Reviewed By: arsenm, #amdgpu
Differential Revision: https://reviews.llvm.org/D157265
Reduction and Scan are implemented using `Iterative`
and `DPP` strategy for `float` type.
Reviewed By: arsenm, #amdgpu
Differential Revision: https://reviews.llvm.org/D156301
This provides a uniform way to lower into the relevant instructions across all generations.
Differential Revision: https://reviews.llvm.org/D158468
Change-Id: I1f7ba4b15ee470738535cf1c7d177a11fc471e43
CMP(A,C)||CMP(B,C) => CMP(MIN/MAX(A,B), C)
CMP(A,C)&&CMP(B,C) => CMP(MIN/MAX(A,B), C)
If the operands are proven to be non NaN, then the optimization can be applied
for all predicates.
We can apply the optimization for the following predicates for FMINNUM/FMAXNUM
(for quiet and signaling NaNs) and for FMINNUM_IEEE/FMAXNUM_IEEE if we can prove
that the operands are not signaling NaNs.
- ordered lt/le and ||
- ordered gt/ge and ||
- unordered lt/le and &&
- unordered gt/ge and &&
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D155267
We were basing the defer the fast case to codegen based on the fdiv
itself, and not looking for a foldable sqrt input.
https://reviews.llvm.org/D158127
Introducing rsq contract flags is wrong, and also requires some level
of approximate functions. AMDGPUCodeGenPrepare already should handle
the f32 cases with appropriate flags, and I don't see how new
situations to handle would arise during legalization (other than cases
involving the rcp intrinsic, which instcombine tries to
handle). AMDGPUCodeGenPrepare does need to learn better handling of
rcp/rsq for f64 though, which we never bothered to handle well.
Removes another obstacle to correctly lowering sqrt.
https://reviews.llvm.org/D158099
Summary:
Emit .actual_access metadata for the deduced argument access qualifier,
and .access for kernel_arg_access_qual.
Reviewers:
arsenm
Differential Revision:
https://reviews.llvm.org/D157451
The AMDGPU backend uses a pass to transform calls to the `printf`
function to a built-in verision for either HIP or OpenCL. Currently this
does not respect `-fno-builtin` and is always emitted. This allows the
user to turn off this functionality as is standard for these types of
built-in transformations. The motivation behind this change is to allow
the `libc` project to provide a linkable version of the `printf`
function in the future.
Reviewed By: sameerds
Differential Revision: https://reviews.llvm.org/D158477
Summary:
Under -O0, device-libs may still emit these instructions under conditions.
So we need to remove them with warning if not compatible.
Fixes: SWDEV-417219
Reviewers:
arsenm, Pierre-vh and b-sumner
Differential Revision:
https://reviews.llvm.org/D158316
This extends the existing legalization combine to fold G_BUILD_VECTOR where the
sources are all from the same G_UNMERGE, to handle cases where some of the
lanes are undef. This comes up in the legalization of <3 x ..> vectors in
AArch64, where they are padded with undef.
There are two choices for what to create. This patch just removes the
G_BUILD_VECTOR/G_UNMERGE, losing the information about which lanes are undef.
The alternative would be to generate an identity G_SHUFFLE_VECTOR with undef
lanes marked as undef. I think both have advantages and disadvantages.
Differential Revision: https://reviews.llvm.org/D158063
Lower formal arguments and returns for functions with the
`amdgpu_cs_chain` and `amdgpu_cs_chain_preserve` calling conventions:
* Put `inreg` arguments into SGPRs, starting at s0, and other arguments
into VGPRs, starting at v8. No arguments should end up on the stack, if
we don't have enough registers we should error out.
* Lower the return (which is always void) as an S_ENDPGM.
* Set the ScratchRSrc register to s48:51, as described in the docs.
* Set the SP to s32, matching amdgpu_gfx. This might be revisited in a
future patch.
Differential Revision: https://reviews.llvm.org/D153517
Some opcodes in MIR are defined to be convergent by the target by setting
IsConvergent in the corresponding TD file. For example, in AMDGPU, the opcodes
G_SI_CALL and G_INTRINSIC* are marked as convergent. But this is too
conservative, since calls to functions that do not execute convergent operations
should not be marked convergent. This information is available in LLVM IR.
The new flag MIFlag::NoConvergent now allows the IR translator to mark an
instruction as not performing any convergent operations. It is relevant only on
occurrences of opcodes that are marked isConvergent in the target.
Differential Revision: https://reviews.llvm.org/D157475
This will be useful to avoid the bit-casting noise
required to extend support for Floating Point
Operations in atomic optimizer for DPP in D156301
Reviewed By: arsenm, #amdgpu
Differential Revision: https://reviews.llvm.org/D156647
The wwm register spill pseudos are currently defined for VGPR_32
regclass. It causes a verifier error for gfx908 or above as the
regalloc sometimes restores the values to the vector superclass AV_32.
Fixing it by supporting AV wwm-spill pseudos as well.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D155646
Keep components of UNMERGE larger after running the Artifact Combiner on it.
This was intended to help with <v16i64> = G_SEXT <v16i16>, but implementation
for legalizing EXT is in a following patch, therefore a test for this case
will be included in the following patch
Differential Revision: https://reviews.llvm.org/D157715
Apparently the spec has overloads for fmin/fmax and ldexp with one of
the operands as scalar. We need to broadcast the scalars to the vector
type.
https://reviews.llvm.org/D158077
