Renames the current lowering scheme to "module" and introduces two new
ones, "kernel" and "table", plus a "hybrid" that chooses between those three
on a per-variable basis.
Unit tests are set up to pass with the default lowering of "module" or "hybrid"
with this patch defaulting to "module", which will be a less dramatic codegen
change relative to the current. This reflects the sparsity of test coverage for
the table lowering method. Hybrid is better than module in every respect and
will be default in a subsequent patch.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D139433
OMod was disabled if OpSel was enabled, but that restriction is more
specific than necessary. Any VOP3 with float operands can use OMod.
On GFX11, FMAC_F16_e64 can use op_sel.
Previously, SIFoldOperands and convertToThreeAddress were accidentally correct when
they reinterpreted the zero OMod operand on V_FMAC_F16_e64 as the OpSel operand on
V_FMA_F16_gfx9_e64. Now we explicitly add op_sel if required.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D139469
We no longer need to increase vector size to 16 for intrinsics that use more
than 8 vgprs for addr. There is no image intrinsic that needs more than 12
so all currently existing cases will be covered. Using incorrect size was
causing an error in instruction selection because instructions were updated
to require new types (9x32, 10x32, 11x32, 12x32).
Differential Revision: https://reviews.llvm.org/D139546
SOPK_Pseudo was not inheriting from SOP_Pseudo at all, and some other
Pseudo classes were needlessly redefining things that were already
defined by SOP_Pseudo.
Differential Revision: https://reviews.llvm.org/D139527
Move some code that checks if an instruction is a waitcount into a separate
function, mainly to aid readability in the logic where it is used.
Differential Revision: https://reviews.llvm.org/D139522
Update spill code to account for new vector types with
bit widths: 288, 320, 352, 384.
Related to D138205.
Differential Revision: https://reviews.llvm.org/D139203
s_endpgm is a special SOPP instruction in that its operand is optional
and if it is not present then we don't want to print a space after the
mnemonic.
Previously this was handled by defaulting real_name to the mnemonic with
a trailing space, and having s_endpgm override it to be the mnemonic
with no trailing space.
This patch implements a different approach where the separator between
Mnemonic and AsmOperands defaults to a space, but s_endpgm overrides it
to be the empty string.
Differential Revision: https://reviews.llvm.org/D139412
We were only allowing these med3 patterns if the operands were known
to not be NaN, but we should also allow it if the calls to max/min
have the `nnan` or `fast` flags.
Differential Revision: https://reviews.llvm.org/D139506
Renames the current lowering scheme to "module" and introduces two new
ones, "kernel" and "table", plus a "hybrid" that chooses between those three
on a per-variable basis.
Unit tests are set up to pass with the default lowering of "module" or "hybrid"
with this patch defaulting to "module", which will be a less dramatic codegen
change relative to the current. This reflects the sparsity of test coverage for
the table lowering method. Hybrid is better than module in every respect and
will be default in a subsequent patch.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D139433
Remove check that instruction has src_modifiers because
isRegOrImmWithInputMods essentially includes this check.
isMod is no longer used after the refactor. Also, was unreachable case
inside for loop. clamp and omod operands are handled after the loop.
Differential Revision: https://reviews.llvm.org/D139194
This patch adds forward declarations of raw_ostream to those header
files that are relying on the forward declaration of raw_ostream in
llvm/include/llvm/ADT/Optional.h.
I'm planning to move operator<< for Optional<T> and std::optional<T>
from Optional.h to llvm/include/llvm/Support/raw_ostream.h. Once I do
so, we no longer need to forward-declare raw_ostream in Optional.h.
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
This one-linear fix improves compilation time for about ~40% on ASAN enabled code.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D136069
The function makes liveness tests for the entire live register set for every instruction it passes by.
This becomes very slow on high RP regions such as ASAN enabled code.
Instead only uses of last tracked instruction should be tested and this greatly improves compilation time.
This patch revealed few bugs in SIFormMemoryClauses and PreRARematStage::sinkTriviallyRematInsts which should
be fixed first.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D136267
Don't bother pre-shrinking "s_cmp_lg_u32 reg, 0" to s_cmpk_lg_u32
because 0 is already an inline constant so the s_cmpk form is no
smaller.
This is just for consistency with the surrounding code and to simplify a
downstream patch.
Differential Revision: https://reviews.llvm.org/D138993
Fixes a longstanding TODO in the codebase where we were using S_GETREG + shift to do something that could simply be done with an inline constant (register).
Patch based on D31874 by @kzhuravl
Depends on D137767
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D137542
Following up on the removal of BufferPSV in commit 43b86bf992 ("AMDGPU:
Remove BufferPseudoSourceValue")
It is unclear what exactly the right address space for images should be.
They seem morally closest to buffers, so that's what I went with. In
practical terms, address space 7 is better than address space 0 because
it can't alias with LDS.
Differential Revision: https://reviews.llvm.org/D138949
This function removes the mentioned function, as it only does two
checks which are already implemented as part of
SelectionDAG::isKnownNeverNaN - which is called there.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D138938
We were missing legality checks. The device library build was broken
for targets without f16 support. Technically the first pattern isn't
tested by this patch; it only triggers with the isBeforeLegalize check
in performAndCombine removed. I'm not sure how to trick this into
appearing post-legalization.
The use of a PSV for buffer intrinsics is misleading because it may be
misinterpreted as all buffer intrinsics accessing the same address in
memory, which is clearly not true.
Instead, build MachineMemOperands without a pointer value but with an
address space, so that address space-based alias analysis can still
work.
There is a lot of test churn because previously address space 4
(constant address space) was used as an address space for buffer
intrinsics. This doesn't make much sense and seems to have been an
accident -- see the change in
AMDGPUTargetMachine::getAddressSpaceForPseudoSourceKind.
Differential Revision: https://reviews.llvm.org/D138711
This patch fixes a "failed to annotate CFG" error in
SIAnnotateControlFlow. The problem occurs when there are
divergent and uniform unreachable/return blocks in the same
region. In this case, AMDGPUUnifyDivergentExitNodes does not
create a unified block so the region contains multiple exits.
StructurizeCFG does not work properly when there are multiple
exits, so the neccessary CFG transformations do not occur along
divergent control flow. Subsequently, SIAnnotateControlFlow
processes the path to the divergent exit block, but may only
partially process blocks along a unform control flow path to
another exit block.
This patch fixes the bug by creating a single exit block when
there is a divergent exit block in the function.
Differential revision: https://reviews.llvm.org/D136892
class support and introduce GlobalISel implementation for AMDGPU
Uses existing SelectionDAG lowering of the llvm.amdgcn.class intrinsic
for llvm.is.fpclass
