- Add clang built-ins + sema/codegen
- Add IR Intrinsic + verifier
- Add DAG/GlobalISel codegen for the intrinsics
- Add lowering in SIMemoryLegalizer using a MMO flag.
Support tail calls to whole wave functions (trivial) and from whole wave
functions (slightly more involved because we need a new pseudo for the
tail call return, that patches up the EXEC mask).
Move the expansion of whole wave function return pseudos (regular and
tail call returns) to prolog epilog insertion, since that's where we
patch up the EXEC mask.
Correctly model these without AV_* operands. This is another
step towards removing the special casing in
TargetInstrInfo::getRegClass. Also add some tests for this.
Make sure the MFMA VGPR to AGPR InstrMapping table is complete.
I think I got everything, except the full cross product of input
types with the mfma scale intrinsics. Also makes sure we have
coverage for smfmac and mfma_scale cases.
This should not require aligned registers. Fixes expensive_checks
test failure. I don't see a better way until the new system
to specify the alignment per register is done.
The instruction definitions for loads and stores do not
accurately model the operand constraints of loads and stores
with AGPRs. They use AV register classes, plus a hack
a hack in getRegClass/getOpRegClass to avoid using AGPRs or
AV classes with the multiple operand cases, but it did not
consider the 3 operand case.
Model this correctly by using separate all-VGPR and all-AGPR
variants for the cases with multiple data operands.
This does regress the assembler errors on gfx908 for the
multi-operand cases. It now reports a generic operand
invalid error for GPU instead of the specific message
that agpr loads and stores aren't supported.
In the future AMDGPURewriteAGPRCopyMFMA should be taught
to replace the VGPR forms with the AGPR ones.
Most of the diff is fighting the DS pseudo structure. The
mnemonic was being used as the key to SIMCInstr, which is a
collision in the AGPR case. We also need to go out of our way
to make sure we are using the gfx9+ variants of the pseudos
without the m0 use. The DS multiclasses could use a lot of
cleanup.
Fixes#155777
This patch implements a correctly rounded expansion of the frem
instruction in LLVM IR. This is useful for target architectures for
which such an expansion is too involved to be implement in ISel
Lowering. The expansion is based on the code from the AMD device libs
and has been tested successfully against the OpenCL conformance tests on
amdgpu. The expansion is implemented in the preexisting "expand-fp"
pass. It replaces the expansion of "frem" in ISel for the amdgpu target;
it is enabled for targets which do not directly support "frem" and for
which no matching "fmod" LibCall is available.
---------
Co-authored-by: Matt Arsenault <Matthew.Arsenault@amd.com>
It specifies the target machine as AMDGPU for
AMDGPUAnnotateUniformValues pass-related test (that uses UA). Before in
its absense, the UA would consider everything Uniform resulting in
setting metadata incorrectly for AMDGPU. Now, after specifying the
AMDGPU, the UA would be rightful sets the right metadata as the test
gets commpiled for AMDGPU.
Reapplying now that buildbot has picked up the new configuration
that does not use -treat-scalable-fixed-error-as-warning.
-----
This removes the `LLVM_ENABLE_STRICT_FIXED_SIZE_VECTORS` cmake option
and the `-treat-scalable-fixed-error-as-warning` opt flag.
We stopped treating these as warnings by default a long time ago
(62f09d788f9fc540db12f3cfa2f98760071fca96), so I don't think it makes
sense to retain these options at this point. Accessing a scalable
TypeSize as fixed should always result in an error.
Handle a special case for copies from AGPR VGPR on the MFMA inputs.
If the "input" is really a subregister def, we will not see the
usual copy to VGPR for src2, only the read of the subregister def.
Not sure if this pattern appears in practice.
In future changes we will have more AV_ virtual registers, which
currently
block the formation of write2. Most of the time these registers can
simply
be constrained to VGPR, so do that.
Also relaxes the constraint in flat merging case. We already have the
necessary
code to insert copies to the original result registers, so there's no
point
in avoiding it.
Addresses the easy half of #155769
It is not safe to use isOperandLegal on an instruction that does
not have a complete set of operands. Unforunately the APIs are
not set up in a convenient way to speculatively check if an instruction
will be legal in a hypothetical instruction. Build all the operands
and then verify they are legal after. This is clumsy, we should have
a more direct check for will these operands give a legal instruction.
This seems to fix a missed optimization in the gfx11 test. The
fold was firing for gfx1150, but not gfx1100. Both should support
vop3 literals so I'm not sure why it wasn't working before.
Re-enable it by adding a wait on vm_vsrc before every barrier "start"
instruction in GFX10/11/12 CU mode.
This is a less strong wait than what we do without BackOffBarrier, thus
this shouldn't introduce
any new guarantees that can be abused, instead it relaxes the guarantees
we have now to the bare
minimum needed to support the behavior users want (fence release +
barrier works).
There is an exact memory model in the works which will be documented
separately.
This removes the `LLVM_ENABLE_STRICT_FIXED_SIZE_VECTORS` cmake option
and the `-treat-scalable-fixed-error-as-warning` opt flag.
We stopped treating these as warnings by default a long time ago
(62f09d788f9fc540db12f3cfa2f98760071fca96), so I don't think it makes
sense to retain these options at this point. Accessing a scalable
TypeSize as fixed should always result in an error.
GEPs are often in the form `gep [N x %T], ptr %p, i64 0, i64 %idx`.
Canonicalize these to `gep %T, ptr %p, i64 %idx`.
This enables transforms that only support one GEP index to work and
improves CSE.
Various transforms were recently hardened to make sure they still work
without the leading index.
This fixes an old FIXME, where (workitem ID X) / (wavefrront size) would
never be marked uniform if it was possible that there would be Y and Z
dimensions. Now, so long as the required size of the X dimension is a
power of 2, dividing that dimension by the wavefront size creates a
uniform value.
Furthermore, if the required launch size of the X dimension is a power
of 2 that's at least the wavefront size, the Y and Z workitem IDs are
now marked uniform.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
When zero cost instructions are hoisted, the simplifyHoistedPhi function
was setting incoming phi values which were not dominating the use
causing runtime failure. This was set to poison by rebuildSSA function.
This commit fixes the issue.
Let's do the lowering of non-split into split barriers in a new IR pass,
AMDGPULowerIntrinsics. That way, there is no code duplication between
SelectionDAG and GlobalISel. This simplifies some upcoming extensions to
the code.
The handling of AGPR vs. VGPR operand restrictions is
broken and results in bugs like #155777 and missed optimizations.
Add some baseline tests for future improvements.
Avoid bundling a SCHED_BARRIER with memops. Memops are still can be
bundled and a SCHED_BARRIER ends the bundle. (e.g. [load, load, ...
SCHED_BARRIER(exclusive from the bundle) ), This is to honor the
SCHED_BARRIERs maximally without intervention of bundling.
If a SCHED_BARRIER is placed in a bundle between memops, the
SCHED_BARRIER is not used during IGroupLPMutation in postra mi-sched
phase. In addition, bundling memory ops with in-between SCHED_BARRIER
can prevent that SCHED_BARRIER or any neighboring SCHED_BARRIER being
honored. As users already provided SCHED_BARRIER between memory ops,
don't bundle it together with memory ops. Bypassing any bundling in a
MBB with a SCHED_BARRIER removes all those problems occur.
We do not need this in the attributor, because `ST.getWavesPerEU`
accounts for both the waves-per-eu and flat-workgroup-size attributes.
If the waves-per-eu values are not valid, it drops them. In the
attributor, we only need to propagate the values without using
intermediate flat workgroup size values.
Fixes SWDEV-550257.
Dead VGPR->SGPR copies were converted to IMPLICIT_DEF assignments that
were unused. Prevent these from being created and update the numerous
affected tests.
According to `llvm-project/llvm/docs/AMDGPUUsage.rst::L5212` the
`GRANULATED_WAVEFRONT_SGPR_COUNT`, which is `compute_pgm_rsrc1[6:9]` has
to be 0 for gfx10+ arch
---------
Co-authored-by: Matt Arsenault <Matthew.Arsenault@amd.com>
When going through the ISD::EXTRACT_ELEMENT case, `KnownSign - rIndex *
BitWidth`
could produce a negative. When a negative is produced, the lower bound
of the `std::clamp` is returned. Change that lower bound to one to avoid
potential underflows, because the expectation is that
`ComputeNumSignBits`
should always return at least 1.
Fixes#155452.
