When generating waitcounts before a use or def skip VGPRs. We never have
a real implicit VGPR operands on memory instructions, it is only for
super-reg liveness accounting.
Some other instructions (MOVRELS as an example) may have real implicit
VGPR uses though.
This is less then ideal but most of the problems observed with spills.
This attempts to fold:
```
%1:_(<2 x s32>), %2:_(<2 x s32>) = G_UNMERGE_VALUES %0:_(<4 x s32>)
%3:_(s32), %4:_(s32) = G_UNMERGE_VALUES %1
```
Into a single UNMERGE:
```
%3:_(s32), %4:_(s32), %5:_(s32), %6:_(s32) = G_UNMERGE_VALUES %0
```
This transform already exists, this patch alters it to occur when the
result UNMERGE is considered legal.
It does not try to transform where the result would be extracting a
subelement from a vector at the moment, as the code is not setup to
handle it.
```
%1:_(s32), %2:_(s32) = G_UNMERGE_VALUES %0:_(<2 x s32>)
%3:_(s16), %4:_(s16) = G_UNMERGE_VALUES %1
```
This helps us reduce the amount of legalization artefacts, especially
from widened vectors padded with undef.
This reverts commit 63b2595846b86b4e4eb9afba5e97dd64e8135c10.
(llvmorg-20-init-6782-g63b2595846b8)
A few bots have been failing on `inst-select-unmerge-values.mir`
The accumulateUsedDefed() was missing if block prologue interference
check does not pass. This would cause incorrect register dependency,
which cause incorrect sinking.
Previously we would fail an assertion in RemoveNodeFromCSEMaps after
lowering:
t3: ch = llvm.amdgcn.s.barrier.join t0, TargetConstant:i64<2973>,
Constant:i32<0>
to:
t6: ch = S_BARRIER_JOIN_IMM TargetConstant:i32<0>
Both SGPR->VGPR and VGPR->AGPR spilling code give a fixup to the spill
frame indices referred in debug instructions so that they can be
entirely removed. The stack argument is present at 0th index in
DBG_VALUE and at 2nd index for DBG_VALUE_LIST.
Fixes: SWDEV-484156
Some flat instructions have an saddr operand. When 'null' is provided as
saddr, it may have the same encoding as another instruction. For
example, the instructions 'global_atomic_add v1, v2, null' and
'global_atomic_add v[1:2], v2, off' have the same encoding. This patch
disallows having null as saddr.
Unlike with implicitly preloaded data UserSGPRs firmware is unable to
handle cases where SGPRs for kernel arguments contain preloaded data but
not are not explicitly referenced in the kernel. We need to include
these preloaded SGPRs in the GRANULATED_WAVEFRONT_SGPR_COUNT calculation
to not clobber SGPRs in adjacent waves.
Add test for readfirstlane with i1 type to demonstrate the lowering
works.
Also simplify existing tests a bit - the declarations are not strictly
needed anymore.
Support true16 and fake16 format for more VOP1 instructions in MC
This patch updates the true16 and fake16 vop_profile for the following
instructions and update the asm/dasm tests:
V_CVT_F16_U16
V_CVT_F16_I16
V_CVT_U16_F16
V_CVT_I16_F16
V_CVT_NORM_U16_F16
V_CVT_NORM_I16_F16
V_FREXP_EXP_I16_F16
Since this patch introduce fake16 instructions for V_CVT_F16_U16, it
address an issue in fix-sgprs-copy-f16 test which is brought up here
https://github.com/llvm/llvm-project/pull/104510#discussion_r1742499668
This should fix expensive_checks failures.
These now hit a new verifier error. The first test is explicitly
testing the failing case produces an error, so it should skip
the verifier. The unsupported call tests should also error (although
it would be better if it errored via a different path).
Addition of a check in the MachineVerifier to detect and report illegal
vector registers to SGPR copies in the AMDGPU backend, ensuring correct
code generation.
We can enforce this check only after SIFixSGPRCopies pass.
This is half-fix in the pipeline with the help of isSSA MachineFuction
property, the check is happening for passes after phi-node-elimination.
…ntElimination
ArgumentPromotion and DeadArgumentElimination passes could change
function signatures but the function name remains the same as before the
transformation. This makes it hard for tracing with bpf programs where
user tends to use function signature in the source. See discussion [1]
for details.
This patch added suffix to functions whose signatures are changed. The
suffix lets users know that function signature has changed and they need
to impact the IR or binary to find modified signature before tracing
those functions.
The suffix for ArgumentPromotion is ".argprom" and the suffixes for
DeadArgumentElimination are ".argelim" and ".retelim". The suffix also
gives user hints about what kind of transformation has been done.
With this patch, I built a recent linux kernel with full LTO enabled. I
got 4 functions with only argpromotion like
```
set_track_update.argelim.argprom
pmd_trans_huge_lock.argprom
...
```
I got 1058 functions with only deadargelim like
```
process_bit0.argelim
pci_io_ecs_init.argelim
...
```
I got 3 functions with both argpromotion and deadargelim
```
set_track_update.argelim.argprom
zero_pud_populate.argelim.argprom
zero_pmd_populate.argelim.argprom
```
[1] https://github.com/llvm/llvm-project/issues/104678
Implicit defs and uses on spill stores were accounted as real defs and
uses, while only exist for liveness accounting. As a result unneded
waits were generated.
Fixes: SWDEV-484177
This reverts commit
7792b4ae79.
The problem was a conflict with
e55d6f5ea2
"[AMDGPU] Simplify and improve codegen for llvm.amdgcn.set.inactive
(https://github.com/llvm/llvm-project/pull/107889)"
which changed the syntax of V_SET_INACTIVE (and thus made my MIR test
crash).
...if only we had a merge queue.
- Updated `phi_moveimm_subreg_input` test case to introduce
sub-registers as PHI input operands.
Currently subreg is making the testcase in non-SSA format, need to fix
this by giving subreg as an input operand to PHI instead defining the
subreg register.
This change is relevant for : [[AMDGPU] Add MachineVerifier check to
detect illegal copies from vector register to SGPR
](https://github.com/llvm/llvm-project/pull/105494)
Fix a bug which resulted in selection of s_load_b96 on GFX11, which only
exists in GFX12.
The root cause was a mismatch between legalization and selection. The
condition used to check that the load was uniform in legalization
(SITargetLowering::LowerLOAD) was "!Op->isDivergent()". The condition
used to detect a non-uniform load during selection
(AMDGPUDAGToDAGISel::isUniformLoad()) was
"N->isDivergent() && !AMDGPUInstrInfo::isUniformMMO(MMO)". This makes a
difference when IR uniformity analysis has more information than SDAG's
built in analysis. In the test case this is because IR UA reports that
everything is uniform if isSingleLaneExecution() returns true, e.g. if
the specified max flat workgroup size is 1, but SDAG does not have this
optimization.
The immediate fix is to use the same condition to detect uniform loads
in legalization and selection. In future SDAG should learn about
isSingleLaneExecution(), and then it could probably stop relying on IR
metadata to detect uniform loads.
Reverts llvm/llvm-project#108173
si-init-whole-wave.mir crashes on some buildbots (although it passed
both locally with sanitizers enabled and in pre-merge tests).
Investigating.
Always generate v_cndmask_b32 instead of modifying exec around
v_mov_b32. This is expected to be faster because
modifying exec generally causes pipeline stalls.
This is a large patch includes the MC level support for V_CVT_F16_F32,
V_CVT_F32_F16 and V_LDEXP_F16 in true16 format.
This patch includes the asm/disasm changes to encode/decode the 16bit
vsrc, vdst and src modifieres for vop and dpp format. This patch is a
dependency for many 16 bit instructions while only three instructions
are updated to make it easier to review.
There will be another patch to support these three instructions in the
codeGen level, this patch just replaces these two instructions with its
fake16 format.
trunc (binop X, C) --> binop (trunc X, trunc C) --> binop (trunc X, C`)
Try to narrow the width of math or bitwise logic instructions by pulling
a truncate ahead of binary operators.
Vx and Nx cores consider 32-bit and 64-bit basic arithmetic equal in
costs.
This intrinsic is meant to be used in functions that have a "tail" that
needs to be run with all the lanes enabled. The "tail" may contain
complex control flow that makes it unsuitable for the use of the
existing WWM intrinsics. Instead, we will pretend that the function
starts with all the lanes enabled, then branches into the actual body of
the function for the lanes that were meant to run it, and then finally
all the lanes will rejoin and run the tail.
As such, the intrinsic will return the EXEC mask for the body of the
function, and is meant to be used only as part of a very limited pattern
(for now only in amdgpu_cs_chain functions):
```
entry:
%func_exec = call i1 @llvm.amdgcn.init.whole.wave()
br i1 %func_exec, label %func, label %tail
func:
; ... stuff that should run with the actual EXEC mask
br label %tail
tail:
; ... stuff that runs with all the lanes enabled;
; can contain more than one basic block
```
It's an error to use the result of this intrinsic for anything
other than a branch (but unfortunately checking that in the verifier is
non-trivial because SIAnnotateControlFlow will introduce an amdgcn.if
between the intrinsic and the branch).
The intrinsic is lowered to a SI_INIT_WHOLE_WAVE pseudo, which for now
is expanded in si-wqm (which is where SI_INIT_EXEC is handled too);
however the information that the function was conceptually started in
whole wave mode is stored in the machine function info
(hasInitWholeWave). This will be useful in prolog epilog insertion,
where we can skip saving the inactive lanes for CSRs (since if the
function started with all the lanes active, then there are no inactive
lanes to preserve).
LLPC can generate llvm.amdgcn.image.atomic.swap intrinsic with data
argument as float type as well as float return type. This went unnoticed
until CreateIntrinsic with implicit mangling was used.
