On GFX12+, GLOBAL_INV increments the loadcnt counter but does not write
results to any VGPRs. Previously, we unconditionally inserted
s_wait_loadcnt 0 at function returns even when the only pending loadcnt
was from GLOBAL_INV instructions.
This patch optimizes waitcnt insertion by skipping the loadcnt wait at
function boundaries when no VGPRs have pending loads. This is determined
by checking if any VGPR has a score greater than the lower bound for
LOAD_CNT - if not, the pending loadcnt must be from non-VGPR-writing
instructions like GLOBAL_INV.
The optimization is limited to GFX12+ targets where GLOBAL_INV exists
and uses the extended wait count instructions.
This is a follow-up optimization to PR #135340 which added tracking for
GLOBAL_INV in the waitcnt pass.
This patch enables the multi-group xnack replay mode by
configuring the hardware MODE register at kernel entry.
This aligns the hardware behavior with the compiler's
existing multi-group s_wait_xcnt insertion logic.
In true16 mode, D16 insts are lowered to a pseudo t16 first, and then
lowered to hi/lo inst in MC lowering using D16T16 table.
However, the D16T16 table selects both `flat_load_d16_t16 /
flat_load_d16_t16_saddr` to `flat_load_d16_(hi)_b16` which is wrong.
saddr pseudo inst `flat_load_d16_t16_saddr` should be selected to saddr
hi/lo inst
The global/scratch are correct while the flat seems to be the only one
with this issue.
Refactor the XCnt optimization checks so that they can be checked when
applying a pre-existing waitcnt. This removes unnecessary xcnt waits
when taking a loop backedge.
If we only deal with a one part of 64bit value we can just generate
merge and unmerge which will be either combined away or
selected into copy / mov_b32.
