Selecting the score in SGPRInfo used to require an index which you would
get by calling a getSgprScoresIdx(), which is defined in a different
class.
This patch moves the score selection logic into the SGPRinfo. This makes
the interface simpler and more intuitive.
Also given that SGPRInfo contains only two scores, this patch also
replaces the score array with individual score variables.
Should be NFC.
Async operations transfer data between global memory and LDS. Their
progress is tracked by the ASYNC_CNT counter on GFX1250 and later
architectures. This change introduces the representation of that counter
in SIInsertWaitCnts. For now, the programmer must manually insert
s_wait_asyncnt instructions. Later changes will add compiler assistance
for generating the waits by including this counter in the asyncmark
instructions.
Assisted-by: Claude Sonnet 4.5
This is part of a stack:
- #185813
- #185810
There is no need to clear state at the start or end of the run method,
because a fresh instance of SIInsertWaitcnts is constructed for each run
on a MachineFunction.
Pass MF into the SIInsertWaitcnts constructor instead of the run method.
This is more natural now that SIInsertWaitcnts is constructed once per
MachineFunction and enables future cleanup by initializing more fields
in the constructor that depend on MF.
WaitcntGenerator state depends on MF attributes, so create a new WCG object per MF
until we have a better solution. This patch also adds a test that exercises this.
Even though we stopped creating a new WCG instance in #177689, the behavior
didn't change because SIInsertWaitcnts gets recreated on every MF
(so this patch is practically an NFC).
The loop is collecting the first instruction of each waitcnt kind and is
erasing the rest, with the exception of DEPCTR which needs more checks.
The existing code was factoring out the instruction deletion and the
setting of the collected instruction variables. But the special handling
for DEPCTR and the in-loop deletion of `S_WAITCNT_lds_direct` was just
complicating the logic.
Add support for prefetch patterns where some DS loads are used in the
same iteration (creating flush points) while others remain unflushed at
the backedge.
This complements the existing pure prefetch optimization (PR172728) by
handling cases where partial same-iteration consumption occurs.
Assisted-by: Cursor / claude-4.5-opus-high
This is a cleanup patch that moves the VCCZ specific workaround code
from `SIInsertWaitcnts::insertWaitcntInBlock()` to a separate class and
refactors it a bit to make it easier to read.
The end result is a simpler `insertWaitcntInBlock()`.
Should be NFC.
This patch makes Waitcnt member variables private and replaces their
accesses with calls to set() or get(). This will help us change the
implementation to an a array in the followup patch.
After merging https://github.com/llvm/llvm-project/pull/181760 this loop
no longer erases instructions while iterating. So this patch moves the
iteration increment inside the for statement.
Note: Replacing the loop with a range loop like `for (MachineInstr
&Instr : Block)` is not an NFC.
This patch simplifies the logic of `insertWaitcntInBlock()` by moving
the code that removes the redundant soft xcnt instructions to a new
function: `removeRedundantSoftXcnts()`.
While doing so, this patch also cleans up the logic a bit by dropping
the AtomiRMWState and the corresponding functions.
This helps in several ways:
- insertWaitcntInBlock() will now do what its name suggests, i.e., only
insert and not remove.
- it makes it clear that removal of softxcnts is orthogonal to insertion
of waitcnts.
- we won't have to worry about both erased and new instruction in
insertWaitcntInBlock()'s loop.
The change should be NFC.
Asynchronous operations are memory transfers (usually between the global
memory and LDS) that are completed independently at an unspecified
scope. A thread that requests one or more asynchronous transfers can use
async marks to track their completion. The thread waits for each mark to
be completed, which indicates that requests initiated in program order
before this mark have also completed.
For now, we implement asyncmark/wait operations on pre-GFX12
architectures that support "LDS DMA" operations. Future work will extend
support to GFX12Plus architectures that support "true" async operations.
This is part of a stack split out from #173259
- #180467
- #180466
Co-authored-by: Ryan Mitchell ryan.mitchell@amd.com
Fixes: SWDEV-521121
This patch introduces `get(T)` and `set(T, Val)` functions for Waitcnt
and removes getCounterRef() and getWait(). For this to work we also need
to move InstrCounterType to AMDGPUBaseInfo.h.
Please note that the member variables are still public to keep this
patch small.
They will be replaced in the follow-up patch.
The user of the WaitcntBrackets class shouldn't need to know about how
the scoreboard has been implemented internally. So I think it is best to
provide a higher level API that hides things like scoreUB, scoreLB and
score ranges.
This patch makes getScoreUB(), getScoreLB() and getScoreRange() private
and introduces new functions that don't expose the internal
implementation:
- getOutstanding(T)
- hasPendingVMEM(VMEMID, T)
- empty(T)
I also noticed that getSGPRScore() and getVMemScore() are not used
externally so these are now private.
With this patch we are no longer exposing the internal data structure
that holds the WaitEvents to the user through the `getWaitEventMask()`
API. Instead we only allow the user to query a specific type and get the
corresponding `WaitEventSet` with `getWaitEvents(T)`.
Note: This patch also renames `getWaitEventMask()` to `getWaitEvents()`
because we are no longer returning a mask but instead a `WaitEventSet`
object.
On GFX9, BUFFER_WBL2 is used to write back dirty cache lines and
requires an s_waitcnt vmcnt(0) afterwards to ensure completion.
This patch fixes by incrementing vmcnt for buffer_wbl2 instruction
---------
Co-authored-by: Jay Foad <jay.foad@gmail.com>
The eventCounter() function searches through the array of events. This
array is owned by the WaitcntGenerator class.
This patch moves the function into the WaitcntGenerator class which
helps hide the event array from the user.
It also renames it to getCounterFromEvent().
This should be NFC.
Wait is initialized with all ~0s and by the time it reaches the updated
line it still holds the same value. So Wait.combined(AllZeroWait) is
effectively combining all ~0s with AllZeroWait and given that combined()
returns the min() of the two it should always return AllZeroWait.
So this patch replaces the assignment with `= AllZeroWait` to make it
easier to read.
Before this patch WaitEventType events used to be collected in unsigned
integers that were used as small bit vectors.
This patch introduces a WaitEventSet container class to replace the
integer bit vectors with a class that hides the implementation of common
operations like insertion, removal, union, intersection etc. from the
user.
The WaitEventSet API matches that of a set and not a vector because we
don't care about the order of its contents. Internally though it is
still a bit vector that uses an unsigned integer as its storage, just
like the original implementation.
This patch should not change the functionality.
We inserted the DEALLOC_VGPRS message if there were no pending scratch
stores the first time an S_ENDPGM instruction was visited. But because
this pass uses a worklist to revisit blocks until it reaches a fixed
point, it is possible that pending scratch stores are only discovered on
the second or later visit to a block. Fix this by storing a flag for
each S_ENDPGM instruction which can be updated by later visits.
This patch optimizes the insertion of s_wait_xcnt instruction for
sequences of atomic read-modify-write (RMW) operations in the
SIInsertWaitcnts pass. The Memory Legalizer conservatively inserts a
soft xcnt instruction before each atomic RMW operation as part of PR
168852, which is correct given the nature of atomic operations.
However, for back-to-back atomic RMWs, only the first s_wait_xcnt is
necessary for better runtime performance. This patch tracks atomic
RMW blocks within each basic block and removes redundant soft xcnt
instructions, keeping only the first wait in each sequence. An atomic
RMW block continues through subsequent atomic RMWs and non-memory
instructions (e.g., ALU operations) but is broken by CU-scoped memory
operations, atomic stores, or basic block boundaries.
This reverts commit 0dd03598dca91c93c74b94a714c38a4ffad0ed1c.
Apparently the stack usage on Windows was still large enough to cause
problems for some DirectX games.
`GCNSubtarget.h` contained a large amount of repetitive code following
the pattern `bool HasXXX = false;` for member declarations and `bool
hasXXX() const { return HasXXX; }` for getters. This boilerplate made
the file unnecessarily long and harder to maintain.
This patch introduces an X-macro pattern `GCN_SUBTARGET_HAS_FEATURE`
that consolidates 135 simple subtarget features into a single list. The
macro is expanded twice: once in the protected section to generate
member variable declarations, and once in the public section to generate
the corresponding getter methods. This reduces the file by approximately
600 lines while preserving the exact same API and functionality.
Features with complex getter logic or inconsistent naming conventions
are left as manual implementations for future improvement.
Ideally, these could be generated by TableGen using
`GET_SUBTARGETINFO_MACRO`, similar to the X86 backend. However,
`AMDGPU.td` has several issues that prevent direct adoption: duplicate
field names (e.g., `DumpCode` is set by both `FeatureDumpCode` and
`FeatureDumpCodeLower`), and inconsistent naming conventions where many
features don't have the `Has` prefix (e.g., `FlatAddressSpace`,
`GFX10Insts`, `FP64`). Fixing these issues would require renaming fields
in `AMDGPU.td` and updating all references, which is left for future
work.
Add support for flushing DS_CNT in loop preheaders when the loop uses
values that were DS-loaded outside the loop. This is similar to the
existing VMEM loop optimization.
Assisted-by: Cursor / claude-4.5-opus-high
Reference issue: https://github.com/ROCm/llvm-project/issues/67
This patch adds support for expanding s_waitcnt instructions into
sequences with decreasing counter values, enabling PC-sampling profilers
to identify which specific memory operation is causing a stall.
This is controlled via:
Clang flag: -mamdgpu-expand-waitcnt-profiling /
-mno-amdgpu-expand-waitcnt-profiling
Function attribute: "amdgpu-expand-waitcnt-profiling"
When enabled, instead of emitting a single waitcnt, the pass generates a
sequence that waits for each outstanding operation individually. For
example, if there are 5 outstanding memory operations and the target is
to wait until 2 remain:
**Original**:
s_waitcnt vmcnt(2)
**Expanded**:
s_waitcnt vmcnt(4)
s_waitcnt vmcnt(3)
s_waitcnt vmcnt(2)
The expansion starts from (Outstanding - 1) down to the target value,
since waitcnt(Outstanding) would be a no-op (the counter is already at
that value).
- Uses ScoreBrackets to determine the actual number of outstanding
operations
- Only expands when operations complete in-order
- Skips expansion for mixed event types (e.g., LDS+SMEM on same counter)
- Skips expansion for scalar memory (always out-of-order)
Releated previous work for Reference
- **PR**: llvm/llvm-project#79236 (related `-amdgpu-waitcnt-forcezero`)
---------
Co-authored-by: Pierre van Houtryve <pierre.vanhoutryve@amd.com>
The original design was:
- WaitcntBrackets::simplifyWaitcnt(Wait) updates Wait based on the
current state of WaitcntBrackets, removing unnecesary waits.
- WaitcntBrackets::applyWaitcnt(Wait) updates WaitBrackets based on
Wait, updating the state by applying the specified waits.
This was changed by #164357 which started calling applyWaitcnt from
simplifyWaitcnt.
This patch restores the original design without any significant
functional changes. There is some code duplication because both
simplifyWaitcnt and applyWaitcnt need to understand how XCNT interacts
with other counters like LOADCNT and KMCNT.
Start documenting the ABI conventions for dependency counters on
function call and return.
Stop pretending that SIInsertWaitcnts can handle anything other than the
default documented behavior.
This reverts commit 008c875be85732f72c4df4671167f5be79f449eb.
PR #162077 / #171779 shrunk the WaitcntBrackets class by using DenseMaps
instead of large arrays, so the size of a temporary WaitcntBrackets
allocated on the stack is no longer a concern.
With this patch on Linux I measured the stack size of
SIInsertWaitcnts::run increasing from 456 bytes to 632 bytes.
