This patch introduces `get(T)` and `set(T, Val)` functions for Waitcnt
and removes getCounterRef() and getWait(). For this to work we also need
to move InstrCounterType to AMDGPUBaseInfo.h.
Please note that the member variables are still public to keep this
patch small.
They will be replaced in the follow-up patch.
This patch implements a custom printer/parser for the immediate operand
of s_wait_alu that prints/parses the decoded counter values.
Format:
```
.<counter1>_<value1>_<counter2>_<value2>
```
Example:
`s_wait_alu .VaVdst_1_VmVsrc_1`
; Which is equivalent to this:
`s_wait_alu 8167`
Features:
- If a counter is at its maximum value it won't get printed.
- The parser will error out if a counter is greater or equal to its max
value.
- If all counters are disabled we can use 'AllOff'.
- For now we also accept numeric values for backwards compatibility with
older MIR.
Note: This is similar to https://github.com/llvm/llvm-project/pull/96004
but for `s_wait_alu`.
This PR handles`v_pk_fmac_f16` inline constant encoding/decoding
differences between pre-GFX11 and GFX11+ hardware.
- Pre-GFX11: fp16 inline constants produce `(f16, 0)` - value in low 16
bits, zero in high.
- GFX11+: fp16 inline constants are duplicated to both halves `(f16,
f16)`.
Fixes#94116.
Many `SubtargetFeature` definitions in `AMDGPU.td` follow a repetitive
pattern where a `FeatureXYZ` is paired with a `HasXYZ` predicate. This
creates significant code duplication.
This PR introduces `AMDGPUSubtargetFeature` multiclass that generates
both the `SubtargetFeature` and its corresponding `Predicate` from a
single definition. The multiclass accepts an optional `GenPredicate`
parameter (default 1) to skip predicate generation when not needed.
Not converted:
- Features with dependencies - multiclass doesn't support this yet. Will
do it in a follow-up.
- Features with irregular predicates (e.g., Predicate without
`AssemblerPredicate`, negated `Predicate`, complex multi-feature
conditions). For those without `AssemblerPredicate`, this can be done by
adding an extra optional argument to indicate whether
`AssemblerPredicate` is needed. Will be done in a follow-up.
- Features where field name doesn't match the `HasXYZ` pattern.
148 features converted, saving ~529 lines of code.
Reference issue: https://github.com/ROCm/llvm-project/issues/67
This patch adds support for expanding s_waitcnt instructions into
sequences with decreasing counter values, enabling PC-sampling profilers
to identify which specific memory operation is causing a stall.
This is controlled via:
Clang flag: -mamdgpu-expand-waitcnt-profiling /
-mno-amdgpu-expand-waitcnt-profiling
Function attribute: "amdgpu-expand-waitcnt-profiling"
When enabled, instead of emitting a single waitcnt, the pass generates a
sequence that waits for each outstanding operation individually. For
example, if there are 5 outstanding memory operations and the target is
to wait until 2 remain:
**Original**:
s_waitcnt vmcnt(2)
**Expanded**:
s_waitcnt vmcnt(4)
s_waitcnt vmcnt(3)
s_waitcnt vmcnt(2)
The expansion starts from (Outstanding - 1) down to the target value,
since waitcnt(Outstanding) would be a no-op (the counter is already at
that value).
- Uses ScoreBrackets to determine the actual number of outstanding
operations
- Only expands when operations complete in-order
- Skips expansion for mixed event types (e.g., LDS+SMEM on same counter)
- Skips expansion for scalar memory (always out-of-order)
Releated previous work for Reference
- **PR**: llvm/llvm-project#79236 (related `-amdgpu-waitcnt-forcezero`)
---------
Co-authored-by: Pierre van Houtryve <pierre.vanhoutryve@amd.com>
The 16-bit immediate operand of s_waitcnt_depctr / s_wait_alu has some
unused bits. Previously codegen would set these bits to 1, but setting
them to 0 matches the SP3 assembler behaviour better, which in turn
means that we can print them using the human readable SP3 syntax:
s_wait_alu 0xfffd ; unused bits set to 1
s_wait_alu 0xff9d ; unused bits set to 0
s_wait_alu depctr_va_vcc(0) ; unused bits set to 0, human readable
Note that the set of unused bits changed between GFX10.1 and GFX10.3.
There should normally be no need to generate implicit lit64()
modifiers on the assembler side. It's the encoder's responsibility
to recognise literals that are implicitly 64 bits wide.
The exceptions are where we rewrite floating-point operand values
as integer ones, which would not be assembled back to the original
values unless wrapped into lit64().
Respect explicit lit() modifiers for non-inline values as
necessary to avoid regressions in MC tests. This change still
doesn't prevent use of inline constants where lit()/lit64 is
specified; subject to a separate patch.
On disassembling, only create lit64() operands where necessary for
correct round-tripping.
Add round-tripping tests where useful and feasible.
This removes special case processing in TargetInstrInfo::getRegClass to
fixup register operands which depending on the subtarget support AGPRs,
or require even aligned registers.
This regresses assembler diagnostics, which currently work by hackily
accepting invalid cases and then post-rejecting a validly parsed
instruction.
On the plus side this now emits a comment when disassembling unaligned
registers for targets with the alignment requirement.
Users of the backend are expected to enable dynamic VGPRs via the
`amdgpu-dynamic-vgpr-block-size` attribute instead of the subtarget
features (see https://github.com/llvm/llvm-project/pull/133444).
We have proper encoding facilities to encode operands and instructions;
there's no need to pollute the MC representation with encoding details.
Supposed to be an NFCI, but happens to fix some re-encoded instruction
codes in disassembler tests.
The 64-bit operands are to be addressed in following patches introducing
MC-level representation for lit() and lit64() modifiers, to then be
respected by both the assembler and disassembler.
Since many code are connected, this also changes how workgroup id is lowered.
Co-authored-by: Jay Foad <jay.foad@amd.com>
Co-authored-by: Ivan Kosarev <ivan.kosarev@amd.com>
Since the register file was increased that is no longer valid to
call VGPR_32RegClass.getNumregs() to get a total number of arch
registers available on a subtarget.
Fixes: SWDEV-550425
PR #149247 made the MD accessible by the backend so we can now leverage
it in the memory model. The first use case here is detecting if a flat op
can access scratch memory.
Benefits both the MemoryLegalizer and InsertWaitCnt.
Sec. 4.6.7.1 of the gfx1250 SPG states that if an SGPR is used
as an operand, only one SGPR will be read for both the low and high
operations. As a result, the corresponding bits in `op_sel` and
`op_sel_hi` must be the same when the operand is an SGPR.
Co-authored-by: Tian, Shilei <Shilei.Tian@amd.com>
Co-authored-by: Tian, Shilei <Shilei.Tian@amd.com>
WMMA XDL instructions are tracked as TRANs ops and the compiler should
consider them the same as TRANS in S_DELAY_ALU insertion. We use a searchable
table for the InsertDelayAlu pass to recognize these WMMA XDL instructions.
Co-authored-by: Stefan Stipanovic <Stefan.Stipanovic@amd.com>
