If we exit the loop at a non SALU state instruction we have to return
the next instruction because we will insert before the instruction we
return. The check before the loop already did this for cases we start on
a non SALU state instruction by returning `I`. This is now done
afterwards.
Before, when selecting candidates to rematerialize, we would only
consider SGPR candidates when there was an excess of SGPR registers.
Failing to eliminate the excess would result in spills to VGPRs.
This is normally not an issue, unless spilling to VGPRs results in
excess VGPRs.
This patch does 2 things:
* It relaxes the GCNRPTarget success criteria: now we accept regions
where we spill SGPRs to VGPRs, as long as this does not end up in
excess VGPRs.
* It changes isSaveBeneficial to consider the excess VGPRs (which
includes the SGPRs that would be spilled to VGPR).
With these changes, the compiler rematerializes VGPRs when the excess
SGPRs would result in VGPR excess.
This has some unaddressed flaws: we should attempt to rematerialize
SGPRs
first in order to eliminate the SGPR excess that results in VGPR excess.
Related to SWDEV-549940
Register classes of sources also has to be restrained to lo128.
There are few regression with register coalescing in true16 mode
though, but otherwise it fails verification.
This is needed to support functionality in the AMDGPU scheduler. Various
passes have been modified to preserve MBFI to ensure that this change
does not introduce new invocations of MBFI. Some targets have passes
reordered, but there are no new runs of MBFI.
This has no functional effect since these source fields are unused, but
it can avoid some conservative stalls due to these instructions looking
like they read from an SGPR, since 0 is the encoding for s0. Using 0x80
is more benign since it is the encoding for inline immediate 0.
Fixes: SWDEV-574953
This is to display a bug in real true16 mode that we do not have
an allocatable 16-bit VGPR class and these instructions do not
have VOP3 forms for allocatable VGPR_16 to be used. To use these
instructions 'VGPR_16_Lo128' must be allocable.
When the llvm.amdgcn.wave.shuffle intrinsic is called with a uniform
Index operand, it is effectively the same as the llvm.amdgcn.readlane
intrinsic. This change handles this situation and replaces it with the
readlane intrinsic
---------
Signed-off-by: Domenic Nutile <domenic.nutile@gmail.com>
Test case is using sed command `sed 's/,/,\n/g'` to split a line.
On AIX that is not working with the AIX system's `sed`
AIX external BB fails from
https://lab.llvm.org/buildbot/#/builders/64/builds/6911
Here substitute:
`sed 's/,/,\n/g'`
with:
`tr ',' '\n'`
but because `tr` does not keeps the comma, also needed to change looked
for texts i.e. to remove the comma `,` from them since it is not needed
for the correctness.
Co-authored-by: Daniel Chen <cdchen@ca.ibm.com>
Corrected various spelling mistakes such as 'occurred', 'receiver',
'initialized', 'length', and others in comments, variable names,
function names, and documentation throughout the project. These
changes improve code readability and maintain consistency in naming
and documentation.
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
Previously we were casting v2bf16 to i32, unlike the f16 case. Simplify
this by using the natural vector type. This is probably a leftover from
before v2bf16 was treated as legal. This is preparation for fixing a
miscompile in globalisel.
This is a significant refactoring of the scheduler's rematerialization
stage meant to improve rematerialization capabilities and lay strong
foundations for future improvements.
As before, the stage identifies scheduling regions in which RP must be
reduced (so-called "target regions"), then rematerializes registers to
try and achieve the desired reduction. All regions affected by
rematerializations are re-scheduled, and, if the MIR is deemed worse
than before, rematerializations are rolled back to leave the MIR in its
pre-stage state.
The core contribution is a scoring system to estimate the benefit of
each rematerialization candidate. This score favors rematerializing
candidates which, in order, would
1. (if the function is spilling) reduce RP in highest-frequency target
regions,
2. be rematerialized to lowest-frequency target regions, and
3. reduce RP in the highest number of target regions.
All rematerialization opportunities are initially scored and
rematerialized in decreasing score order until RP objectives are met or
pre-computed scores diverge from reality; in the latter case remaining
candidates are re-scored and the process repeats. New tests in
`machine-scheduler-rematerialization-scoring.mir` showcase how the
scoring system dictates which rematerialization are the most beneficial
and therefore performed first
A minor contribution included in this PR following previous feedback is
that rollback now happens in-place i.e., without having to re-create the
rematerialized MI. This leaves original slot indices and registers
untouched. We achieve this by temporarily switching the opcode of
rollback-able instructions to a debug opcode during re-scheduling so
that they are ignored.
Add support for flushing DS_CNT in loop preheaders when the loop uses
values that were DS-loaded outside the loop. This is similar to the
existing VMEM loop optimization.
Assisted-by: Cursor / claude-4.5-opus-high
Reference issue: https://github.com/ROCm/llvm-project/issues/67
This patch adds support for expanding s_waitcnt instructions into
sequences with decreasing counter values, enabling PC-sampling profilers
to identify which specific memory operation is causing a stall.
This is controlled via:
Clang flag: -mamdgpu-expand-waitcnt-profiling /
-mno-amdgpu-expand-waitcnt-profiling
Function attribute: "amdgpu-expand-waitcnt-profiling"
When enabled, instead of emitting a single waitcnt, the pass generates a
sequence that waits for each outstanding operation individually. For
example, if there are 5 outstanding memory operations and the target is
to wait until 2 remain:
**Original**:
s_waitcnt vmcnt(2)
**Expanded**:
s_waitcnt vmcnt(4)
s_waitcnt vmcnt(3)
s_waitcnt vmcnt(2)
The expansion starts from (Outstanding - 1) down to the target value,
since waitcnt(Outstanding) would be a no-op (the counter is already at
that value).
- Uses ScoreBrackets to determine the actual number of outstanding
operations
- Only expands when operations complete in-order
- Skips expansion for mixed event types (e.g., LDS+SMEM on same counter)
- Skips expansion for scalar memory (always out-of-order)
Releated previous work for Reference
- **PR**: llvm/llvm-project#79236 (related `-amdgpu-waitcnt-forcezero`)
---------
Co-authored-by: Pierre van Houtryve <pierre.vanhoutryve@amd.com>
Add support for G_PTRMASK but we are missing p8 (buffer resource) due to
a legalizer issue in GlobalISel which does not occur on SelectionDAG:
`LLVM ERROR: unable to legalize instruction: %17:_(p8) = G_PTRMASK %0:_,
%22:_(s128) (in function: v_ptrmask_buffer_resource_variable_i48)`
Added a FIXME to indicate this issue.
The COPY inserted for liverange split during sgpr-regalloc
pipeline currently breaks the BB prolog during the subsequent
vgpr-regalloc phase while spilling and/or splitting the vector
liveranges. This patch fixes it by correctly including the
LR split instructions during sgpr-regalloc and wwm-regalloc
pipelines into the BB prolog.
LocalStackSlotAllocation pass disallows negative offsets with respect to
a base register. The pass ends up introducing a new register for such
frame references. This patch helps LocalStackSlotAlloca to additionally
consider the immediate offset of an instruction, when sorting frame refs
- hence, avoiding negative offsets and maximizing reuse of the existing
registers.
Summary:
We already have a way to get the block count using the old grid size
lookup and dividing it by the number of threads. We did not want to make
a new intrinsic to do the same thing, so this optimization pattern
matches on this usage to automatically optimize it to the new form. This
should improve performance of old kernels by converting branches into a
simple index lookup and removing the division.
This patch precommits a test where base offsets are negative. PR
[171012](https://github.com/llvm/llvm-project/pull/171012 ) will
eliminate negative offsets by sorting the scratch instructions.
In some cases, the placeholder itself can be used as the value for its
corresponding block in `SSAUpdater`, and later used as an incoming value
in another block in `GetValueInMiddleOfBlock`. If we erase it too early,
this can lead to a use-after-erase.
Before, GlobalISel would still return true for lowering the intrinsic
for GFX7 and earlier even though the required ds_bpermute_b32
instruction is not supported. After this change, GlobalISel will
properly report failure to select in this case. Testing is updated
appropriately.
Signed-off-by: Domenic Nutile <domenic.nutile@gmail.com>
This change adds a new intrinsic for AMDGPU that implements a wave
shuffle, allowing arbitrary swizzling between lanes using an index. In
the initial version of this commit, there was an issue in one of the
tests added that returned a signal, causing testing to fail when
combined with another recent change to 'not'.
For context on the initial commit see #167372
---------
Signed-off-by: Domenic Nutile <domenic.nutile@gmail.com>
Co-authored-by: Jay Foad <jay.foad@gmail.com>
