FullLTO has to run this early before module splitting occurs otherwise
module splitting won't work as expected. There was a targeted fix for
fortran on another branch that disables the LTO run but that'd break
full LTO module splitting entirely.
Test changes are due to metadata indexes shifting.
See #122891
Canonicalize it to a G_AND instead so that ISel patterns can pick it
up and ignore it, as the shift instructions only read low bits.
G_ZEXT would be lowered to a v/s_and anyway in most cases.
I'm also looking at making a DAG version of this in a separate patch.
Before V_CNDMASK_B32_e64 gets converted to SDWA form, a conversion to
V_CNDMASK_B32_e32 occurs.
The vcc use of this instruction must be fixed into a vcc_lo use for wave32.
This fix only happens after the final conversion to the SDWA form. This led
to a compiler error in situations where the conversion to SDWA aborts.
Make sure that the vcc-fix gets applied even if the SDWA conversion is
not completed.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Add inreg test for sgpr purpose
This is the second PR after
https://github.com/llvm/llvm-project/pull/135729.
To test sgpr inputs and outputs, using inreg cases for bit-conversions
---------
Co-authored-by: Matt Arsenault <Matthew.Arsenault@amd.com>
This reapplies 067caaa and 382a085 (reverting b35f6e2) with fixes to
issues detected by the address sanitizer (MIs have to be removed from
live intervals before being removed from their parent MBB).
Original commit description below.
AMDGPU scheduler's `PreRARematStage` attempts to increase function
occupancy w.r.t. ArchVGPR usage by rematerializing trivial
ArchVGPR-defining instruction next to their single use. It first
collects all eligible trivially rematerializable instructions in the
function, then sinks them one-by-one while recomputing occupancy in all
affected regions each time to determine if and when it has managed to
increase overall occupancy. If it does, changes are committed to the
scheduler's state; otherwise modifications to the IR are reverted and
the scheduling stage gives up.
In both cases, this scheduling stage currently involves repeated queries
for up-to-date occupancy estimates and some state copying to enable
reversal of sinking decisions when occupancy is revealed not to
increase. The current implementation also does not accurately track
register pressure changes in all regions affected by sinking decisions.
This commit refactors this scheduling stage, improving RP tracking and
splitting the stage into two distinct steps to avoid repeated occupancy
queries and IR/state rollbacks.
- Analysis and collection (`canIncreaseOccupancyOrReduceSpill`). The
number of ArchVGPRs to save to reduce spilling or increase function
occupancy by 1 (when there is no spilling) is computed. Then,
instructions eligible for rematerialization are collected, stopping as
soon as enough have been identified to be able to achieve our goal
(according to slightly optimistic heuristics). If there aren't enough of
such instructions, the scheduling stage stops here.
- Rematerialization (`rematerialize`). Instructions collected in the
first step are rematerialized one-by-one. Now we are able to directly
update the scheduler's state since we have already done the occupancy
analysis and know we won't have to rollback any state. Register
pressures for impacted regions are recomputed only once, as opposed to
at every sinking decision.
In the case where the stage attempted to increase occupancy, and if both
rematerializations alone and rescheduling after were unable to improve
occupancy, then all rematerializations are rollbacked.
This is a NFC patch.
This patch duplicate GFX11plus runlines and apply them with
"+mattr=+real-true16" and "+mattr=-real-true16" on more gfx11/gfx12
tests. And then update the test with the update script
Moves kernarg preload logic to its own module pass. Cloned function
declarations are removed when preloading hidden arguments. The inreg
attribute is now added in this pass instead of AMDGPUAttributor. The
rest of the logic is copied from AMDGPULowerKernelArguments which now
only check whether an arguments is marked inreg to avoid replacing
direct uses of preloaded arguments. This change requires test updates to
remove inreg from lit tests with kernels that don't actually want
preloading.
And related "[AMDGPU] Regenerate mfma-loop.ll test"
Introduce memory error detected by Asan #125885.
This reverts commit 382a085a95b0abeac77b150b7b644b372bd08e78.
This reverts commit 067caaafb58a156d0d77229422607782a639f5b5.
We had custom lowering for the wider vectors of f16, but missing
test coverage for them. Also add more vector tests for bf16, and
split the bf16 cases into separate files so we can add globalisel
run lines.
Flow blocks are generated code that don't really correspond to any
location in the source, so principally they should have empty DebugLocs.
Practically, setting these debug locs leads to redundant is_stmts being
generated after #108251, causing stepping test failures in the ROCm GDB
test suite.
Fixes SWDEV-502134
This is a NFC patch.
Add '-mattr=-real-true16' to gfx11/gfx12 test.
GISEL is not fully supported in true16 mode yet. However we might want
to turn on true16 mode for SDAG as default first. This patch is
preparing for this mode shift in the short future so we can have a small
patch to turn it on
Update the f64 to f16 lowering for targets which support f16 types.
For unsafe mode, lowered to two FP_ROUND. (This patch
https://reviews.llvm.org/D154528 stops from combining these two FP_ROUND
back). In safe mode, select LowerF64ToF16 (round-to-nearest-even
rounding mode)
The initial patch (#135782 caused issues because it emits an error, and llc is sensitive to it.
It also caused compiler-rt/lib/scudo/standalone/tests/wrappers_cpp_test.cpp to fail.
Use warnings instead + reject lowering. That way, the fallback path is used without llc/clang returning a failure code.
If fallback isn't enabled then the warnings provide context as to why lowering failed.
Original commit description for #135782:
Instead of printing something to dbgs (which is not visible to all users),
emit a diagnostic like the DAG does. We still crash later because we fail to
select the inline assembly, but at least now users will know why it's crashing.
In a future patch we could also recover from the error like the DAG does, so the
lowering can keep going until it either crashes or gives a different error later.
AMDGPU scheduler's `PreRARematStage` attempts to increase function
occupancy w.r.t. ArchVGPR usage by rematerializing trivial
ArchVGPR-defining instruction next to their single use. It first
collects all eligible trivially rematerializable instructions in the
function, then sinks them one-by-one while recomputing occupancy in all
affected regions each time to determine if and when it has managed to
increase overall occupancy. If it does, changes are committed to the
scheduler's state; otherwise modifications to the IR are reverted and
the scheduling stage gives up.
In both cases, this scheduling stage currently involves repeated queries
for up-to-date occupancy estimates and some state copying to enable
reversal of sinking decisions when occupancy is revealed not to
increase. The current implementation also does not accurately track
register pressure changes in all regions affected by sinking decisions.
This commit refactors this scheduling stage, improving RP tracking and
splitting the stage into two distinct steps to avoid repeated occupancy
queries and IR/state rollbacks.
- Analysis and collection (`canIncreaseOccupancyOrReduceSpill`). The
number of ArchVGPRs to save to reduce spilling or increase function
occupancy by 1 (when there is no spilling) is computed. Then,
instructions eligible for rematerialization are collected, stopping as
soon as enough have been identified to be able to achieve our goal
(according to slightly optimistic heuristics). If there aren't enough of
such instructions, the scheduling stage stops here.
- Rematerialization (`rematerialize`). Instructions collected in the
first step are rematerialized one-by-one. Now we are able to directly
update the scheduler's state since we have already done the occupancy
analysis and know we won't have to rollback any state. Register
pressures for impacted regions are recomputed only once, as opposed to
at every sinking decision.
In the case where the stage attempted to increase occupancy, and if both
rematerializations alone and rescheduling after were unable to improve
occupancy, then all rematerializations are rollbacked.
Instead of printing something to dbgs (which is not visible to all users),
emit a diagnostic like the DAG does. We still crash later because we fail to
select the inline assembly, but at least now users will know why it's crashing.
In a future patch we could also recover from the error like the DAG does, so the
lowering can keep going until it either crashes or gives a different error later.
[[Change-Id:
I36894fc36e6e6214930fae67f2ca35999abf3b88](https://github.com/llvm/llvm-project/pull/132137)
](https://github.com/llvm/llvm-project/pull/132137) landed, but was
breaking bots. In the rebase, I accidentally brought in unintended
changes. I have reverted the commit
8b9ae65d51
and fixed the rebase.
This PR is to land the original patch / extension to the test.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Register assembly printer passes in the pass registry.
This makes it possible to use `llc -start-before=<target>-asm-printer ...` in tests.
Adds a `char &ID` parameter to the AssemblyPrinter constructor to allow
targets to use the `INITIALIZE_PASS` macros and register the pass in the
pass registry. This currently has a default parameter so it won't break
any targets that have not been updated.
This provides the `disable-schedmodel-in-sched-mi` flag. Using this, we
will disable the SchedModel / Itineraries during scheduling. This has
the effect of not using any latency / hardware resource information for
scheduling decisions.
We have the `schedmodel` flag, but this disables the `SchedModel` for
all passes. This allows disabling only for scheduling while preserving
the behavior of other passes (e.g. MachineLICM). This is conceptually
similar to other flags like `enable-aa-sched-mi`
i16 can be selected into sgpr32 or vgpr16 in isel lowering in true16
mode. And thus, it creates cases that we copy from vgpr16 to sgpr32 in
ext selection and this seems inevitable without sgpr16 support.
legalize the src/dst reg when we decide to lower this special copy to a
readfirstlane in fix-sgpr-copy pass and add a lit test
Extension of https://reviews.llvm.org/D141101 to remove the define flag
from the last stack memory access. Fixes case where COPY instructions
are used for some of the stack restoration, but the copies get optimized
away during the machine-cp pass.
Prior to this change, was possible to produce the following code:
$agpr16_agpr17_agpr18_agpr19 = SCRATCH_LOAD_DWORDX4_ST 64, 0, implicit
$exec, implicit $flat_scr, implicit-def
$agpr16_agpr17_agpr18_agpr19_agpr20_agpr21_agpr22_agpr23_agpr24_agpr25_agpr26_agpr27_agpr28_agpr29_agpr30_agpr31
:: (load (s128) from %stack.17, align 4, addrspace 5)
$agpr20_agpr21_agpr22_agpr23 = SCRATCH_LOAD_DWORDX4_ST 80, 0, implicit
$exec, implicit $flat_scr :: (load (s128) from %stack.17 + 16, align 4,
addrspace 5)
$agpr24_agpr25_agpr26_agpr27 = SCRATCH_LOAD_DWORDX4_ST 96, 0, implicit
$exec, implicit $flat_scr :: (load (s128) from %stack.17 + 32, align 4,
addrspace 5)
$agpr31 = COPY $agpr112, implicit
$agpr16_agpr17_agpr18_agpr19_agpr20_agpr21_agpr22_agpr23_agpr24_agpr25_agpr26_agpr27_agpr28_agpr29_agpr30_agpr31
$agpr30 = COPY $agpr208, implicit
$agpr16_agpr17_agpr18_agpr19_agpr20_agpr21_agpr22_agpr23_agpr24_agpr25_agpr26_agpr27_agpr28_agpr29_agpr30_agpr31
$agpr28_agpr29 = SCRATCH_LOAD_DWORDX2_ST 112, 0, implicit $exec,
implicit $flat_scr, implicit-def
$agpr16_agpr17_agpr18_agpr19_agpr20_agpr21_agpr22_agpr23_agpr24_agpr25_agpr26_agpr27_agpr28_agpr29_agpr30_agpr31
:: (load (s64) from %stack.17 + 48, align 4, addrspace 5)
where `$agpr30 = COPY $agpr208` would be optimized away by `machine-cp`
pass. Instead, change to:
$agpr28_agpr29 = SCRATCH_LOAD_DWORDX2_ST 112, 0, implicit $exec,
implicit $flat_scr :: (load (s64) from %stack.17 + 48, align 4,
addrspace 5)
Fixes#131386.
Made the simple fix, but I'm not completely comfortable with this change
since the reason for the previous inclusion of `IsLastSubReg` is unclear
to me.
@krzysz00
This is a NFC patch.
Clean up three test for true16 mode:
1. remove strayed test line
2. remove t16 test line from fake16 mir test
3. update check-label to shrink test size
The VOP3 form of the V_CNDMASK_B32 instruction takes a carry-in
operand. The conversion to SDWA implies a conversion to VOP2 form
which reads from VCC instead.
Convert V_CNDMASK_B32_e64 instructions that might be converted to SDWA
to V_CNDMASK_B32_e32 first and introduce a copy of the carry-in operand
to VCC.
Closes#133431.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Legalize the amdgcn.dead intrinsic to work with types other than i32. It
still generates IMPLICIT_DEFs.
Remove some of the previous code for selecting/reg bank mapping it for
32-bit types, since everything is done in the legalizer now.
