This fixes an allocation failure in the new test.
In cases where getLargestLegalSuperClass can inflate the register class,
rematerialization could effectively undo a split which was done to
inflate
the register class, if the defining instruction can only write a
subclass
and the use can read the superclass.
Some of the x86 tests changes look like improvements, but some are
likely regressions.
I'm not entirely sure this is the correct place to fix this. It also
seems more complicated than necessary, but the decision to change
the register class is far removed from the point where the decision
to split the virtual register is made. I'm also also not sure if this
should be considering the register classes of all the use indexes
in getUseSlots, rather than just checking if this use index instruction
reads the register.
Failure to sink a candidate should not block us from attempting to sink
other candidates. There are mechanisms in place to handle the case where
the failed to be sunk instruction uses an instruction that gets sunk (we
do not delete the original instruction corresponding with the sunk
instruction if it still has uses).
The si-peephole-sdwa pass adjusts the selections on sdwa instructions to
the selections on their operands during its conversions. For instance,
if an instruction selects `BYTE_0` and its operand selects `WORD_1`, the
combined selection should be `BYTE_2`, i.e. "`BYTE_0` of `WORD_1`". The
existing implementation does not always handle this correctly in some
complex situations with instructions across different basic blocks as
demonstrated by the test cases included in this PR.
This PR adds an additional selection combination step to the conversion
to fix this issue. It reverts the changes made by PR #123942 which had
disabled the conversion of preexisting SDWA instructions completely as a
quick fix.
---------
Co-authored-by: Jeffrey Byrnes <Jeffrey.Byrnes@amd.com>
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
This was trying to hack around the intermediate VGPR requirement
to copy to AGPRs on gfx908. We should still use a copy for all
reg-to-reg cases. This should matter less these days, as we
reserve a VGPR to handle it when required (and no end to end tests
need updating).
This was also an obstacle to handling this fold for input registers
which are larger than 32-bits.
If the ControlFlowHub did not perform any change to the control flow,
there is no need to repair SSA, update the loop structure, and verify a
bunch of things. This is not completely NFC though, repairSSA introduced
PHI nodes with a single entry that are now missing.
My code went from 400+ seconds to 1 second, since no loop required the
exits to be unified, but there were many "complex" loops.
True16 Add OpSel when optimizing exec mask
True16 VOPCX have the opsel argument. Add it when we create these
instructions in SIOptimizeExecMasking.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Support true16 format for v_alignbit_b32 in MC.
Since we are replacing `v_alignbit_b32` to
`v_alignbit_b32_t16/v_alignbit_b32_fake16` in Post-GFX11, have to update
the CodeGen pattern for `v_alignbit_b32_fake16` to get CodeGen test
passing. There is no pattern modified/created, but just replacing the
`v_alignbit_b32` with fake16 format.
Some of the true16 CodeGen test are impacted since `v_alignbit_b32`
selection are removed in Post-GFX11 while `v_alignbit_b32_t16` are not
yet supported. The CodeGen patch for `v_alignbit_b32_t16` will be done
in the following patch.
I have a pending patch which improves the codegen in the original IR
version, such that the allocation no longer fails. I'm still trying
to preserve the failure from IR, but add a version with a snapshot
of the current MIR before the failing RA run.
The MI scheduler skips regions containing a single MI during scheduling.
This can prevent targets that perform multi-stage scheduling and move
MIs between regions during some stages to reason correctly about the
entire IR, since some MIs will not be assigned to a region at the
beginning.
This makes the machine scheduler no longer skip single-MI regions. Only
a few unit tests are affected (mainly those which check for the
scheduler's debug output).
The real legality check is peformed later anyway, so this was
unnecessarily blocking immediate folds in handled cases.
This also stops folding s_fmac_f32 to s_fmamk_f32 in a few tests,
but that seems better. The globalisel changes look suspicious,
it may be mishandling constants for VOP3P instructions.
There's little point to trying to commute an instruction if the
two operands are already the same.
This avoids an assertion in a future patch, but this likely isn't the
correct fix. The worklist management in SIFoldOperands is dodgy, and
we should probably fix it to work like PeepholeOpt (i.e. stop looking
at use lists, and fold from users). This is an extension of the already
handled special case which it's trying to avoid folding an instruction
which is already being folded.
Since LowerBufferFatPointers runs before PreISelIntrinsicLowering, which
normally handles unsupported memcpy()s,, and since you can't have a
`noalias {ptr addrspace(8), i32}` becasue it crashes later passes,
manually expand memcpy()s involving buffer fat pointers to loops.
Additionally, though they're unlikely to be used, this commit adds
support for memset().
This commit doesn't implement writing direct-to-LDS loads as the
intrinsics, but leaves the option in the future.
Enables 16-bit values to be spilled to scratch.
Note, the memory instructions used are defined as reading and writing
VGPR_32, but do not clobber the unspecified 16-bits of those registers,
and so spills and reloads of lo and hi halves of the registers work.
M0 can only be written to by the SALU, so `v_readfirstlane_b32 m0` is
effectively useless. Represent this by restricting the dest RC of that
instruction to `SReg_32_XM0` which excludes M0.
There is a lot of test changes due to the register class changing, but
most changes are trivial. In some cases, an extra register and
`s_mov_b32` is needed.
Fixes SWDEV-513269
In DAGCombiner.cpp preserve range metadata when load is narrowed to load
LSBs if original range metadata bounds can fit in the narrower type.
Utilize preserved range metadata to reduce 64-bit shl to 32-bit shl.
---------
Signed-off-by: John Lu <John.Lu@amd.com>
This fixes an assert after allocation failure.
Rather than collecting failed virtual registers and hacking
on the uses after the fact, directly hack on the uses and rewrite
the registers to the dummy assignment immediately.
Previously we were bypassing LiveRegMatrix and directly assigning
in the VirtRegMap. This resulted in inconsistencies where illegal
overlapping assignments were missing. Rather than try to hack in
some system to manage these in LiveRegMatrix (i.e. hacking around
cases with invalid iterators), avoid this by directly using the
physreg. This should also allow removal of special casing in
virtregrewriter for failed allocations.
Reapply "RegAlloc: Fix verifier error after failed allocation (#119690)"
This reverts commit 0c50054820799578be8f62b6fd2cc3fbc751c01e.
Reapply with more fixes to avoid expensive_checks failures. Make sure to
call splitSeparateComponents after shrinkToUses, and update the VirtRegMap
with the split registers. Also set undef on all physical register aliases to
the assigned register.
Move physreg handling. Not sure if necessary
Remove intervals from regunits. Not sure if necessary
Even if an inline asm doesn't have memory effects, we can't assume it's
safe to speculate: it could trap, or cause undefined behavior. At the
LLVM IR level, this is handled correctly: we don't speculate inline asm
(unless it's marked "speculatable", but I don't think anyone does that).
Codegen also needs to respect this restriction.
This change stops Early If Conversion and similar passes from
speculating an INLINEASM MachineInstr.
Some uses of isSafeToMove probably could be switched to a different API:
isSafeToMove assumes you're hoisting, but we could handle some forms of
sinking more aggressively. But I'll leave that for a followup, if it
turns out to be relevant.
See also discussion on gcc bugtracker
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102150 .
Enable gisel selection for uaddsat and usubsat in true16 flow
This patch includes:
1. Added VGPR_16_Lo128/VGPR_16 to register bank and update register info
for recognizing 16bit regclass id and bit width
2. uaddsat/usubsat test update
The Target hook convertSelectOfConstantsToMath() needs to be used within
SimplifySelectCC helper combine function in SelectionDAG Isel, where
generic select folding with constants is happening into simple maths op
using the condition as it is.
It necessarily fixes#121145.
There were 2 parallel fold check mechanisms, so consistently use the
fold list. The worklist management here is still not good. Other types
of folds are not using it, and we should probably rewrite the pass to
look more like peephole-opt.
This should be an alternative fix to skipping commute if the operands
are the same (#127562). The new test is still not broken as-is, but
demonstrates failures in a future patch.
In case of variable offset of a GEP that can be optimized out, promote
alloca is updated to use the refereshed index to avoid an assertion.
Issue found by fuzzer.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
We need to promote 8/16-bit cases to 32-bit. Unfortunately we are
missing demanded bits optimizations on readfirstlane, so we end up
emitting
an and instruction on the input. I'm also surprised this pass isn't
handling
half or bfloat yet.
The hasOneUse check was failing in any case where the load was part of a chain - we should only be checking if the loaded value has one use, and any updates to the chain should be handled by the fold calling shouldReduceLoadWidth.
I've updated the x86 implementation to match, although it has no effect here yet (I'm still looking at how to improve the x86 implementation) as the inner for loop was discarding chain uses anyway.
By using SDValue::hasOneUse instead this patch exposes a missing dependency on the LLVMSelectionDAG library in a lot of tools + unittests, which resulted in having to make SDNode::hasNUsesOfValue inline.
Noticed while fighting the x86 regressions in #122671
Remove the MIR scan to detect whether AGPRs are used or not,
and the special case for callable functions. This behavior was
confusing, and not overridable. The amdgpu-no-agpr attribute was
intended to avoid this imprecise heuristic for how many AGPRs to
allocate. It was also too confusing to make this interact with
the pending amdgpu-num-agpr replacement for amdgpu-no-agpr.
Also adds an xfail-ish test where the register allocator asserts
after allocation fails which I ran into.
Future work should reintroduce a more refined MIR scan to estimate
AGPR pressure for how to split AGPRs and VGPRs.
T16D16 table is implemented in
https://github.com/llvm/llvm-project/pull/127673
this is a follow up patch to add load/store pseudo for:
flat_store
global_load/global_store
scratch_load/scratch_store
in true16 mode and updated the codegen test file
