Fixes https://github.com/llvm/llvm-project/issues/158415.
After `resolveTargetShuffleInputsAndMask` and other modifications on
`Ops` and `Mask`, unused inputs in `Ops` are erased, and may leave `Ops`
empty. This patch handles such cases before calling the final
`combineX86ShuffleChain`。
There is a cache on the known-bit computed by global-isel. It only works
inside a single query to computeKnownBits, which limits its usefulness,
and according to the tests can sometimes limit the effectiveness of
known-bits queries. (Although some AMD tests look longer). Keeping the
cache valid and clearing it at the correct times can also require being
careful about the functions called inside known-bits queries.
I measured compile-time of removing it and came up with:
```
7zip 2.06405E+11 2.06436E+11 0.015018992
Bullet 1.01298E+11 1.01186E+11 -0.110236169
ClamAV 57942466667 57848066667 -0.16292023
SPASS 45444466667 45402966667 -0.091320249
consumer 35432466667 35381233333 -0.144594317
kimwitu++ 40858833333 40927933333 0.169118877
lencod 70022366667 69950633333 -0.102443457
mafft 38439900000 38413233333 -0.069372362
sqlite3 35822266667 35770033333 -0.145812474
tramp3d 82083133333 82045600000 -0.045726
Average -0.068828739
```
The last column is % difference between with / without the cache. So in
total it seems to be costing slightly more to keep the current
known-bits cache than if it was removed. (Measured in instruction count,
similar to llvm-compile-time-tracker).
The hit rate wasn't terrible - higher than I expected. In the
llvm-test-suite+external projects it was hit 4791030 times out of
91107008 queries, slightly more than 5%.
Note that as globalisel increases in complexity, more known bits calls
might be made and the numbers might shift. If that is the case it might
be better to have a cache that works across calls, providing it doesn't
make effectiveness worse.
After #144666 we now support vectorizing loops with induction variables
with EVL tail folding. The induction updates don't use VP intrinsics to
avoid VL toggles but instead rely on RISCVVLOptimizer. However
RISCVVLOptimizer can't reason about cycles or recurrences today, which
means we are left with a VL toggle to VLMAX:
# %bb.1: # %for.body.preheader
li a2, 0
vsetvli a3, zero, e32, m2, ta, ma
vid.v v8
.LBB0_2: # %vector.body
# =>This Inner Loop Header: Depth=1
sub a3, a1, a2
sh2add a4, a2, a0
vsetvli a3, a3, e32, m2, ta, ma
vle32.v v10, (a4)
add a2, a2, a3
vadd.vv v10, v10, v8
vse32.v v10, (a4)
vsetvli a4, zero, e32, m2, ta, ma
vadd.vx v8, v8, a3
bne a2, a1, .LBB0_2
This patch teaches RISCVVLOptimizer to reason about recurrences so we
can remove the VLMAX toggle:
# %bb.1: # %for.body.preheader
li a2, 0
vsetvli a3, zero, e32, m2, ta, ma
vid.v v8
.LBB0_2: # %vector.body
# =>This Inner Loop Header: Depth=1
sub a3, a1, a2
sh2add a4, a2, a0
vsetvli a3, a3, e32, m2, ta, ma
vle32.v v10, (a4)
add a2, a2, a3
vadd.vv v10, v10, v8
vse32.v v10, (a4)
vadd.vx v8, v8, a3
bne a2, a1, .LBB0_2
With this we remove a significant number of VL toggles and vsetvli
instructions across llvm-test-suite and SPEC CPU 2017 with tail folding
enabled, since it affects every loop with an induction variable.
This builds upon the work in #124530 where we started computing what VL
each instruction demanded, and generalizes it to an optimistic sparse
dataflow analysis:
- We begin by optimistically assuming no VL is used by any instruction,
and push instructions onto the worklist starting from the bottom.
- For each instruction on the worklist we apply the transfer function,
which propagates the VL needed by that instruction upwards to the
instructions it uses. If a use's demanded VL changes, it's added to the
worklist.
- Eventually this converges to a fixpoint when all uses have been
processed and every demanded VL has been propagated throughout the
entire use-def chain. Only after this is the DemandedVL map accurate.
Some implementation details:
- The roots are stores (or other unsupported instructions not in
`isSupportedInstr`) or copies to physical registers (they fail the
`any_of(MI.defs(), isPhysical)` check)
- This patch untangles `getMinimumVLForUser` and `checkUsers`.
`getMinimumVLForUser` now returns how many lanes of an operand are read
by an instruction, whilst `checkUsers` checks that an instruction and
its users have compatible EEW/EMULs.
- The `DemandedVL` struct was added so that we have a default
constructor of 0 for `DenseMap<const MachineInstr *, DemandedVL>
DemandedVLs`, so we don't need to check if a key exists when looking
things up.
There was no measurable compile time impact on llvm-test-suite or SPEC
CPU 2017. The analysis will always terminate, there are more details in
this EuroLLVM talk here:
https://www.youtube.com/watch?v=Mfb5fRSdJAcFixes#149354
Since many code are connected, this also changes how workgroup id is lowered.
Co-authored-by: Jay Foad <jay.foad@amd.com>
Co-authored-by: Ivan Kosarev <ivan.kosarev@amd.com>
Add hasVendorXTHeadCondMov to shouldFoldSelectWithSingleBitTest.
The optimizations in these functions is equally applicable to these.
I changed the RUN line for xtheadcondmove in condops.ll to use XTHeadBs
to get coverage of the hasBEXTILike changes. I didn't think it was
worth an additional RUN line and check prefix.
This change was motivated by CK where many VMCNT(0)'s were generated due
to instructions lacking !alias.scope metadata. The two causes of this
were:
1) LowerLDSModule not tacking on scope metadata on a single LDS variable
2) IPSCCP pass before inliner replacing noalias ptr derivative with a
global value, which made inliner unable to track it back to the noalias
ptr argument.
However, it turns out that IPSCCP losing the scope information was
largely ineffectual as ScopedNoAliasAA was able to handle asymmetric
condition, where one MemLoc was missing scope, and still return NoAlias
result.
AMDGPU however was checking for existence of scope in SIInsertWaitcnts
and conservatively treating it as aliasing all and inserted VMCNT(0)
before DS_READs, forcing it to wait for all previous LDS DMA
instructions.
Since we know that ScopedNoAliasAA can handle asymmetry, we should also
allow AA query to determine if two MIs may alias.
Passed PSDB.
Previous attempt to address the issue in IPSCCP, likely stalled:
https://github.com/llvm/llvm-project/pull/154522
This solution may be preferrable over that as issue only affects AMDGPU.
The neutral values for these are -1U, and 0 respectively. We already
have good arithmetic lowerings for selects with one arm equal to these
values. smin/smax are a bit harder, and will be a separate change.
Somewhat surprisingly, this looks to be a net code improvement in all of
the configurations. With both zbb, it's a clear win. With only zicond,
we still seem to come out ahead because we reduce the number of ziconds
needed (since we lower min/max to them). Without either zbb or zicond,
we're a bit more of wash, but the available arithmetic sequences are
good enough that doing the select unconditionally before using branches
for the min/max is probably still worthwhile?
This pull request improves support for scalar floating-point conversions
from integer vectors on AArch64, specifically for the `scvtf` and
`ucvtf` instructions. It fixes pattern matching so that single-element
conversions from vectors now generate the expected scalar instructions
and adds a new test to verify correct behavior for extracting a lane
from a widened vector.
**Pattern matching and code generation improvements:**
* Added new patterns in `AArch64InstrInfo.td` to correctly match
conversions from `v2i32` to `v1f64` using `scvtf` and `ucvtf`, ensuring
the scalar instructions (`scvtf d0, s0` and `ucvtf d0, s0`) are
generated when extracting a single lane.
**Test updates and additions:**
* Updated `scvtf_f64i32_simple` and `ucvtf_f64i32_simple` tests in
`fprcvt-cvtf.ll` to reflect the correct generation of scalar
instructions, removing previous comments about incorrect codegen and
showing the expected output.
* Added a new test `uitofp_sext_v2i32_extract_lane0` to verify correct
code generation when extracting a lane from a widened vector and
converting to double.
Builtins for reading the streaming vector length are canonicalised to
use the aarch64.sme.cntsd intrinisic and a multiply, i.e.
- cntsb -> cntsd * 8
- cntsh -> cntsd * 4
- cntsw -> cntsd * 2
This patch also removes the LLVM intrinsics for cnts[b,h,w], and adds
patterns to improve codegen when cntsd is multiplied by a constant.
I have no idea why this was here. MIMG atomics use tied operands
for the input and output, so AV classes should have always worked.
We have poor test coverage for AGPRs with atomics, so add a partial
set. Everything seems to work OK, although it seems image cmpswap
always uses VGPRs unnecessarily.
We currently only support partial.reduce.add in the case where we are
performing a multiply-accumulate. Now add support for any partial
reduction where the input is being extended, where we can take advantage
of extadd_pairwise.
Pre-commit test case for exploitation of `xxsel` for ternary operations
of the pattern. This adds support for v4i32, v2i64, v16i8 and v8i16
operand types for the following patterns.
The following are the patterns involved in the change:
```
ternary(A, and(B,C), nor(B,C))
ternary(A, B, nor(B,C))
ternary(A, C, nor(B,C))
ternary(A, xor(B,C), nor(B,C))
ternary(A, not(C), nor(B,C))
ternary(A, not(B), nor(B,C))
ternary(A, nand(B,C), nor(B,C))
ternary(A, or(B,C), eqv(B,C))
ternary(A, nor(B,C), eqv(B,C))
ternary(A, not(C), eqv(B,C))
ternary(A, nand(B,C), eqv(B,C))
ternary(A, and(B,C), not(C))
ternary(A, B, not(C))
ternary(A, xor(B,C), not(C))
ternary(A, or(B,C), not(C))
ternary(A, not(B), not(C))
ternary(A, nand(B,C), not(C))
ternary(A, and(B,C), not(B))
ternary(A, xor(B,C), not(B))
ternary(A, or(B,C), not(B))
ternary(A, nand(B,C), not(B))
ternary(A, B, nand(B,C))
ternary(A, C, nand(B,C))
ternary(A, xor(B,C), nand(B,C))
ternary(A, or(B,C), nand(B,C))
ternary(A, eqv(B,C), nand(B,C))
```
Exploitation of `xxeval` for the above patterns to be added as a follow
up.
Co-authored-by: Tony Varghese <tony.varghese@ibm.com>
Find a common subclass instead of directly checking for a subclass
relationship. This fixes folding logic for unaligned register defs
into aligned use contexts. e.g., a vreg_64 def into an av_64_align2
use should be able to find the common subclass vreg_align2. This
avoids regressions in future patches.
Checking the subclass was also redundant on the subregister path;
getMatchingSuperRegClass is sufficient.
Fix regression after 62f2641d603db9aef99dd5c434a1dfe7d3f56346. Previous
patch handled the register case, but the memory case snuck another use
of ptr_rc_tailcall hidden inside i64mem_TC
Restrict to VGPR only (VRegSrc_32) for mfma scale operands to workaround
a hardware design defect: For all Inline/SGPR constants, SP HW use bits
[30:23] as the scale.
TODO: We may still be able to allow Inline Constants/SGPR, with a proper
shift, to obtain a potentially better performance.
Fixes: SWDEV-548629
This adds some basic verification for the new OPERAND_SHIFT_MSL and the
existing OPERAND_IMPLICIT_IMM_0 immediate operand types, that should be
264/272 or 0 respectively.
The existing isel pattern felt like it was emitting more instructions
than an isel pattern probably should. We were also missing opportunities
to fold the innermost instructions with surrounding instructions.
I tried to move the expansion to lowering, but we got a little too
aggressive folding the (not (slli (not))) with other operations in some
tests and created code with constants that are hard to materialize and
missed using TH_FF0. We could probably have fixed that with a
RISCVISD::TH_FF0 node.
While investigating, I tried using the default promotion. The results
aren't obviously worse than the previous codegen. And in some case they
are obviously better.
Some passes like LoopSimplify/SimplifyCFF are running between IRPasses
and ISelPrepare. This is an issue because the structurizer generates
OpSelectionMerge/OpLoopMerge instructions at specific places, and those
passes are moving them.
Moving the structurizer later solves this issue.
The optimisation in canRemovePTestInstr tries to remove ptest instructions when
the predicate is the result of a WHILEcc. This patch extends the support to
WHILEcc (predicate pair) by:
- Including the WHILEcc_x2 intrinsics in isPredicateCCSettingOp, allowing
performFirstTrueTestVectorCombine to create the PTEST.
- Setting the isWhile flag for the predicate pair instructions in tablegen.
- Looking through copies in canRemovePTestInstr to test isWhileOpcode.
- Enable s_or_b64/s_and_b64/s_xor_b64 for v2i32. Add various additional
combines to make use of these newly legalised instructions.
- Update several tests and separate legacy r600 tests where necessary.
This pr removes the falling back to SDISel of rvv intrinsics and marks
them legalized in the legalize pass. Another pr would be created for
regbankselect pass to make vf form intriniscs have the right scalar
register bank.
Add G_AMDGPU_LOAD_D16 generic instructions and GINodeEquivs for them,
this will import D16 load patterns to global-isel's tablegened
instruction selector.
For newly imported patterns to work add combines for G_AMDGPU_LOAD_D16
in AMDGPURegBankCombiner.
Cover all the missing cases and add very detailed tests for each rule.
In summary:
- Flat and Scratch, addrspace(0) and addrspace(5), loads are always
divergent.
- Global and Constant, addrspace(1) and addrspace(4), have real uniform
loads, s_load, but require additional checks for align and flags in mmo.
For not natural align or not uniform mmo do uniform-in-vgpr lowering.
- Private, addrspace(3), only has instructions for divergent load, for
uniform do uniform-in-vgpr lowering.
- Store rules are simplified using Ptr32 and Ptr64.
All operands need to be vgpr.
Some tests have code size regression since they use more sgpr instructions,
marked with FixMe comment to get back to later.
This change introduces legalization for `G_ATOMIC_CMPXCHG` and
`G_ATOMIC_CMPXCHG_WITH_SUCCESS`. Additionally, support for the
`riscv_masked_cmpxchg intrinsic` is added to legalizeIntrinsic, ensuring
that masked compare-and-exchange operations are recognized during
legalization.
---------
Co-authored-by: Kane Wang <kanewang95@foxmail.com>
Similar to the implementation in
https://github.com/llvm/llvm-project/pull/104411 , the `fmin.s`/`fmax.s`
instructions follow IEEE 754-2019 semantics, and
`G_FMINIMUMNUM`/`G_FMAXIMUMNUM` are legal.
