The grouped vector register is modeled as a single register, e.g. V2M2,
which is actually V2 and V3. We need to decompose the grouped vector
register(if any) to individual vector register when emitting CFIs in
prologue.
Fixed https://github.com/llvm/llvm-project/issues/94500
Ventana doesn't have RV32 cores so the instructions aren't really
supported for RV32, but there's nothing specifically 64-bit about them.
My goal here is to fix cannot select errors if XVentanaCondOps is
enabled on RV32. Alternatively, we could quality the lowering code to
also check IsRV64 so that we don't create RISCVISD::CZERO* nodes. Fixing
the isel patterns seemed simpler.
Fixes#100855.
This patch folds `fmul X, (fcopysign 1.0, Y)` into `fsgnjx X, Y`. This
pattern exists in some graphics applications/math libraries.
Alive2: https://alive2.llvm.org/ce/z/epyL33
Since fpimm +1.0 is lowered to a load from constant pool after
OpLegalization, I have to introduce a new RISCVISD node FSGNJX and fold
this pattern in DAGCombine.
Closes https://github.com/dtcxzyw/llvm-opt-benchmark/issues/1072.
We can teach RISCVVectorPeephole to detect when an AVL is equal to the
VLMAX when the exact VLEN is known and use the VLMAX sentinel instead,
and in doing so remove the need for getVLOp in RISCVISelLowering. This
keeps all the VLMAX logic in one place.
A lot of cases have differing AVLs which aren't foldable, update them so the peephole triggers on them and add explicit cases for non-foldable AVLs.
Also rename it to vmv.v.v-peephole.ll since it's not actually a DAG combine.
And remove a TODO, it's correct to fold if the two passthrus are the same.
RISCVGatherScatterLowering is the last user of
riscv_masked_strided_{load,store} after #98131 and #98112, this patch
changes it to emit the VP equivalent instead. This allows us to remove
the masked_strided intrinsics so we have only have one lowering path.
riscv_masked_strided_{load,store} didn't have AVL operands and were
always VLMAX, so this passes in the fixed or scalable element count to
the EVL instead, which RISCVVectorPeephole should now convert to VLMAX
after #97800.
For loads we also use a vp_select to get passthru (mask undisturbed)
behaviour
This adds a new vector peephole that converts unmasked, VLMAX
vleN.v/vseN.v to their whole register equivalents.
It replaces the existing tablegen patterns on ISD::LOAD/ISD::STORE and
is a bit more general since it also catches VP loads and stores and
@llvm.riscv intrinsics.
The heavy lifting of detecting a VLMAX AVL and an all-ones mask is
already taken care of by existing peepholes.
Rebase of #84114. I've only included the core changes to frame layout
calculation & CFI generation which sidesteps the regressions found after
merging #84114. Since these changes are a necessary precursor to the
overall fix and are themselves slightly beneficial as CFI is now
generated correctly, I think it is reasonable to merge this first step.
---
For very large stack frames, the offset from the stack pointer to a
local can be more than 2^31 which overflows various `int` offsets in the
frame lowering code.
This patch updates the frame lowering code to calculate the offsets as
64-bit values and fixes CFI to use the corrected sizes.
After this patch, additional work is needed to fix offset truncations in
each target's codegen.
As discussed at the last sync-up call, mark Zacas as experimental until
this ABI issue is resolved
<https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/444>.
Don't return Zacas in getHostCPUFeatures (leaving a TODO there) as even if requesting detection of "native" features, the user likely doesn't want to automatically opt in to experimental codegen.
We need to use the minimum size of the scalable type and the correct
stack ID.
The code in the PR is still invalid because the instruction used doesn't
have a pointer operand. This is diagnosed later when the assembler
parses it.
Fixes#99782
Just like for regular IR we need to treat SELECT as conditionally
blocking poison in SelectionDAG. So (unless the condition itself is
poison) the result is only poison if the selected true/false value is
poison.
Thus, when doing DAG combines that turn SELECT into arithmetic/logical
operations (e.g. AND/OR) we need to make sure that the new operations
aren't more poisonous. One way to do that is to use FREEZE to make
sure the operands aren't posion.
This patch aims at fixing the kind of miscompiles reported in
https://github.com/llvm/llvm-project/issues/84653
and
https://github.com/llvm/llvm-project/issues/85190
Solution is to make sure that we insert FREEZE, if needed to make
the fold sound, when using the foldBoolSelectToLogic and
foldVSelectToSignBitSplatMask DAG combines.
Allow pushing freeze through SETCC and SELECT_CC even if there are
multiple "maybe poison" operands. In the past we have limited it to
a single "maybe poison" operand, but it seems profitable to also
allow the multiple operand scenario.
One goal here is to avoid some regressions seen in review of
https://github.com/llvm/llvm-project/pull/84924
when solving the select->and miscompiles described in
https://github.com/llvm/llvm-project/issues/84653
With correct test update.
Original message:
We were only checking that the node from the worklist is a supported
root. We weren't checking the strategy or any of its operands unless it
was the original node. For any other node, we just rechecked the
original node's strategy and operands.
The effect of this is that we don't do all of the transformations at
once. Instead, when there were multiple possible nodes to transform we
would only do them as each node was visited by the main DAG combine
worklist.
The test shows a case where we widened an instruction without removing
all of the uses of the vsext. The sext is shared by one node that shares
another sext node with the root another node that doesn't share anything
with the root.
We were only checking that the node from the worklist is a supported
root. We weren't checking the strategy or any of its operands unless it
was the original node. For any other node, we just rechecked the
original node's strategy and operands.
The effect of this is that we don't do all of the transformations at
once. Instead, when there were multiple possible nodes to transform we
would only do them as each node was visited by the main DAG combine
worklist.
The test shows a case where we widened an instruction without removing
all of the uses of the vsext. The sext is shared by one node that shares
another sext node with the root another node that doesn't share anything
with the root.
The included test case contains X0 as a def register. X0 is considered a
constant register when it is a use. When its a def, it means to throw
away the result value.
If we treat it as a constant register here, we will execute the continue
and not assign `DefReg` to any register. This will cause a crash when
trying to get the register class for `DefReg` after the loop.
By only checking isConstantPhysReg for uses, we will reach the `return
false` a little further down and stop processing this instruction.
A nxvXi64 VMV_V_X_VL on RV32 sign extends its 32 bit input to 64 bits.
If that input is positive, the sign extend can also be considered as a
zero extend.
Add more patterns isel patterns for vmseq.vi and friends with
the constant splat on the left hand side.
We can't trust the canonicalization in SimplifySetCC to keep constants
to the RHS when the splat is VMV_V_X_VL for i64 on RV32.
When folding, we currently check if the pseudo's result is not lanewise
(e.g. vredsum.vs or viota.m) and bail if we're changing the mask.
However we also need to check for the AVL too.
This patch bails if the AVL changed for these pseudos, and also renames
the pseudo table property to be more explicit.
Like #93321, this patch also tries to solve the conflict usage of x7 for
fastcc and Zicfilp. But this patch removes x7 from fastcc directly. Its
purpose is to reduce the code complexity of #93321, and we also found
that it at most increase 0.02% instruction count for most benchmarks and
it might be benefit for benchmarks.
The previous expansion of [US]CMP was done using two selects and two
compares. It produced decent code, but on many platforms it is better to
implement [US]CMP nodes by performing the following operation:
```
[us]cmp(x, y) = (x [us]> y) - (x [us]< y)
```
This patch adds this new expansion, as well as a hook in TargetLowering to allow some targets to still use the select-based approach. AArch64 and SystemZ are currently the only targets to prefer the former approach, but other targets may also start to use it if it provides for better codegen.
Both of them are ratified now.
https://wiki.riscv.org/display/HOME/Ratified+Extensions
This patch does not set them to non-experimental, since Zicfilp lacks
lld support and Zicfiss also lacks compiler-rt/libunwind support.
A splat of a <n x i64> on RV32 will get lowered as a zero strided load
anyway (and won't match any .vx splat patterns), so don't expand it to a
scalar load + splat to avoid writing it to the stack.
This patch makes zero strided VP loads always be expanded to a scalar
load and splat even if +optimized-zero-stride-load is present.
Expanding it allows more .vx splat patterns to be matched, which is
needed to prevent regressions in #98111.
If the feature is present, RISCVISelDAGToDAG will combine it back to a
zero strided load.
The RV32 test diff also shows how need to emit a zero strided load
either way after expanding an SEW=64 strided load. We could maybe fix
this in a later patch by not doing the expand if SEW>XLEN.
This is a follow up to ff8a03a7. On the review for that, I'd suggested a
stylistic rework, but after discussion we decided to move forward with
the fix as it was. This change is a small part of that suggested rework.
Once I sat down and wrote the code, I think I've convinced myself of an
entirely different approach (tbd), but for the moment, let's use a
lambda to share code so that we can pickup a missed optimization, and
reduce some duplication.
---------
Co-authored-by: Luke Lau <luke_lau@icloud.com>
This is a recommit of #98140. The old commit should be rebased on #98205
which changes the feature of hardware zero stride optimization.
It's a similar patch as a214c521f8763b36dd400b89017f74ad5ae4b6c7 for
vp.stride.load. Some targets prefer pattern (vmv.v.x (load)) instead of
vlse with zero stride.
Currently before forwarding an AVL we do a simple non-exhaustive check
to see if its LiveInterval is extendable. But we also need to check for
this when we're extending an AVL's LiveInterval via merging the
VSETVLIInfos in transferBefore with equally zero AVLs.
Rather than trying to conservatively prevent these cases, this inserts a
copy of the AVL instead if we don't know we'll be able to extend it.
This is likely to be more robust, and even if the extra copy is
undesirable these cases should be rare in practice.
- Add `MachineVerifierPass`.
- Use complete `MachineVerifierPass` in `VerifyInstrumentation` if
possible.
`LiveStacksAnalysis` will be added in future, all other analyses are
done.
Implement XCVmac intrinsics for CV32E40P according to the specification.
This commit is part of a patch-set to upstream the vendor specific
extensions of CV32E40P that need LLVM intrinsics to implement Clang
builtins.
Contributors: @CharKeaney, @ChunyuLiao, @jeremybennett, @lewis-revill,
@NandniJamnadas, @PaoloS02, @serkm, @simonpcook, @xingmingjie.
Most of the time when we coalesce and delete a vsetvli, we shrink the
LiveInterval of its AVL register now that there is one less use. However
there's one edge case we were missing where if we have two vsetvlis with
no users of vl or vtype in between, we coalesced a vsetvli without
shrinking it's AVL.
This fixes it by shrinking the LiveInterval whenever we delete a
vsetvli, and also makes the LiveIntervals consistent in-situ by not
removing the use before shrinking.
This fixes a -verify-machineinstrs assertion in an MIR test case I found
while investigating
https://github.com/llvm/llvm-project/pull/97264#issuecomment-2218036877.
I couldn't recreate this at the LLVM IR level, seemingly because
RISCVInsertVSETVLI will just avoid inserting extra vsetvlis that don't
need coalesced.
Doing so allows one side to fold entirely into the mask applied to the
other recursive call (or a vmerge.vv at worst). This is a generalization
of the existing IsSelect case (both operands are selects), so I removed
that code in the process.
This actually started as an attempt to remove the IsSelect bit as I'd
thought it was fully redundant
with the recursive formulation, but digging into test deltas revealed
that we depended on that
to catch the majority of the identity cases, and that in turn we were
missing some cases where only RHS was an identity.
Given an AVL that's computed from vlenb, if it's equal to VLMAX then we
can replace it with the VLMAX sentinel value.
The main motiviation is to be able to express an EVL of VLMAX in VP
intrinsics whilst emitting vsetvli a0, zero, so that we can replace
llvm.riscv.masked.strided.{load,store} with their VP counterparts.
This is done in RISCVVectorPeephole (previously RISCVFoldMasks, renamed
to account for the fact that it no longer just folds masks) instead of
SelectionDAG since there are multiple places places where VP nodes are
lowered that would have need to have been handled.
This also avoids doing it in RISCVInsertVSETVLI as it's much harder to
lookup the value of the AVL, and in RISCVVectorPeephole we can take
advantage of DeadMachineInstrElim to remove any leftover
PseudoReadVLENBs.
This extension consists of 8 additional 16-bit compressed forms for
existing standard load/store opcodes.
These opcodes are found in some RISC-V microcontrollers from WCH /
Nanjing Qinheng Microelectronics.
As discussed in the Discourse forums, this uses incompatible extension
and opcode names vs the vendor binary toolchain. The chosen names
instead follow the conventions for other vendor extensions listed on the
"riscv-non-isa" project.
It's a similar patch as a214c521f8763b36dd400b89017f74ad5ae4b6c7 for
vp.stride.load. Some targets prefer pattern (vmv.v.x (load)) instead of
vlse with zero stride.
It's IR version of #97798.
