This patches adds a 32 bit register class for use with Zfinx instructions. This makes them more similar to F instructions and allows us to only spill 32 bits.
I've added CodeGenOnly instructions for load/store using GPRF32 as that gave better results than insert_subreg/extract_subreg.
Function arguments use this new GPRF32 register class for f32 arguments with Zfinx. Eliminating the need to use RISCVISD::FMV* nodes.
This is similar to #107446 which adds a 16 bit register class.
This patches adds a 16 bit register class for use with Zhinx
instructions. This makes them more similar to Zfh instructions and
allows us to only spill 16 bits.
I've added CodeGenOnly instructions for load/store using GPRF16 as that
gave better results than insert_subreg/extract_subreg. I'm using FSGNJ
for GPRF16 copy with Zhinx as that gave better results. Zhinxmin will
use ADDI+subreg operations.
Function arguments use this new GPRF16 register class for f16 arguments
with Zhinxmin. Eliminating the need to use RISCVISD::FMV* nodes.
I plan to extend this idea to Zfinx next.
Most of the constants fli can generate are positive numbers. We can use
fli+fneg to generate their negative versions.
Previously, we considered such negative constants as "legal" and let
isel generate the fli+fneg. However, it is useful to expose the fneg to
DAG combines to fold with fadd to produce fsub or with fma to produce
fnmadd, fnmsub, or fmsub.
This patch moves the fneg creation to lowering so that the fneg will be
visible to the last DAG combine.
I might move the rest of Zfa handling from isel to lowering as a follow
up.
Fixes#107772.
selectFPImm previously handled cases where an FPImm could be
materialized in an integer register.
We can generalize this to cases where a value was in an integer register
and then copied to a scalar FP register to be used by a vector
instruction.
In the affected test, the call lowering code used up all of the FP
argument registers and started using GPRs. Now we use integer vector
instructions to consume those GPRs instead of moving them to scalar FP
first.
We currently fold a vmerge.vvm into its true operand if the true operand
is a masked pseudo with the same mask.
We can move this over to RISCVVectorPeephole by instead splitting it up
into a smaller peephole which converts it to a vmv.v.v first. The
existing foldVMV_V_V peephole will then take care of folding it if
needed.
This is very similar to the existing all-ones mask peephole and we could
potentially do it inside of it. I opted to put it in a separate peephole
to make it easier to reason about, given that the duplication is small,
but I could be persuaded either way.
This patch handles target lowering and calling convention.
For target lowering, the vector tuple type represented as multiple
scalable vectors is now changed to a single `MVT`, each `MVT` has a
corresponding register class.
The load/store of vector tuples are handled as the same way but need
another vector insert/extract instructions to get sub-register group.
Inline assembly constraint for vector tuple type can directly be modeled
as "vr" which is identical to normal vector registers.
For calling convention, it no longer needs an alternative algorithm to
handle register allocation, this makes the code easier to maintain and
read.
Stacked on https://github.com/llvm/llvm-project/pull/97994
In #106110 we had to mark v[f]slide1down.vx as
ActiveElementsAffectResult since the elements in the body depend on VL.
However it doesn't depend on the mask, so this was overly conservative
and broke the vmerge peephole.
We can recover this by splitting up ActiveElementsAffectResult into VL
and Mask bits, so we can more accurately model v[f]slide1down.vx and
re-enable the peephole.
In order to support -unaligned-scalar-mem properly, we need to be more
careful with immediates of global variables. We need to guarantee that
adding 4 in RISCVExpandingPseudos won't overflow simm12. Since we don't
know what the simm12 is until link time, the only way to guarantee this
is to make sure the base address is at least 8 byte aligned.
There were also several corner cases bugs in immediate folding where we
would fold an immediate in the range [2044,2047] where adding 4 would
overflow. These are not related to unaligned-scalar-mem.
Fixed an incorrect cast.
Original message:
If c1 is a shifted mask with c3 leading zeros and c4 trailing zeros. If
c2 is greater than c3, we can use (srli (srai y, c2 - c3), c3 + c4)
followed by a SHXADD with c4 as the X amount.
Without Zba we can use (slli (srli (srai y, c2 - c3), c3 + c4), c4).
Alive2: https://alive2.llvm.org/ce/z/AwhheR
This caused an assert to fire:
llvm/include/llvm/Support/Casting.h:566:
decltype(auto) llvm::cast(const From &) [To = llvm::ConstantSDNode, From = llvm::SDValue]:
Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
see comment on the PR.
> If c1 is a shifted mask with c3 leading zeros and c4 trailing zeros. If
> c2 is greater than c3, we can use (srli (srai y, c2 - c3), c3 + c4)
> followed by a SHXADD with c4 as the X amount.
>
> Without Zba we can use (slli (srli (srai y, c2 - c3), c3 + c4), c4).
> Alive2: https://alive2.llvm.org/ce/z/AwhheR
This reverts commit 514481736cf943464125ef34570a7df0a19290de.
If c1 is a shifted mask with c3 leading zeros and c4 trailing zeros. If
c2 is greater than c3, we can use (srli (srai y, c2 - c3), c3 + c4)
followed by a SHXADD with c4 as the X amount.
Without Zba we can use (slli (srli (srai y, c2 - c3), c3 + c4), c4).
Alive2: https://alive2.llvm.org/ce/z/AwhheR
This is split off from #71764, and moves only the vmv.v.v part of
performCombineVMergeAndVOps to work on MachineInstrs.
In retrospect trying to handle PseudoVMV_V_V and PseudoVMERGE_VVM in the
same function makes the code quite hard to read, so this just does it in
a separate peephole.
This turns out to be simpler since for PseudoVMV_V_V we don't need to
convert the Src instruction to a masked variant, and we don't need to
create a fake all ones mask.
If x is a shl by 32 and c1 is an simm12, we would prefer to use a
SRAIW+ANDI. This prevents selecting the slli to a separate slli
instruction.
Fixes regression from #101868
Give the (and (srl/shl X, C2), C1) handling its owns private `C1` variable
it can modify using known zeros. This may be out of sync with
N1C->getZExtValue().
Add a separate const C1 for (and (sra X, C2), C1) and (and X, C).
This copy will always be in sync with N1C->getZExtValue().
Remove the IsC1Mask and IsC1ANDI variables and compute them at their
usage.
Use N1C->getSExtValue() when calling isInt. This shouldn't be a
functional change since we already checked that it was a mask. In
order for it to be a mask and a negative number, it would need to be -1
which should have been removed by DAG combine.
As noted in
https://github.com/llvm/llvm-project/pull/100367/files#r1695442138,
RISCVMaskedPseudoInfo currently stores two things, whether or not a
masked pseudo has an unmasked variant, and whether or not it's
element-wise.
These are separate things, so this patch splits the latter out into the
underlying instruction's TSFlags to help make the semantics of #100367
more clear.
To the best of my knowledge the only non-element-wise instructions in V
are:
- vredsum.vs and other reductions
- vcompress.vm
- vms*f.m
- vcpop.m and vfirst.m
- viota.m
In vector crypto the instructions that operate on element groups are
conservatively marked (this might be fine to relax later given since
non-EGS multiple vls are reserved), as well as the SiFive extensions and
XTHeadVdot.
Make "break" consistently the "if" body and the "return false" the
last thing in each case.
This makes it easier to add different conditions for different operands
of some instructions and makes everything more consistent.
As noted in
https://github.com/llvm/llvm-project/pull/100367/files#r1695448771, we
currently fold in vmerge.vvms and vmv.v.vs into their ops even if the
EEW is different which leads to an incorrect transform.
This checks the op's EEW via its simple value type for now since there
doesn't seem to be any existing information about the EEW size of
instructions. We'll probably need to encode this at some point if we
want to be able to access it at the MachineInstr level in #100367
The tablegen patterns all have isRV32. I did not check if any of them
could naively support RV64.
Fixes#101067 and probably other bugs like it we haven't found yet.
When folding, we currently check if the pseudo's result is not lanewise
(e.g. vredsum.vs or viota.m) and bail if we're changing the mask.
However we also need to check for the AVL too.
This patch bails if the AVL changed for these pseudos, and also renames
the pseudo table property to be more explicit.
We currently only fold a vmerge into a masked true operand if the vmerge
has an all-ones mask, since we end up keeping the mask from the true
operand.
But if the masks are the same then we can still fold, because vmerge and
true have the same passthru. If an element was masked off in the
original vmerge, it will also be masked off in the resulting true, and
will have the same passthru value.
The motivation for this is to lower masked VP loads and stores with
passthrus to masked RVV instructions. Normally you can express a masked
RVV instruction with a mask undisturbed passthru via a combination of a
VP op with an all-ones mask and a vp.merge. But for loads and stores you
need the same mask on the VP op as well as the vp.merge.
We currently don't fold a vmerge if it has an implicit-def passthru and
its true operand also has a passthru (i.e. tied dest).
This restriction was added in https://reviews.llvm.org/D151596, back
whenever we had separate TU/TA pseudos. It looks like it was added
because the policy might not have been handled correctly.
However the policy should be set correctly if we relax this restriction
today, since we compute the policy differently now that we have removed
the TU/TA distinction in our pseudos.
We use a TUMU policy, and relax it to TAMU iff the vmerge's passthru is
implicit-def.
The reasoning behind this being that the tail elements always come from
the vmerge's passthru[^1], so if vmerge's passthru is implicit-def then
the tail is also implicit-def. So a tail agnostic policy is OK.
[^1]: unless the VL was shrunk, but in this case which case we
conservatively use TUMU.
- Fix build with `EXPENSIVE_CHECKS`
- Remove unused `PassName::ID` to resolve warning
- Mark `~SelectionDAGISel` virtual so AArch64 backend can work properly
This reverts commit de37c06f01772e02465ccc9f538894c76d89a7a1 to
de37c06f01772e02465ccc9f538894c76d89a7a1
It still breaks EXPENSIVE_CHECKS build. Sorry.
Port selection dag isel to new pass manager.
Only `AMDGPU` and `X86` support new pass version. `-verify-machineinstrs` in new pass manager belongs to verify instrumentation, it is enabled by default.
This follows the same pattern as 20e62658735a1b03ecadc. Although we
can't reduce the number of instructions used, if we are able to use a
sign-extended 6-bit immediate then the 16-bit c.li instruction can be
selected (thus saving code size). Although this _could_ be gated so it
only happens if C is enabled, I've opted not to because at worst it's
neutral and it doesn't seem helpful to add unnecessary divergence
between the RVC and non-RVC paths.
If a segmented load has an undefined passthru then it will be selected
as a reg_sequence with implicit_def operands, which currently slips
through the implicit_def -> noreg peephole.
This patch fixes this so we're able to infer if the passthru is
undefined without the need for looking through vreg definitions with
MachineRegisterInfo, which will help with moving RISCVInsertVSETVLI to
LiveIntervals in #70549
This is the insert_subvector equivalent to #79949, where we can avoid
sliding up by the full LMUL amount if we know the exact subregister the
subvector will be inserted into.
This mirrors the lowerEXTRACT_SUBVECTOR changes in that we handle this
in two parts:
- We handle fixed length subvector types by converting the subvector to
a scalable vector. But unlike EXTRACT_SUBVECTOR, we may also need to
convert the vector being inserted into too.
- Whenever we don't need a vslideup because either the subvector fits
exactly into a vector register group *or* the vector is undef, we need
to emit an insert_subreg ourselves because RISCVISelDAGToDAG::Select
doesn't correctly handle fixed length subvectors yet: see d7a28f7ad
A subvector exactly fits into a vector register group if its size is a
known multiple of the size of a vector register, and this adds a new
overload for TypeSize::isKnownMultipleOf for scalable to scalable
comparisons to help reason about this.
I've left RISCVISelDAGToDAG::Select untouched for now (minus relaxing an
invariant), so that the insert_subvector and extract_subvector code
paths are the same.
We should teach it to properly handle fixed length subvectors in a
follow-up patch, so that the "exact subregsiter" logic is handled in one
place instead of being spread across both RISCVISelDAGToDAG.cpp and
RISCVISelLowering.cpp.
Marking them as `hasSideEffects=1` stops some optimizations.
According to `Target.td`:
> // Does the instruction have side effects that are not captured by any
> // operands of the instruction or other flags?
> bit hasSideEffects = ?;
It seems we don't need to set `hasSideEffects` for vleNff since we have
modelled `vl` as an output operand.
As for saturating instructions, I think that explicit Def/Use list
is kind of side effects captured by any operands of the instruction,
so we don't need to set `hasSideEffects` either. And I have just
investigated AArch64's implementation, they don't set this flag and
don't add `Def` list.
These changes make optimizations like `performCombineVMergeAndVOps`
and MachineCSE possible for these instructions.
As a consequence, `copyprop.mir` can't test what we want to test in
https://reviews.llvm.org/D155140, so we replace `vssra.vi` with a
VCIX instruction (it has side effects).
Reviewers: jacquesguan, topperc, preames, asb, lukel97
Reviewed By: topperc, lukel97
Pull Request: https://github.com/llvm/llvm-project/pull/90049
Fixed vectors have their sext/zext operands legalized to _VL nodes, so
we need to handle them in the patterns.
This adds a riscv_ext_vl_oneuse pattern since we don't care about the
type of extension used for the shift amount, and extends
Low8BitsSplatPat to handle other _VL nodes. We don't actually need to
check the mask or VL there since none of the _VL nodes have passthru
operands.
The remaining test cases that are widening from i8->i64 need to be
handled by extending combineBinOp_VLToVWBinOp_VL.
This also fixes Low8BitsSplatPat incorrectly checking the vector size
instead of the element size to determine if the splat value might have
been truncated below 8 bits.
Previously we used memory like we do to move between GPRs and FPR64 with
the D extension on RV32.
We can instead use REG_SEQUENCE/EXTRACT_SUBREG to inform register
allocation how to do the copy without memory.
If the vector type is a fixed vector type, we convert it to a container
scalable vector type to compute its reg class. But we need to keep the
old
fixed type so we create a result node with the same type.
This code path is currently dead so I haven't been able to create a test
case
for it. But I have an upcoming patch for insert_subvector lowering that
will
exercise this.
We can currently select insert_subvector and extract_subvector nodes in
RISCVISelDAGToDAG (this is after custom legalizing in RISCVISelLowering)
with fixed subvector types.
However decomposeSubvectorInsertExtractToSubRegs is based off of
scalable subvectors where the indices are scaled by vscale, so any index
other than 0 will be wrong.
For insert_subvector the vector being inserted into needs to be undef as
well, because it assumes we can replace a whole subregister which isn't
always the case for fixed subvectors (e.g. insert <2 x i32> into <4 x
i32> at index 0 with vlen=128).
We currently maintain these invariants in RISCVISelLowering, so this
adds asserts in RISCVISelDAGToDAG so we don't break them.
We've now got enough of these in tree that we can see which patterns
appear to be idiomatic. As such, extract a helper for checking
if we know the exact VLEN.
