We were previously checking a combination of the vector policy op and
the opcode to determine if we needed to skip copying the passthru from
a masked pseudo to an unmasked pseudo.
However we can just do this by checking
RISCVII::isFirstDefTiedToFirstUse, which is a proxy for whether or not
a pseudo has a passthru operand.
This should hopefully remove the need for the changes in #123106
SDNode::use_iterator now returns an SDUse& when dereferenced.
SDNode::user_iterator returns SDNode*. SDNode::use_begin/use_end/uses
work on use_iterator. SDNode::user_begin/user_end/users work on
user_iterator.
We can now write range based for loops using SDUse& and SDNode::uses().
I've converted many of these in this patch. I didn't update loops that
have additional variables updated in their for statement.
Some loops use SDNode::use_iterator::getOperandNo() which also prevents
using range based for loops. I plan to move this into SDUse in a follow
up patch.
This function is most often used in range based loops or algorithms
where the iterator is implicitly dereferenced. The dereference returns
an SDNode * of the user rather than SDUse * so users() is a better name.
I've long beeen annoyed that we can't write a range based loop over
SDUse when we need getOperandNo. I plan to rename use_iterator to
user_iterator and add a use_iterator that returns SDUse& on dereference.
This will make it more like IR.
These instructions should always be created with Log2SEW=0 and an LMUL
based on SEW=8. This is used by the vsetvli pass to know these
instructions only care about the ratio and not the specific value.
Looks like I fixed riscv_vmsge(u)_mask intrinsics years ago, but forgot
the unmasked intrinsics.
I'm working on an enhancement to our MachineVerifier checks that will
require VMNAND and VMSET to have Log2SEW=0.
Replace LMUL suffixes with _B1, _B2, etc. This matches what we do
for other mask only instructions like VCPOP_M, VFIRST_M, VMSBF_M,
VLM, VSM, etc.
Now all pseudoinstructions that use Log2SEW=0 will be consistently
named.
These are suffixed with B1, B2, B4, B8, B16, B32, or B64 which I think
these were supposed to match the naming of the vbool types from C where
the number should be SEW/LMUL. So the smallest mask is 64 and the
largest is 1. This provides a compact syntax for describing the 7
possible ratios between LMUL and SEW.
We had the instruction names in the opposite order.
Masking out most significant bits can be done with shl followed by srl
with same shift amount. If this is followed by a shl, we could instead
srl by a smaller amount of bits.
This transform is already implemented in tablegen for masking out
32 most significant bits.
Emits better code for e.g.
float *index(float *p, int i)
{
return p + (i & (1 << 30) - 1);
}
As suggested by Craig, this tries to merge the two sets of register
classes created in #112983, GPRPair* and GPRF64Pair*.
- I added some explicit annotations to `RISCVInstrInfoD.td` which fixed
the type inference issues I was seeing from tablegen for select
patterns.
- I've had to make the behaviour of `splitValueIntoRegisterParts` and
`joinRegisterPartsIntoValue` cover more cases, because you cannot
bitcast to/from untyped (the bitcast would otherwise have been inserted
automatically by TargetLowering code).
- I apparently didn't need to change `getNumRegisters` again, which
continues to tell me there's a bug in the code for tied inputs. I added
some more test coverage of this case but it didn't seem to help find the
asserts I was finding before - I think the difference is between the
default behaviour for integers which doesn't apply to floats.
- There's still a difference between BuildGPRPair and BuildPairF64 (and
the same for SplitGPRPair and SplitF64). I'm not happy with this, I
think it's quite confusing, as they're very similar, just differing in
whether they give a `untyped` or a `f64`. I haven't really worked out
how the DAGCombiner copes if one meets the other, I know we have some of
this for the f64 variants already, but they're a lot more complex than
the GPRPair variants anyway.
This patch adds support for getting even-odd general purpose register
pairs into and out of inline assembly using the `R` constraint as
proposed in riscv-non-isa/riscv-c-api-doc#92
There are a few different pieces to this patch, each of which need their
own explanation.
- Renames the Register Class used for f64 values on rv32i_zdinx from
`GPRPair*` to `GPRF64Pair*`. These register classes are kept broadly
unmodified, as their primary value type is used for type inference
over selection patterns. This rename affects quite a lot of files.
- Adds new `GPRPair*` register classes which will be used for `R`
constraints and for instructions that need an even-odd GPR pair. This
new type is used for `amocas.d.*`(rv32) and `amocas.q.*`(rv64) in
Zacas, instead of the `GPRF64Pair` class being used before.
- Marks the new `GPRPair` class legal as for holding a `MVT::Untyped`.
Two new RISCVISD node types are added for creating and destructing a
pair - `BuildGPRPair` and `SplitGPRPair`, and are introduced when
bitcasting to/from the pair type and `untyped`.
- Adds functionality to `splitValueIntoRegisterParts` and
`joinRegisterPartsIntoValue` to handle changing `i<2*xlen>` MVTs into
`untyped` pairs.
- Adds an override for `getNumRegisters` to ensure that `i<2*xlen>`
values, when going to/from inline assembly, only allocate one (pair)
register (they would otherwise allocate two). This is due to a bug in
SelectionDAGBuilder.cpp which other backends also work around.
- Ensures that Clang understands that `R` is a valid inline assembly
constraint.
- This also allows `R` to be used for `f64` types on `rv32_zdinx`
architectures, where doubles are stored in a GPR pair.
This patches adds a 32 bit register class for use with Zfinx instructions. This makes them more similar to F instructions and allows us to only spill 32 bits.
I've added CodeGenOnly instructions for load/store using GPRF32 as that gave better results than insert_subreg/extract_subreg.
Function arguments use this new GPRF32 register class for f32 arguments with Zfinx. Eliminating the need to use RISCVISD::FMV* nodes.
This is similar to #107446 which adds a 16 bit register class.
This patches adds a 16 bit register class for use with Zhinx
instructions. This makes them more similar to Zfh instructions and
allows us to only spill 16 bits.
I've added CodeGenOnly instructions for load/store using GPRF16 as that
gave better results than insert_subreg/extract_subreg. I'm using FSGNJ
for GPRF16 copy with Zhinx as that gave better results. Zhinxmin will
use ADDI+subreg operations.
Function arguments use this new GPRF16 register class for f16 arguments
with Zhinxmin. Eliminating the need to use RISCVISD::FMV* nodes.
I plan to extend this idea to Zfinx next.
Most of the constants fli can generate are positive numbers. We can use
fli+fneg to generate their negative versions.
Previously, we considered such negative constants as "legal" and let
isel generate the fli+fneg. However, it is useful to expose the fneg to
DAG combines to fold with fadd to produce fsub or with fma to produce
fnmadd, fnmsub, or fmsub.
This patch moves the fneg creation to lowering so that the fneg will be
visible to the last DAG combine.
I might move the rest of Zfa handling from isel to lowering as a follow
up.
Fixes#107772.
selectFPImm previously handled cases where an FPImm could be
materialized in an integer register.
We can generalize this to cases where a value was in an integer register
and then copied to a scalar FP register to be used by a vector
instruction.
In the affected test, the call lowering code used up all of the FP
argument registers and started using GPRs. Now we use integer vector
instructions to consume those GPRs instead of moving them to scalar FP
first.
We currently fold a vmerge.vvm into its true operand if the true operand
is a masked pseudo with the same mask.
We can move this over to RISCVVectorPeephole by instead splitting it up
into a smaller peephole which converts it to a vmv.v.v first. The
existing foldVMV_V_V peephole will then take care of folding it if
needed.
This is very similar to the existing all-ones mask peephole and we could
potentially do it inside of it. I opted to put it in a separate peephole
to make it easier to reason about, given that the duplication is small,
but I could be persuaded either way.
This patch handles target lowering and calling convention.
For target lowering, the vector tuple type represented as multiple
scalable vectors is now changed to a single `MVT`, each `MVT` has a
corresponding register class.
The load/store of vector tuples are handled as the same way but need
another vector insert/extract instructions to get sub-register group.
Inline assembly constraint for vector tuple type can directly be modeled
as "vr" which is identical to normal vector registers.
For calling convention, it no longer needs an alternative algorithm to
handle register allocation, this makes the code easier to maintain and
read.
Stacked on https://github.com/llvm/llvm-project/pull/97994
In #106110 we had to mark v[f]slide1down.vx as
ActiveElementsAffectResult since the elements in the body depend on VL.
However it doesn't depend on the mask, so this was overly conservative
and broke the vmerge peephole.
We can recover this by splitting up ActiveElementsAffectResult into VL
and Mask bits, so we can more accurately model v[f]slide1down.vx and
re-enable the peephole.
In order to support -unaligned-scalar-mem properly, we need to be more
careful with immediates of global variables. We need to guarantee that
adding 4 in RISCVExpandingPseudos won't overflow simm12. Since we don't
know what the simm12 is until link time, the only way to guarantee this
is to make sure the base address is at least 8 byte aligned.
There were also several corner cases bugs in immediate folding where we
would fold an immediate in the range [2044,2047] where adding 4 would
overflow. These are not related to unaligned-scalar-mem.
Fixed an incorrect cast.
Original message:
If c1 is a shifted mask with c3 leading zeros and c4 trailing zeros. If
c2 is greater than c3, we can use (srli (srai y, c2 - c3), c3 + c4)
followed by a SHXADD with c4 as the X amount.
Without Zba we can use (slli (srli (srai y, c2 - c3), c3 + c4), c4).
Alive2: https://alive2.llvm.org/ce/z/AwhheR
This caused an assert to fire:
llvm/include/llvm/Support/Casting.h:566:
decltype(auto) llvm::cast(const From &) [To = llvm::ConstantSDNode, From = llvm::SDValue]:
Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
see comment on the PR.
> If c1 is a shifted mask with c3 leading zeros and c4 trailing zeros. If
> c2 is greater than c3, we can use (srli (srai y, c2 - c3), c3 + c4)
> followed by a SHXADD with c4 as the X amount.
>
> Without Zba we can use (slli (srli (srai y, c2 - c3), c3 + c4), c4).
> Alive2: https://alive2.llvm.org/ce/z/AwhheR
This reverts commit 514481736cf943464125ef34570a7df0a19290de.
If c1 is a shifted mask with c3 leading zeros and c4 trailing zeros. If
c2 is greater than c3, we can use (srli (srai y, c2 - c3), c3 + c4)
followed by a SHXADD with c4 as the X amount.
Without Zba we can use (slli (srli (srai y, c2 - c3), c3 + c4), c4).
Alive2: https://alive2.llvm.org/ce/z/AwhheR
This is split off from #71764, and moves only the vmv.v.v part of
performCombineVMergeAndVOps to work on MachineInstrs.
In retrospect trying to handle PseudoVMV_V_V and PseudoVMERGE_VVM in the
same function makes the code quite hard to read, so this just does it in
a separate peephole.
This turns out to be simpler since for PseudoVMV_V_V we don't need to
convert the Src instruction to a masked variant, and we don't need to
create a fake all ones mask.
If x is a shl by 32 and c1 is an simm12, we would prefer to use a
SRAIW+ANDI. This prevents selecting the slli to a separate slli
instruction.
Fixes regression from #101868
Give the (and (srl/shl X, C2), C1) handling its owns private `C1` variable
it can modify using known zeros. This may be out of sync with
N1C->getZExtValue().
Add a separate const C1 for (and (sra X, C2), C1) and (and X, C).
This copy will always be in sync with N1C->getZExtValue().
Remove the IsC1Mask and IsC1ANDI variables and compute them at their
usage.
Use N1C->getSExtValue() when calling isInt. This shouldn't be a
functional change since we already checked that it was a mask. In
order for it to be a mask and a negative number, it would need to be -1
which should have been removed by DAG combine.
As noted in
https://github.com/llvm/llvm-project/pull/100367/files#r1695442138,
RISCVMaskedPseudoInfo currently stores two things, whether or not a
masked pseudo has an unmasked variant, and whether or not it's
element-wise.
These are separate things, so this patch splits the latter out into the
underlying instruction's TSFlags to help make the semantics of #100367
more clear.
To the best of my knowledge the only non-element-wise instructions in V
are:
- vredsum.vs and other reductions
- vcompress.vm
- vms*f.m
- vcpop.m and vfirst.m
- viota.m
In vector crypto the instructions that operate on element groups are
conservatively marked (this might be fine to relax later given since
non-EGS multiple vls are reserved), as well as the SiFive extensions and
XTHeadVdot.
Make "break" consistently the "if" body and the "return false" the
last thing in each case.
This makes it easier to add different conditions for different operands
of some instructions and makes everything more consistent.
As noted in
https://github.com/llvm/llvm-project/pull/100367/files#r1695448771, we
currently fold in vmerge.vvms and vmv.v.vs into their ops even if the
EEW is different which leads to an incorrect transform.
This checks the op's EEW via its simple value type for now since there
doesn't seem to be any existing information about the EEW size of
instructions. We'll probably need to encode this at some point if we
want to be able to access it at the MachineInstr level in #100367
The tablegen patterns all have isRV32. I did not check if any of them
could naively support RV64.
Fixes#101067 and probably other bugs like it we haven't found yet.
When folding, we currently check if the pseudo's result is not lanewise
(e.g. vredsum.vs or viota.m) and bail if we're changing the mask.
However we also need to check for the AVL too.
This patch bails if the AVL changed for these pseudos, and also renames
the pseudo table property to be more explicit.
We currently only fold a vmerge into a masked true operand if the vmerge
has an all-ones mask, since we end up keeping the mask from the true
operand.
But if the masks are the same then we can still fold, because vmerge and
true have the same passthru. If an element was masked off in the
original vmerge, it will also be masked off in the resulting true, and
will have the same passthru value.
The motivation for this is to lower masked VP loads and stores with
passthrus to masked RVV instructions. Normally you can express a masked
RVV instruction with a mask undisturbed passthru via a combination of a
VP op with an all-ones mask and a vp.merge. But for loads and stores you
need the same mask on the VP op as well as the vp.merge.
