Currently the CFI offset for RVV registers are not handled entirely,
this patch add those information for either stack unwinding or
debugger to work correctly on RVV callee-saved stack object.
Depends On D154576
Differential Revision: https://reviews.llvm.org/D156846
This is largely a revert of commit
e81796671890b59c110f8e41adc7ca26f8484d20.
As #88029 shows, there exists hardware that only supports unaligned
scalar.
I'm leaving how this gets exposed to the clang interface to a future
patch.
Bug fix: Handle RVV return type in calling convention correctly.
Return values are handled in a same way as function arguments.
One thing to mention is that if a type can be broken down into
homogeneous
vector types, e.g. {<vscale x 4 x i32>, {<vscale x 4 x i32>, <vscale x 4
x i32>}},
it is considered as a vector tuple type and need to be handled by tuple
type rule.
Copy the pointer info, flags, alignment, AAInfo, and ranges, but let
getLoad rebuild the MMO using the scalable type used for the the new
load/store. This makes sure the LLT minimum size matches the ContainerVT
minimum size. This is important since vscale_range may have been used to
determine that the fixed vector was the exact size of a scalable vector.
Fixes#88799
Machine scheduler will suppress register pressure when the scheduling
window is too small, but now it doesn't consider i64 register type,
and this MR extends it into i64 register type, so architecture like
RISCV64 that only supports i64 interger register will have the same
behavior like RISCV32.
Instead of bitcasting to a scalar integer, try to split the other vector
type that is legal. We already handle the case where both types need to
be split.
Fixes#88576
We recently optimized the code when the Odd vector was undef to fix a
poison bug.
There are additional optimizations we can do if the even vector is
undef. With Zvbb, we can use a single vwsll. Without Zvbb, we can use a
vzext.vf2 and a vsll.
This patch allows `combineBinOp_VLToVWBinOp_VL` to handle patterns like
`(splat_vector (sext op))` or `(splat_vector (zext op))`. Then we can
use `vwadd.vx` and `vwadd.w` for such a case.
### Source code
```
define <vscale x 8 x i64> @vwadd_vx_splat_sext(<vscale x 8 x i32> %va, i32 %b) {
%sb = sext i32 %b to i64
%head = insertelement <vscale x 8 x i64> poison, i64 %sb, i32 0
%splat = shufflevector <vscale x 8 x i64> %head, <vscale x 8 x i64> poison, <vscale x 8 x i32> zeroinitializer
%vc = sext <vscale x 8 x i32> %va to <vscale x 8 x i64>
%ve = add <vscale x 8 x i64> %vc, %splat
ret <vscale x 8 x i64> %ve
}
```
### Before this patch
[Compiler Explorer](https://godbolt.org/z/sq191PsT4)
```
vwadd_vx_splat_sext:
sext.w a0, a0
vsetvli a1, zero, e64, m8, ta, ma
vmv.v.x v16, a0
vsetvli zero, zero, e32, m4, ta, ma
vwadd.wv v16, v16, v8
vmv8r.v v8, v16
ret
```
### After this patch
```
vwadd_vx_splat_sext
vsetvli a1, zero, e32, m4, ta, ma
vwadd.vx v16, v8, a0
vmv8r.v v8, v16
ret
```
This follows on from #87616, but includes the tests with codegen
differences. These are presumably due to the fact that the splat is now
a constant expression. They don't seem to affect anything that we were
specifically testing for.
If the subtarget has +zvbb then we can attempt folding shl and shl_vl to
vwsll nodes.
There are few test cases where we still don't pick up the vwsll:
- For fixed vector vwsll.vi on RV32, see the FIXME for VMV_V_X_VL in
fillUpExtensionSupport for support implicit sign extension
- For scalable vector vwsll.vi we need to support ISD::SPLAT_VECTOR, see
#87249
In NodeExtensionHelper we keep track of the VL and mask of the operand
being extended and check that they are the same as the root node's.
However for the nodes that we support, none of them have a passthru
operand with the exception of RISCV::VMV_V_X_VL, but we check that it's
passthru is undef anyway.
So it's safe to just discard the extend node's VL and mask and just use
the root's instead. (This is the same type of reasoning we use to treat
any vmset_vl as an all ones mask)
This allows us to match some more cases where we mix VP/non-VP/VL nodes,
but these don't seem to appear in practice. The main benefit from this
would be to simplify the code.
A handy shorthand for specifying the shufflevector(insertelement(poison,
foo, 0), poison, zeroinitializer) splat pattern was introduced in
#74620.
Some of the RISC-V tests were converted over to use this new form in
dbb65dd330cc1696d7ca3dedc7aa9fa12c55a075, this patch handles the rest
which didn't have any codegen diffs.
This not only converts some constant expressions to the new form, but
also instruction sequences that weren't previously constant expressions
to constant expressions as well. In some cases this affects codegen, but
these have been omitted here and will be handled in a separate PR.
All of these test cases had iXLen in their name which got replaced
by sed. This prevented FileCheck from finding the function. The other
test cases in these files do not have that issue.
When the encoding of register tuples are aligned, we can use a copy
with larger LMUL to reduce copies.
Reviewers: preames, topperc, lukel97
Reviewed By: topperc, lukel97
Pull Request: https://github.com/llvm/llvm-project/pull/84455
On RV64, we legalize zexts of i1s to (vselect m, (splat_vector i64 1),
(splat_vector i64 0)), where the splat_vectors are implicitly
truncating.
When the vselect is used by a binop we want to pull the vselect out via
foldSelectWithIdentityConstant. But because vectors with an element size
< i64 will truncate, isNeutralConstant will return false.
This patch handles truncating splats by getting the APInt value and
truncating it. We almost don't need to do this since most of the neutral
elements are either one/zero/all ones, but it will make a difference for
smax and smin.
I wasn't able to figure out a way to write the tests in terms of select,
since we need the i1 zext legalization to create a truncating
splat_vector.
This supercedes #87236. Fixed vectors are unfortunately not handled by
this patch (since they get legalized to _VL nodes), but they don't seem
to appear in the wild.
Fixed vectors have their sext/zext operands legalized to _VL nodes, so
we need to handle them in the patterns.
This adds a riscv_ext_vl_oneuse pattern since we don't care about the
type of extension used for the shift amount, and extends
Low8BitsSplatPat to handle other _VL nodes. We don't actually need to
check the mask or VL there since none of the _VL nodes have passthru
operands.
The remaining test cases that are widening from i8->i64 need to be
handled by extending combineBinOp_VLToVWBinOp_VL.
This also fixes Low8BitsSplatPat incorrectly checking the vector size
instead of the element size to determine if the splat value might have
been truncated below 8 bits.
Hi all,
This patch is a follow-up of #79101. It migrates logic from
`visitVSELECT` to `visitVP_SELECT` to simplify `vp.select`. With this
patch we can do the following combinations:
```
vp.select undef, T, F --> T (if T is a constant), F otherwise
vp.select <condition>, undef, F --> F
vp.select <condition>, T, undef --> T
vp.select false, T, F --> F
vp.select <condition>, T, T --> T
```
I'm a total newbie to llvm and I'm sure there's room for improvements in
this patch. Please let me know if you have any advice. Thank you in
advance!
If the odd vector is undef or poison, the widening add and multiply trick
doesn't work unless we freeze the odd vector.
Unfortunately, freezing doesn't work when the operand is provably
undef/poison. MIR doesn't have a representation for freeze so it
just becomes a COPY from IMPLICIT_DEF which freely propagates undef
to each operand independently.
To work around this, check for undef explicitly and lower to a VZEXT_VL
of the even vector. This produces better code than we'd get from a
freeze anyway.
I've left a FIXME for adding a freeze. I'll do that as a separate patch
as it affects other tests and doesn't help with the new test.
The interleave lowering relies on a math trick that requires passing
the odd vector to two math instructions. In order to be correct
these instructions must see the same value.
If the odd vector is provably poison or undef, SelectionDAG will
create a vwadd and vwmaccu where the operand is a copy from IMPLICIT_DEF.
Later this will become just the undef flag on the operand. This
gives the register allocator freedom to pick a different register
for each instruction.
DAGCombiner (or InstCombine) will convert an add to an or if the bits
are disjoint, which can prevent what was originally an (add {s,z}ext,
{s,z}ext) from being selected as a vwadd.
This teaches combineBinOp_VLToVWBinOp_VL to recover it by treating it as
an add.
This narrows unsigned and signed div and rem nodes via
combineBinOpOfZExt.
Unlike other binary ops, there are no widening div or rem instructions.
So we will end up with an extra vzext.vf2.
However I'm assuming that div/rem are expensive enough that by reducing
their EMUL we will gain back the cost.
Alive2 proof: https://alive2.llvm.org/ce/z/Et_L6y
[RISCV] RISCV vector calling convention (1/2)
This is the vector calling convention based on
https://github.com/riscv-non-isa/riscv-elf-psabi-doc,
the idea is to split between "scalar" callee-saved registers
and "vector" callee-saved registers. "scalar" ones remain the
original strategy, however, "vector" ones are handled together
with RVV objects.
The stack layout would be:
|--------------------------| <-- FP
| callee-allocated save |
| area for register varargs|
|--------------------------|
| callee-saved registers | <-- scalar callee-saved
| (scalar) |
|--------------------------|
| RVV alignment padding |
|--------------------------|
| callee-saved registers | <-- vector callee-saved
| (vector) |
|--------------------------|
| RVV objects |
|--------------------------|
| padding before RVV |
|--------------------------|
| scalar local variables |
|--------------------------| <-- BP
| variable size objects |
|--------------------------| <-- SP
Note: This patch doesn't contain "tuple" type, e.g. vint32m1x2.
It will be handled in https://github.com/riscv-non-isa/riscv-elf-psabi-doc (2/2).
Differential Revision: https://reviews.llvm.org/D154576
Unlike add, sub and mul, we don't have widening instructions for div,
rem and logical ops, so we don't have any test coverage if we were to
extend combineBinOpOfZExts to handle them.
Adding tests coincidentally revealed that logical ops are already
narrowed as a generic DAG combine via
DAGCombiner::hoistLogicOpWithSameOpcodeHands. So we don't actually need
to run combineBinOpOfZExts on them.
Building on #86248, we can also narrow the width of a mul of zexts.
This is specifically legal because on RVV we always extend to the next
power of 2 width, and multiplying two N bit integers produces a maximum
value of 2\*N bits.
So as long as we keep an inner zext of 2\*N, we will have enough space
for the multiply and won't overflow.
Alive2 proof: https://alive2.llvm.org/ce/z/XteYyb
We have the ISD nodes for representing signed and unsigned absolute
difference. For RISCV, we have vector min/max in the base vector
extension, so we can expand to the sub(max,min) lowering.
We could almost use the default expansion, but since fixed length
min/max are custom (not legal), the default expansion doesn't cover the
fixed vector cases. The expansion here is just a copy of the generic
code specialized to allow the custom min/max nodes to be created so they
can in turn be legalized to the _vl variants.
Existing DAG combines handle the recognition of absolute difference
idioms and conversion into the respective ISD::ABDS and ISD::ABDU nodes.
This change does have the net effect of potentially pushing a free
floating zero/sign extend after the expansion, and we don't do a great
job of folding that into later expressions. However, since in general
narrowing can reduce required work (by reducing LMUL) this seems like
the right general tradeoff.
This builds on the previously added absolute difference cases, and adds
the reduction at the end. This is mostly interesting for examining
impact of extend placement when changing the abdu lowering.
This generalizes the combine added in #82455 to other binary ops,
beginning with adds in this patch.
Because the two zext operands are always +ve when treated as signed, and
we don't get any overflow since the add is carried out in at least N * 2
bits of the narrow type, the result of the add will always be +ve. So we
can use a zext for the outer extend, unlike sub which may produce a -ve
result from two +ve operands.
Although we could still use sext for add, I plan to add support for
other binary ops like mul in a later patch, but mul requires zext to be
correct (because the maximum value will take up the full N * 2 bits). So
I've opted to use zext here too for consistency.
Alive2 proof: https://alive2.llvm.org/ce/z/PRNsUM
Remove getSizeOrUnknown call when MachineMemOperand is created. For Scalable
TypeSize, the MemoryType created becomes a scalable_vector.
2 MMOs that have scalable memory access can then use the updated BasicAA that
understands scalable LocationSize.
Original Patch by Harvin Iriawan
Co-authored-by: David Green <david.green@arm.com>
This is needed to support non-intrinsic functions returning tuple types
which are represented as structs with scalable vector types in IR.
I suspect this may have been broken since
https://reviews.llvm.org/D158115
We can remove the restriction that the narrow type needs to be exactly
EEW / 2 for scalable ISD::{ADD,SUB,MUL} nodes. This allows us to perform
the combine even if we can't fully fold the extend into the widening op.
VP intrinsics already do this, since they are lowered to _VL nodes which
don't have this restriction.
The "exactly EEW / 2" narrow type restriction prevented us from emitting
V{S,Z}EXT_VL nodes with i1 element types which crash when we try to
select them, since no other legal type is double the size of i1, see the
test case added in this PR `i1_zext`. So to preserve this, this adds a
check for i1 narrow types instead.
This is a reimplementation of the combine added in #83035 but as a
lowering instead of a combine, so we don't regress the test case added
in e59f120e3a14ccdc55fcb7be996efaa768daabe0 by interfering with the
strided load combine
Previously the combine had to concatenate the split vectors with
insert_subvector instead of concat_vectors to prevent an infinite
combine loop. And the reasoning behind keeping it as a combine was
because if we emitted the insert_subvector during lowering then we
didn't fold away inserts of undef subvectors.
However it turns out we can avoid this if we just do this in lowering
and select a concat_vector directly, since we get the undef folding for
free with `DAG.getNode(ISD::CONCAT_VECTOR, ...)` via foldCONCAT_VECTORS.
Previously we used BuildPairF64 and SplitF64 only if Zfa was supported
since they will select register file moves that are only available with
Zfa.
We recently changed the handling of BuildPairF64/SplitF64 for Zdinx to
not go through memory so we should use that for bitcast.
That leaves the D without Zfa case that does need to go through memory.
Previously we let type legalization expand to loads and stores using a
new stack temporary created for each bitcast. After this patch we will
create the loads ands stores in the custom inserter and share the same
stack slot for all. This also allows DAGCombiner to optimize when
bitcast is mixed with BuildPairF64/SplitF64.
This adds a reduced test case for the regression seen in x264 with #83035.
If the intermediate concatenating shuffles are large enough then the
splitting combine will prevent the strided load combine which is
preferable.
