We can support these via libcalls in libgcc/compiler-rt or integer
operations for fneg/fabs/fcopysign. fp128 values will be passed in two
64-bit GPRs according to the psABI.
Supporting RV32 requires sret which is not supported by libcall handling
in LegalizerHelper.cpp yet. It doesn't call canLowerReturn.
A spread(2) shuffle is just a interleave with an undef lane. The
existing lowering was reusing the even lane for the undef value. This
was entirely legal, but non-optimal.
When we have legal instructions we want to promote to sXLen and let isel
pattern matching removing the and/sext_inreg.
When using a libcall we want to use a 'si' libcall for small types
instead of 'di'. To match the RV64 ABI, we need to sign extend `unsigned
int` arguments. We reuse the shouldSignExtendTypeInLibCall hook from
SelectionDAG.
The DAG has the same instructions: the signed and unsigned absolute
difference of it's input. For AArch64, they map to uabd and sabd for
Neon and SVE. The Neon and SVE instructions will require custom
patterns.
They are pseudo opcodes and are not imported by the IRTranslator. We
need combines to create them.
PowerPC, ARM, and AArch64 have native instructions.
/// i.e trunc(abs(sext(Op0) - sext(Op1))) becomes abds(Op0, Op1)
/// or trunc(abs(zext(Op0) - zext(Op1))) becomes abdu(Op0, Op1)
For GlobalISel, we are going to write the combines in MIR patterns.
see:
llvm/test/CodeGen/AArch64/abd-combine.ll
- [ ] combine into abd
- [ ] legalize and add td patterns
It doesn't make sense to add a new generic ISD to handle riscv tuple
type. Instead we use `SPLAT_VECTOR` for ISD and further lower to
`VMV_V_X`.
Note: If there's `visitSPLAT_VECTOR` in generic DAG combiner, it needs
to skip riscv vector tuple type.
Stack on https://github.com/llvm/llvm-project/pull/114329
We can extend the existing SHL+TRUNC lowering used for deinterleave2 for
deinterleave4, and deinterleave8 when the result types are small enough
to allow the shift to be legal. On RV64, this means i8 and i16 results
for deinterleave4 and i8 results for deinterleave8.
I'd already included a few cases for spread(N) in the decompress(N) variants,
but rename for clarity and add a couple more edge cases.
i.e. spread(N, 0) produces a, undef, b, undef, ...
This is analogous to febbf91 which added shuffle lowering using
vcompress; we can do the same thing in the deinterleave2 lowering path
which is used for scalable vectors.
Note that we can further improve this for high lmul usage by adjusting
how we materialize the mask (whose result is at most m1 with a known bit
pattern). I am deliberately staging the work so that the changes to
reduce register pressure are more easily evaluated on their own merit.
This adds separate tests for getOperandInfo to try and get better
coverage over all the possible operands.
It's done in MIR as it makes checking incompatible EEWs and EMULs
easier, which is difficult (but not impossible) to do in LLVM IR.
It tests each instruction "format", e.g. vop.vv/vwop.vv/vwop.wv etc.,
and for each format it tests the individual operands.
For each operand it tests when the EEW and EMUL match, when the EEW
doesn't match, and when the EMUL doesn't match.
Instead of directly lowering to vnsrl_vl and having custom pattern
matching for that case, we can just lower to a (legal) shift and
truncate, and let generic pattern matching produce the vnsrl.
The major motivation for this is that I'm going to reuse this logic to
handle e.g. deinterleave4 w/ i8 result.
The test changes aren't particularly interesting. They're minor code
improvements - I think because we do slightly better with the
insert_subvector patterns, but that's mostly irrelevant.
Bug fix FCLASS instruction in RISCV. The bug is due the fact that FCLASS
has an input float register and output GPR this caused reg bank select
regression.
In order to coalesce a vsetvli with a register AVL into a previous
vsetvli, we need to make sure that the AVL register is reachable at the
previous vsetvli.
Back in pre-RA vsetvli insertion we just checked to see if the two
virtual registers were the same virtual register, and then this was
hacked around in the move to post-RA. We can instead use live intervals
to check that the reaching definition is the same at both instructions.
On its own this doesn't have much of an impact, but helps a lot in
#118283 and enables coalescing in about 60 of the test cases from that
PR.
Need to force libcall legalization to treat the integer argument as
signed so that it can be promoted to XLen in call lowering for RV64.
Alternatively we could promote the operand before converting to libcall,
but going through call lowering is closer to what SelectionDAG does.
Here we add a scheduling mutation in pre-ra scheduling, which will
add an artificial dependency edge between mask producer and its
previous nearest instruction that uses V0 register.
This prevents the overlap of live intervals of mask registers and
as a consequence we can reduce some spills/moves.
From the test changes, we can see some improvements and also some
regressions (more vtype toggles).
Partially fixes#113489.
Change the intersect for the anticipated algorithm to ignore unknown
when anticipating. This effectively allows VXRM writes speculatively
because it could do a VXRM write even when there's branches where VXRM
is unneeded.
The importance of this change is because VXRM writes causes pipeline
flushes in some micro-architectures and so it makes sense to allow more
aggressive hoisting even if it causes some degradation for the slow
path.
An example is this code:
```
typedef unsigned char uint8_t;
__attribute__ ((noipa))
void foo (uint8_t *dst, int i_dst_stride,
uint8_t *src1, int i_src1_stride,
uint8_t *src2, int i_src2_stride,
int i_width, int i_height )
{
for( int y = 0; y < i_height; y++ )
{
for( int x = 0; x < i_width; x++ )
dst[x] = ( src1[x] + src2[x] + 1 ) >> 1;
dst += i_dst_stride;
src1 += i_src1_stride;
src2 += i_src2_stride;
}
}
```
With this patch, the code above generates a hoisting VXRM writes out of
the outer loop.
This change matches a subset of vcompress patterns during shuffle
lowering. The subset implemented requires a contiguous prefix of
demanded elements followed by undefs. This subset was chosen for two
reasons: 1) which elements to spurious demand is a non-obvious problem,
and 2) my first several attempts at implementing the general case were
buggy. I decided to go with the simple case to start with.
vcompress scales better with LMUL than a general vrgather, and at least
the SpaceMit X60, has higher throughput even at m1. It also has the
advantage of requiring smaller vector constants at one bit per element
as opposed to vrgather which is a minimum of 8 bits per element. The
downside to using vcompress is that we can't fold a vselect into it, as
there is no masked vcompress variant.
For reference, here are the relevant throughputs from camel-cdr's data
table on BP3 (X60):
vrgather.vv v8,v16,v24 4.0 16.0 64.0 256.0
vcompress.vm v8,v16,v24 3.0 10.0 36.0 136.
vmerge.vvm v8,v16,v24,v0 2.0 4.0 8.0 16.0
The largest concern with the extra vmerge is that we locally increase
register pressure. If we do have masking, we also have a passthru,
without the ability to fold that into the vcompress, we need to keep it
alive a bit longer. This can hurt at e.g. m8 where we have very few
architectural registers. As compared with the vrgather.vv sequence, this
is only one additional m1 VREG - since we no longer need the index
vector. It compares slightly worse against vrgatherie16.vv which can use
index vectors smaller than other operands. Note that we could
potentially fold the vmerge if only tail elements are being preserved; I
haven't investigated this.
It is unfortunately hard given our current lowering structure to know if
we're emitting a shuffle where masking will follow. Thankfully, it
doesn't seem to show up much in practice, so I think we can probably
ignore it.
This patch only handles single source compress idioms at the moment.
This is an effort to avoid interacting with other patches on review for
changing how we canonicalize length changing shuffles.
This covers multiple interactions reduced from larger workloads:
1) Rematerializing addi s0, x0, <imm> to avoid the need to spill a CSR, with
the common user being a branch. (i.e. branch on immediate idioms)
2) Rematerializing addi s0, a0, <imm> to avoid the need to spill a CSR, with
the common user being a vector load or store. (i.e. because we don't
have (reg+imm) addressing on vector.)
3) Independent of the previous, we could still shrink wrap these by locally
using a non-CSR, and deferring the move into csr into the non-shrink
wrapped path.
4) Weirdly, MachineCSE is producing a different result when an edge is
manually split. This edge split should be irrelevant to the CSE?
compress is intented to match vcompress from the ISA manual. Note that
deinterleave is a subset of this, and is already tested elsewhere.
decompress is the synthetic pattern defined in same - though we can often
do better than the mentioned iota/vrgather. Note that some of these
can also be expressed as interleave with at least one undef source,
and is already tested elsewhere.
repeat repeats each input element N times in the output. It can be
described as as a interleave operations, but we can sometimes do
better lowering wise.
This MR fixes failed test `CodeGen/RISCV/compress-opt-select.ll`.
It was failed due to previously merged commit `[TTI][RISCV]
Unconditionally break critical edges to sink ADDI (PR #108889)`.
So, regenerated `compress-opt-select` test.
Following up issue #89822, this patch adds opportunity to use tail call
in machine outliner pass.
Also it enables outline patterns with X5(T0) register.
SLEEF math vector library now supports RISC-V target.
Commit: https://github.com/shibatch/sleef/pull/477
This patch enables the use of auto-vectorization with
subsequent replacement by the corresponding SLEEF function.
This looks like a rather weird change, so let me explain why this isn't
as unreasonable as it looks. Let's start with the problem it's solving.
```
define signext i32 @overlap_live_ranges(ptr %arg, i32 signext %arg1) { bb:
%i = icmp eq i32 %arg1, 1
br i1 %i, label %bb2, label %bb5
bb2: ; preds = %bb
%i3 = getelementptr inbounds nuw i8, ptr %arg, i64 4
%i4 = load i32, ptr %i3, align 4
br label %bb5
bb5: ; preds = %bb2, %bb
%i6 = phi i32 [ %i4, %bb2 ], [ 13, %bb ]
ret i32 %i6
}
```
Right now, we codegen this as:
```
li a3, 1
li a2, 13
bne a1, a3, .LBB0_2
lw a2, 4(a0)
.LBB0_2:
mv a0, a2
ret
```
In this example, we have two values which must be assigned to a0 per the
ABI (%arg, and the return value). SelectionDAG ensures that all values
used in a successor phi are defined before exit the predecessor block.
This creates an ADDI to materialize the immediate in the entry block.
Currently, this ADDI is not sunk into the tail block because we'd have
to split a critical edges to do so. Note that if our immediate was
anything large enough to require two instructions we *would* split this
critical edge.
Looking at other targets, we notice that they don't seem to have this
problem. They perform the sinking, and tail duplication that we don't.
Why? Well, it turns out for AArch64 that this is entirely an accident of
the existance of the gpr32all register class. The immediate is
materialized into the gpr32 class, and then copied into the gpr32all
register class. The existance of that copy puts us right back into the
two instruction case noted above.
This change essentially just bypasses this emergent behavior aspect of
the aarch64 behavior, and implements the same "always sink immediates"
behavior for RISCV as well.
A special case in type legalization wasn't accounting for different
operand numbering between FLDEXP and STRICT_FLDEXP.
AArch64 already asked STRICT_FLDEXP to be promoted, but had no test for
it.
This come up in the context of pr 108889. We always end up sinking
the value in the phi if we dispatched via a switch, but not if we'd
dispatched via a branch. This is purely an artifact of current
lowering.
