We can match this directly in isel with the i32 type being legal.
The generic DAG combine will unpromote part of the pattern and
prevent it from being matched in isel.
This is similar to vector.reverse, but only reverses the first EVL
elements.
I extracted this code from our downstream. Some of it may have come from
https://repo.hca.bsc.es/gitlab/rferrer/llvm-epi/ originally.
This will select i32 operations directly to W instructions without
custom nodes. Hopefully this can allow us to be less dependent on
hasAllNBitUsers to recover i32 operations in RISCVISelDAGToDAG.cpp.
This support is enabled with a command line option that is off by
default.
Generated code is still not optimal.
I've duplicated many test cases for this, but its not complete. Enabling this runs all existing lit tests without crashing.
Most of the FP constants supported by FLI are positive. For negative FP
constants X whose positive values is supported by FLI, we can use `(FNEG
(FLI -X))` to materialize X.
This effectively reverts f912d21e673b0
This was originally done for consistency with RISCVISD::ADD_LO so that
all nodes were emitted as RISCVISD nodes.
I've received feedback a couple times that its not worth it. So I'm
putting it back.
We currently shrink the type of vmv_s_x_vl to LMUL=1 when its passthru
is
undef to avoid constraining the register allocator since it ignores
LMUL.
This patch relaxes it for non-undef passthrus, which occurs when
lowering
insert_vector_elt.
The issue #55208 noticed that std::rint is vectorized by the
SLPVectorizer, but a very similar function, std::lrint, is not.
std::lrint corresponds to ISD::LRINT in the SelectionDAG, and
std::llrint is a familiar cousin corresponding to ISD::LLRINT. Now,
neither ISD::LRINT nor ISD::LLRINT have a corresponding vector variant,
and the LangRef makes this clear in the documentation of llvm.lrint.*
and llvm.llrint.*.
This patch extends the LangRef to include vector variants of
llvm.lrint.* and llvm.llrint.*, and lays the necessary ground-work of
scalarizing it for all targets. However, this patch would be devoid of
motivation unless we show the utility of these new vector variants.
Hence, the RISCV target has been chosen to implement a custom lowering
to the vfcvt.x.f.v instruction. The patch also includes a CostModel for
RISCV, and a trivial follow-up can potentially enable the SLPVectorizer
to vectorize std::lrint and std::llrint, fixing #55208.
The patch includes tests, obviously for the RISCV target, but also for
the X86, AArch64, and PowerPC targets to justify the addition of the
vector variants to the LangRef.
The RHS of a shift can have a different type than the LHS. If there are
undefs in the vector, we need the undef added to the RHS to match the
type of any shift amounts that are also added to the vector.
For now just don't add shifts if their RHS and LHS don't match.
Mark any registers as CustomReg and any stack slot as CustomMem.
This allows us to more directly emit the register or memory access for
the high part. Previously we needed a memory access if the low register
was X17 and we assumed the stack offset was 0. If the low part wasn't
X17, we assumed the high register was the next register after the low
register.
This is another part of supporting FP arguments with GISel.
Instead of doing a forward pass for positive strides and a reverse pass
for
negative strides, we can just do one pass by negating the offset if the
pointers do happen to be in reverse order.
We can extend getPtrDiff later in #68726 to handle more constant offset
sequences.
This removes the special case from unpackF64OnRV32DSoftABI. We can use
the default MemLoc handling.
This also allows us to remove a isRegLoc() check from LowerCall.
This part of preparation for supporting FP arguments with GISel.
hasOneUse can be more expensive for nodes with multiple outputs.
It's better to check the opcode first to skip nodes with multiple
outputs.
I have not seen an issue from this, just noticed while reviewing
code for a possible enhancement.
This patch adds stackmap support for RISC-V without targets (i.e. the nop patchable forms).
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D123496
This reverts commit b5ff71e261b637ab7088fb5c3314bf71d6e01da7. As described in
https://github.com/llvm/llvm-project/issues/68730, this appears to have exposed
an existing liveness issue. Revert to green until we can figure out how to
address the root cause.
Note: This was not a clean revert. I ended up doing it by hand.
This builds on the transform introduced in f0505c3, and generalizes to
all integer operations in 45a334d. This change adds support for floating
point sumation.
A couple of notes:
* I chose to leave fmaxnum and fminnum unhandled for the moment. They
have a slightly different set of legality rules.
* We could form strictly sequenced FADD reductions for FADDs without
fast math flags. As the ordered reductions are more expensive, I left
thinking about this as a future exercise.
* This can't yet match the full vector reduce + start value idiom. That
will be an upcoming set of changes.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.
Differential Revision: https://reviews.llvm.org/D158449
When lowering a constant build_vector sequence of doubles on RV32, if
the
addend wasn't zero, or the step/denominator wasn't one, it would crash
trying
to emit an illegal build_vector of <n x i64> with i32 operands, e.g:
t15: v2i64 = BUILD_VECTOR Constant:i32<1>, Constant:i32<1>
This patch fixes this by lowering the splats with
SelectionDAG::getConstant
with the vector type, which handles making it legal via
splat_vector_parts.
This was reverted in 824251c9b349d859a9169196cd9533c619a715ce exposed by this change in a previous patch. Fixed in 199cbec987ee68d70611db8e7961b43c3dbad83e. Original commit message follows.
This builds on the transform introduced in
https://github.com/llvm/llvm-project/pull/67821, and generalizes it for
all integer reduction types.
A couple of notes:
* This will only form smax/smin/umax/umin reductions when zbb is
enabled. Otherwise, we lower the min/max expressions early. I don't care
about this case, and don't plan to address this further.
* This excludes floating point. Floating point introduces concerns about
associativity. I may or may not do a follow up patch for that case.
* The explodevector test change is mildly undesirable from a clarity
perspective. If anyone sees a good way to rewrite that to stablize the
test, please suggest.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.
Differential Revision: https://reviews.llvm.org/D158449
This reverts commit 7a0b9daac9edde4293d2e9fdf30d8b35c04d16a6 and
63bbc250440141b1c51593904fba9bdaa6724280.
I'm seeing issues (e.g. on the GCC torture suite) where
combineBinOpOfExtractToReduceTree is called when the V extensions aren't
enabled and triggers a crash due to RISCVSubtarget::getElen asserting.
I'll aim to follow up with a minimal reproducer. Although it's pretty
obvious how to avoid this crash with some extra gating, there are a few
options as to where that should be inserted so I think it's best to
revert and agree the appropriate fix separately.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.
Differential Revision: https://reviews.llvm.org/D158449
vandn, vrev8 and vro{l,r} are now part of Zvkb, which Zvbb now implies.
This
patch updates the predicates to check for Zvkb instead of Zvbb in the
tablegen
patterns for the SD and VL nodes, as well as some of the lowering logic
in
RISCVISelLowering.
This builds on the transform introduced in
https://github.com/llvm/llvm-project/pull/67821, and generalizes it for
all integer reduction types.
A couple of notes:
* This will only form smax/smin/umax/umin reductions when zbb is
enabled. Otherwise, we lower the min/max expressions early. I don't care
about this case, and don't plan to address this further.
* This excludes floating point. Floating point introduces concerns about
associativity. I may or may not do a follow up patch for that case.
* The explodevector test change is mildly undesirable from a clarity
perspective. If anyone sees a good way to rewrite that to stablize the
test, please suggest.
…t folding.
We want to create a build_vector with narrower elements here. Normally
getNode on the ISD::TRUNCATE will constant fold this to a new
BUILD_VECTOR. If it doesn't constant fold, we end up with a cycle in the
DAG because we truncate the node we are replacing. Constant folding can
fail if one of the elements is an opaque constant.
The failing case I saw involved an opaque constant created by a memset
that was expanded. Not sure exactly what happened after that.
This patch creates a new BUILD_VECTOR with the new type directly.
This change adds two related DAG combines which together will take a
left-reduce scalar add tree of an explode_vector, and will incrementally
form a vector reduction of the vector prefix. If the entire vector is
reduced, the result will be a reduction over the entire vector.
Profitability wise, this relies on vredsum being cheaper than a pair of
extracts and scalar add. Given vredsum is linear in LMUL, and the
vslidedown required for the extract is *also* linear in LMUL, this is
clearly true at higher index values. At N=2, it's a bit questionable,
but I think the vredsum form is probably a better canonical form
anyways.
Note that this only matches left reduces. This happens to be the
motivating example I have (from spec2017 x264). This approach could be
generalized to handle right reduces without much effort, and could be
generalized to handle any reduce whose tree starts with adjacent
elements if desired. The approach fails for a reduce such as (A+C)+(B+D)
because we can't find a root to start the reduce with without scanning
the entire associative add expression. We could maybe explore using
masked reduces for the root node, but that seems of questionable
profitability. (As in, worth questioning - I haven't explored in any
detail.)
This is covering up a deficiency in SLP. If SLP encounters the scalar
form of reduce_or(A) + reduce_sum(a) where a is some common
vectorizeable tree, SLP will sometimes fail to revisit one of the
reductions after vectorizing the other. Fixing this in SLP is hard, and
there's no good reason not to handle the easy cases in the backend.
Another option here would be to do this in VectorCombine or generic DAG.
I chose not to as the profitability of the non-legal typed prefix cases
is very target dependent. I think this makes sense as a starting point,
even if we move it elsewhere later.
This is currently restructed only to add reduces, but obviously makes
sense for any associative reduction operator. Once this is approved, I
plan to extend it in this manner. I'm simply staging work in case we
decide to go in another direction.
Need to add NumSrcElts param to is..Mask functions in
ShuffleVectorInstruction class for better mask analysis. Mask.size() not
always matches the sizes of the permuted vector(s). Allows to better
estimate the cost in SLP and fix uses of the functions in other cases.
Differential Revision: https://reviews.llvm.org/D158449
