Use a setne with 0 instead of a trunc. We know we zero extended the node
so we can get by with a non-zero check only. The truncate lowering
doesn't know that we zero extended so has to mask the lsb.
I don't think DAG combine sees the trunc before we lower it to RISCVISD
nodes so we don't get a chance to use computeKnownBits to remove the
AND.
For f16 with zvfhmin, we promote most ops and VP ops to f32. This does
the same for bf16 with zvfbfmin, so the two fp types should now be in
sync.
There are a few places in the custom lowering where we need to check for
a LMUL 8 f16/bf16 vector that can't be promoted and must be split, this
extracts that out into isPromotedOpNeedingSplit.
In a follow up NFC we can deduplicate the code that sets up the
promotions.
This adds zvfhmin test coverage for fceil, ffloor, fnearbyint, frint,
fround and froundeven and splits them at nxv32f16 to avoid crashing,
similarly to what we do for other nodes that we promote.
This also sets ftrunc to promote which was previously missing. We
already promote the VP version of it, vp_froundtozero.
Marking it as promoted affects some of the cost model tests since
they're no longer expanded.
These tests are good candidate for VL optimization. This is a pre-commit for
PR #108640, but can could probably also be improved by the peephole VL
optimizations.
This is mostly a copy of the AArch64PostLegalizerLoweringPass, except it
removes all of the AArch64 combines.
This pass allows us to lower instructions after the generic
post-legalization combiner has had a chance to run.
We will be adding combines to this pass in future patches.
I realized after spending way too much time looking at this, that
we can avoid objdump entirely here by having the assembly simply
not print the aliases. Once we do that, we can simply autogen
this test, and updates become trivial and understandable.
Two major changes:
- Remove use of sed preprocessing - this was being used to create two
versions of each test, and the result is much more readable if we
just duplicate the tests.
- Use a regex for matching the condition. An upcoming change causes
us to reverse the branch direction (which doesn't matter to the
purpose of these tests at all), so using the regex makes the test
more stable.
Per the comment, this test is intending to test the first constant which
can't be encoded via a c.addi. However, -32 *can* be encoded as in a
c.addi, and all that's preventing it from doing so is the register
allocators choice to use a difference destination register on the
add than it's source. (Which compressed doesn't support.)
The current LLC codegen for this test looks like:
addi a1, a0, -32
li a0, -99
bnez a1, .LBB0_2
li a0, 42
.LBB0_2:
ret
After https://github.com/llvm/llvm-project/pull/108889, we sink the LI, and
the register allocator picks the same source and dest register for the addi
resulting in the c.addi form being emitted. So, to avoid a confusing diff
let's fix the test to check what was originally intended.
This patch adds support for the missing STRICT_UINT_TO_FP and
STRICT_SINT_TO_FP for riscv and adds a test case for rv32 which was
previously crashing.
The code is in line with how other strict_* nodes are handled
(e.g., getting op(1) instead of op(0) when it's a strict node, as op(0)
in a strict node is the entry token).
We currently make sure to check that if folding an op to an f16 widening
op that we have zvfh. We need to do the same for bf16 vectors, but with
the further restriction that we can only combine vfmadd_vl to vfwmadd_vl
(to get vfwmaccbf16.v{v,f}).
The added test case currently crashes because we try to fold an add to a
bf16 widening add, which doesn't exist in zvfbfmin or zvfbfwma
This moves the checks into the extension support checks to keep it one
place.
We're also missing coverage for the vp_reduce_* nodes, but they
currently crash with zvfhmin. It looks like they're getting expanded
instead of promoted.
The InitUndef pass works around a register allocation issue, where undef
operands can be allocated to the same register as early-clobber result
operands. This may lead to ISA constraint violations, where certain
input and output registers are not allowed to overlap.
Originally this pass was implemented for RISCV, and then extended to ARM
in #77770. I've since removed the target-specific parts of the pass in
#106744 and #107885. This PR reduces the pass to use a single
requiresDisjointEarlyClobberAndUndef() target hook and enables it by
default. The hook is disabled for AMDGPU, because overlapping
early-clobber and undef operands are known to be safe for that target,
and we get significant codegen diffs otherwise.
The motivating case is the one in arm64-ldxr-stxr.ll, where we were
previously incorrectly allocating a stxp input and output to the same
register.
This PR is related to #99591. In this PR, instead of modifying how the
legalisation occurs depending on surrounding instructions, we refine
after legalisation.
This PR has two parts:
* `SDPatternMatch/MatchContext`: Modify a little bit the code to match
Operands (used by `m_Node(...)`) and Unary/Binary/Ternary Patterns to
make it compatible with `VPMatchContext`, instead of only `m_Opc`
supported. Some tests were added to ensure no regressions.
* `DAGCombiner`: Add a `foldSubCtlzNot` which detect and rewrite the
patterns using matching context.
Remaining Tasks:
- [ ] GlobalISel
- [ ] Currently the pattern matching will occur even before
legalisation. Should I restrict it to specific stages instead ?
- [ ] Style: Add a visitVP_SUB ?? Move `foldSubCtlzNot` in another
location for style consistency purpose ?
@topperc
---------
Co-authored-by: v01dxyz <v01dxyz@v01d.xyz>
This handles int->fp/fp->int nodes for zvfbfmin, reusing the same parts
that f16 uses with zvfhmin.
There's quite a bit of replication here that can probably be cleaned up
at some point.
According to https://github.com/openhwgroup/cv32e40p/issues/833 this
instruction was renamed last year to remove the 't'.
I used MnemonicAlias to support the old name. Unfortunately, this gives
a generic error if XCValu is not enabled. Since its an old name I hope
this isn't too big of an issue.
CC: @jeremybennett
Fortunately f16 and bf16 are always < EEW, so we can always lower via
widening or narrowing. This means we don't need to add patterns for
vrgather_vv_vl just yet.
Create an FP_EXTEND instead of handling the soft promote directly. This
FP_EXTEND will be visited and soft promoted itself.
This removes a zero extend from the generated code when the f32 type is
itself softened. Previously we softened it as an fp16_to_fp which sees
the operand as an integer type so we extend it. When we soften the
result as an fp_extend we see the source as f16 and don't extend. It
only becomes an integer inside call lowering not by type legalization.
If this extend is really necessary, then we have an issue when an
f16->f32 fp_extend exists in the source and f32 needs to be softened.
This simplifies part of #102503.
Multiple invocations of the pass could interfere with eachother,
preventing some undefs being initialised.
I found it very difficult to create a unit test for this due to it being
dependent on particular allocations of a previous function. However, the
bug can be observed here: https://godbolt.org/z/7xnMo41Gv with the
creation of the illegal instruction `vnsrl.wi v9, v8, 0`
I added these in #108245, but given the sheer number of tests that
will need to be added to cover bf16 promotion to f32 it seems better
to keep them in one place to avoid an explosion of files.
Most of the constants fli can generate are positive numbers. We can use
fli+fneg to generate their negative versions.
Previously, we considered such negative constants as "legal" and let
isel generate the fli+fneg. However, it is useful to expose the fneg to
DAG combines to fold with fadd to produce fsub or with fma to produce
fnmadd, fnmsub, or fmsub.
This patch moves the fneg creation to lowering so that the fneg will be
visible to the last DAG combine.
I might move the rest of Zfa handling from isel to lowering as a follow
up.
Fixes#107772.
The motivation for this is to start promoting bf16 ops to f32 so that we
can mark bf16 as a supported type in
RISCVTTIImpl::isElementTypeLegalForScalableVector and scalably-vectorize
it.
This starts with expanding the nodes that can't be promoted to f32 due
to canonicalizing NaNs, similarly to f16 in #106652.
selectFPImm previously handled cases where an FPImm could be
materialized in an integer register.
We can generalize this to cases where a value was in an integer register
and then copied to a scalar FP register to be used by a vector
instruction.
In the affected test, the call lowering code used up all of the FP
argument registers and started using GPRs. Now we use integer vector
instructions to consume those GPRs instead of moving them to scalar FP
first.
Previously they were legal by default, so the truncstore/extload test
cases would get combined and crash during selection.
These are set to expand for f16 so do the same for bf16.
This adds VL patterns for vfwmaccbf16.vv so that we can handle fixed
length vectors.
It does this by teaching combineOp_VLToVWOp_VL to emit
RISCVISD::VFWMADD_VL for bf16. The change in getOrCreateExtendedOp is
needed because getNarrowType is based off of the bitwidth so returns
f16. We need to explicitly check for bf16.
Note that the .vf patterns don't work yet, since the build_vector splat
gets lowered to a (vmv_v_x_vl (fmv_x_anyexth x)) instead of a vfmv.v.f,
which SplatFP doesn't pick up, see #106637.
GNU ld will error when encountering a pcrel_lo whose corresponding
pcrel_hi is in a different section. [1] introduced a check to help
prevent this issue by preventing outlining in a few circumstances.
However, we can also hit this same issue when outlining from functions
with prefixes ("hot"/"unlikely"/"unknown" from profile information, for
example) as the outlined function might not have the same prefix,
possibly resulting in a "paired" pcrel_lo and pcrel_hi ending up in
different sections.
To prevent this issue, take a similar approach as [1] and additionally
prevent outlining when we see a pcrel_lo and the function has a prefix.
[1]
96c85f80f0Fixes#107520
This is a three deep expression which is deeper than we've otherwise
gone for multiple expansions, but I think it's reasonable to do so. This
covers mul by 50, 100, and 200 which are reasonably common naturally
arising numbers.
Add tests for "sub(select(icmp(a,b),a,b),select(icmp(a,b),b,a)) -> abd(a,b)" patterns that still fail to match to abd nodes
This will hopefully be helped by #108218
The InitUndef pass currently uses the getLargestSuperClass() hook (which
is only used by that pass) to chose the register to initialize. This was done
to reduce the number of undef init pseudos needed, e.g. so that the vrnov0
regclass would use the same pseudo as v0. After #106744 we use a single
generic pseudo, so this is no longer necessary.
This allows us to reduce VLs feeding reduction instructions. In
particular, this means that <3 x Ty> reduce(load) like sequences no
longer require a VL toggle.
This was waiting on 3d72957; now that the latent correctness issue is
fixed, we can expand this transform.
Continuing with #107993 and #108007, this handles the last of the main
rematerializable vector instructions.
There's an extra spill in one of the test cases, but it's likely noise
from the spill weights and isn't an issue in practice.
Even though vmv.v.x has a non constant scalar operand, we can still
rematerialize it because we have split register allocation between
vectors and scalars.
InlineSpiller will check to make sure that the scalar operand is live at
the point where the rematerialization occurs, so this won't extend any
scalar live ranges. However this also means we may not be able to
rematerialize in some cases, as shown in @vmv.v.x_needs_extended.
It might be worthwhile teaching InlineSpiller to extend scalar live
ranges in a future patch. I experimented with this locally and it
reduced spills on 531.deepsjeng_r by a further 3%.
This commit introduces support for outlining functions across modules
using codegen data generated from previous codegen. The codegen data
currently manages the outlined hash tree, which records outlining
instances that occurred locally in the past.
The machine outliner now operates in one of three modes:
1. CGDataMode::None: This is the default outliner mode that uses the
suffix tree to identify (local) outlining candidates within a module.
This mode is also used by (full)LTO to maintain optimal behavior with
the combined module.
2. CGDataMode::Write (`-codegen-data-generate`): This mode is identical
to the default mode, but it also publishes the stable hash sequences of
instructions in the outlined functions into a local outlined hash tree.
It then encodes this into the `__llvm_outline` section, which will be
dead-stripped at link time.
3. CGDataMode::Read (`-codegen-data-use-path={.cgdata}`): This mode
reads a codegen data file (.cgdata) and initializes a global outlined
hash tree. This tree is used to generate global outlining candidates.
Note that the codegen data file has been post-processed with the raw
`__llvm_outline` sections from all native objects using the
`llvm-cgdata` tool (or a linker, `LLD`, or a new ThinLTO pipeline
later).
This depends on https://github.com/llvm/llvm-project/pull/105398. After
this PR, LLD (https://github.com/llvm/llvm-project/pull/90166) and Clang
(https://github.com/llvm/llvm-project/pull/90304) will follow for each
client side support.
This is a patch for
https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.
This fixes#107950 and adds a test case for it. The issue was due to
us incorrectly assuming that we stored a V0Defs entry for every single
instruction.
We actually only store them for instructions that use V0, so when we
updated the V0Def after moving we sometimes ended up copying nullptr
over from an instruction that doesn't use V0 and clearing the V0Def
entry inadvertently.
Because we don't have V0Defs on instructions that don't use V0, the
FIXME was never actually needed in the first place since the
bookkeeping wasn't out of sync to begin with.
That commit also mentioned that a future unmasked to masked pseudo
peephole might need unmasked pseudos to have V0Defs entries, but after
working on this locally it turns out we don't.
This reverts commit ce3648094d44e8c098396a353b215acecb363cda.
In #107827 we now set true's passthru to the false operand if it was
undef. We need to remember to also constrain the regclass in case true
is a masked pseudo which needs its passthrus to be in VR[M*]NoV0
