This patch adds the remaining support for fixed-point arithmetic
instructions (we previously had support for averaging adds and
subtracts).
For saturating adds/subs/multiplies/clips, we can't change `vl` if
`vxsat` is used, since changing `vl` may change its value. So this patch
checks to see if it's dead before considering it a candidate.
After #124066 we started allowing users that are passthrus. However for
widening/narrowing instructions we were returning the wrong operand info
for passthru operands since it originally assumed the operand would
never be a passthru. This fixes it by handling it in IsMODef.
We already had getOperandInfo support, so this marks the instructions as
supported in isCandidate. It also adds support for vfwmaccbf16.v{v,f}
from zvfbfwma
I was running into failed assertions of `isCandidate(UserMI)` in
`getMinimumVLForUser`, but only occurring with
`-enable-machine-outliner=never`. I believe this is a red herring, and
it just so happens the memory allocation pattern on my machine exposed
the bug with that flag.
DemandedVLs is never cleared, which means it accumulates more
MachineInstr pointer keys over time, and it's possible that when e.g.
running on function 'b', a MachineInstr pointer points to the same
memory location used for a candidate in 'a'. This causes the assertion
to fail.
Comment left on #124530 with more information.
We don't want OperandInfo to be visible outside of this translation
unit.
getEMULEqualsEEWDivSEWTimesLMUL is local to this file and declared
static. There's no reason to put it in a namespace.
The motivation for this to allow reducing the vl when a user is a
ternary pseudo, where the third operand is tied and also acts as a
passthru.
When checking the users of an instruction, we currently bail if the user
is used as a passthru because all of its elements past vl will be used
for the tail.
We can allow passthru users if we know the tail of their result isn't
used, which we will have computed beforehand after #124530
It's worth noting that this is all irrelevant of the tail policy,
because tail agnostic still ends up using the passthru.
I've checked that SPEC CPU 2017 + llvm-test-suite pass with this (on
qemu with rvv_ta_all_1s=true)
Fixes#123760
This replaces the worklist by instead computing what VL is demanded by
each instruction's users first, which is done via checkUsers.
The demanded VLs are stored in a DenseMap, and then we can just do a
single forward pass of tryReduceVL where we check if a candidate's
demanded VL is less than its VLOp.
This means the pass should now be linear in complexity, and allows us to
relax the restriction on tied operands in more easily as in #124066.
Whilst adding a cross-block test, I encountered an assertion failure in
the second pass where we check the instruction popped off the worklist
is a candidate.
The leaf instruction %c in this case will be added to the worklist when
its VL is VLMAX, but during the first pass it will have its VL reduced
to 1.
Then in the second pass when its processed via the worklist, isCandidate
will no longer be true due to its VL == 1.
This fixes it by moving the VL == 1 check to tryReduceVL, keeping it
alongside the other VL check for bailing out early as an optimisation.
We currently check for passthrus in two places, on the instruction to
reduce in isCandidate, and on the users in checkUsers.
We cannot reduce the VL if an instruction has a user that's a passthru,
because the user will read elements past VL in the tail.
However it's fine to reduce an instruction if it itself contains a
non-undef passthru. Since the VL can only be reduced, not increased, the
previous tail will always remain the same.
We already bail if the user is tied in checkUsers, which is true for all
passthrus. Remove the check in getOperandLog2EEW so that it only worries
about computing the OperandInfo, and leaves the passthru correctness to
checkUsers.
This implements a suggestion by Craig in PR #123878. We can move the
worklist management out of the per-instruction work and do it once at
the end of scanning all the instructions. This should reduce repeat
visitation of the same instruction when no changes can be made.
Note that this does not remove the inherent O(N^2) in the algorithm.
We're still potentially visiiting every user of every def.
I also included a guard for unreachable blocks since that had been
mentioned as a possible cause. It seems we've rulled that out, but
guarding for this case is still a good idea.
For .wv widening instructions when checking if the opperand is vs1 or
vs2, we take into account whether or not it has a passthru. For tied
pseudos though their passthru is the vs2, and we weren't taking this
into account.
We can just use a std::optional to wrap the operand info instead. The
state field is confusing as we have a "partially known" state where EEW
is known and EMUL is nullopt, but it's still "Known".
All but one of the cases in tree today have EMUL=SEW/EEW*LMUL. Repeating
this each time is verbose and introduces oppurtunity for error. (For
instance, the comment associated with vwmul.vv was out of sync with the
code for same.)
Introduce getOperandLog2EEW and move most complexity to it. Then
introduce getOperandInfo as a wrapper around previous, and special case
the one case which requires it.
---------
Co-authored-by: Luke Lau <luke_lau@icloud.com>
When these instructions are marked nofpexcept, we can optimize them.
There are some added toggles in the output, likley because other
noexcept fp instructions are not part of isSupportedInstr yet. We may
want to avoid marking an instruction as isSupported in the future if any
of its FP users are missing nofpexcept to avoid added toggles. However,
we seem to get some GPRs back as a result of this change, which may
outweigh the cost of avoiding extra toggles.
The plan is to follow this patch up with added support for more FP
instructions in the same way. The instructions in this patch are a
natural starting point because they allow us to test with integer
instructions which have good support already.
Reductions are weird because for some operands, they are vector
registers but only read the first lane. For these operands, we do not
need to check to make sure the EEW and EMUL ratios match. The EEWs,
however, do need to match.
Although we cannot reduce the VL of these instructions (i.e. add to
isSupported) we can add them to getOperandInfo to enable optimization
where the FP vector instruction are users. Most of the instructions are
covered by existing tests, and I added tests for the narrowing
conversions because I was a little unsure whether the dest or the source
was 2*SEW and 2*LMUL.
Prior to this patch, we required that all users had the same VL in order
to optimize. But as the FIXME said, we can use the largest VL to
optimize, as long as we can determine what the largest is. This patch
implements the FIXME.
Use `MO.getOperandNo() == 0` instead of `IsMODef` so naming is clear for the store, since the store should treat its operand 0 like that even though it is not a def.The load should treat its operand 0 def in the same way.
I found this crash when trying to enable the VLOptimizer pass. We need
this patch before we can enable by default. The old assert was not
checking that USE and DEF were vector registers. The correct condition
is guarded at the callsite of tryReduceVL.
We need to adjust getEMULEqualsEEWDivSEWTimesLMUL to account for the
fact that Log2EEW for mask instructions is 0 but their EMUL is
calculated using Log2EEW=3.
