The existing condition for checking whether or not to expand an frem
instruction in expand-fp is not sufficiently precise.
The expansion on other targets than AMDGPU - which is the only intended
user right now - is only prevented due to the interaction with the
MaxLegalFpConvertBitWidth check. Relying on this is conceptually wrong
and limits the use of the pass for other targets and further expansions
(e.g. merging with the similar ExpandLargeDivRem pass).
Change the expansion criterion to always expand frem of a given type
for targets that use "Expand" as the legalization action for the
underlying scalar type and use this to exit the pass early for targets
which do not require any expansions. This requires to change the
frem legalization action for all targets which do not want frem to
be expanded in this pass from "Expand" to "LibCall".
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
This builds on #169858 to fix the divergence in codegen
(https://godbolt.org/z/a9az3h6oq) between two very similar
functions initially observed in #137447 (represented in the diff by test
cases `@transpose_splat_constants` and `@transpose_constants_splat`:
```
int8x16_t f(int8_t x)
{
return (int8x16_t) { x, 0, x, 1, x, 2, x, 3,
x, 4, x, 5, x, 6, x, 7 };
}
int8x16_t g(int8_t x)
{
return (int8x16_t) { 0, x, 1, x, 2, x, 3, x,
4, x, 5, x, 6, x, 7, x };
}
```
The PR uses an additional `isTRNMask` call in
`AArch64TTIImpl::getShuffleCost` to ensure that we treat shuffle masks
as transpose masks even if `isTransposeMask` fails to recognise them
(meaning that `Kind == TTI::SK_Transpose` cannot be relied upon).
Follow-up work could consider modifying `isTransposeMask`, but that
would also impact other backends than AArch64.
This PR adds the cost model for the loop dependence mask intrinsics,
both for cases where they must be expanded and when they can be lowered
for AArch64.
---------
Co-authored-by: Benjamin Maxwell <benjamin.maxwell@arm.com>
@llvm.experimental.vp.splat was originally added in #98731 in order to
prevent VL toggles when optimizing a zero strided load to a scalar load
+ splat on RISC-V: #101329
However, the need to explicitly set the VL operand has been superseded
by RISCVVLOptimizer which can infer this automatically based on the
splat's users, and the use of the vp.splat intrinsic was removed in
#170543.
Now that there are no users of @llvm.experimental.vp.splat internal to
LLVM and it's unlikely we will need it in future due to
RISCVVLOptimizer, this patch removes the intrinsic. I couldn't find any
publicly available out-of-tree users of the intrinsic with a quick
search on GitHub.
A funnel shift with the same first two operands is a rotate. When
`Zbb/Zbkb` is enabled we can use the `ROL(W)/ROR(I)(W)` instruction to
represent this. Add cost model support for this.
Similar to https://github.com/llvm/llvm-project/pull/169335 for AArch64.
New SFB subsets are being added with the scheduler class name as a
suffix, so now is the time to go back to the base extension and add IALU
to its name.
This also:
- Drops a hyphen from the other SFB features for mul and minmax, to more
closely match their scheduling classes.
- Updates the predicates on specific SFB pseudos so we get verifier
errors if we introduce the pseudos when we don't have the right
subtarget feature.
- Updates the SFB Documentation comment to make it no longer
SiFive-specific.
This patch is a rework of #160470 (which was reverted).
With getMemIntrinsicCost() now available, we can re‑land the change and
reduce vp_load_ff boilerplate.
A funnel shift with the first two operands is a rotate. AArch64 has
scalar instructions for ror so can handle a ROTR by a constant or
variant amount cheaply. A ROTL is a ROTR by the opposite amount, which
for constants is cheap and for variable shifts requires an extra neg.
These shuffles can always be implemented using v_perm_b32, and so this
rewrites the analysis from the perspective of "how many v_perm_b32s does
it take to assemble each register of the result?"
The test changes in Transforms/SLPVectorizer/reduction.ll are
reasonable: VI (gfx8) has native f16 math, but not packed math.
Identity masks can be treated as free when scalable vectorization is
possible making the check agnostic of the vectorization policy
fixed/scalable, This allows for aggressive vector combines for identity
shuffle masks.
This patch adds cost model tests for `bfloat` operations with
`+sve-b16b16`. Currently, some of these costs are higher than they
should be as the cost model is assuming `bfloat`s need promotion, but
some of these operations are natively supported with `+sve-b16b16`.
Avoid weird lambda returning function pointer and sink the libcall
logic to where the operation is handled. This allows chaining the
libcall logic to try sincos_stret and fallback to sincos. The resulting
cost seems too low.
We already have cost model code for detecting extending mull multiplies
for the form `mul(ext, ext)`. Since it was added the codegen for mull
has been improved, this attempts to catch the cost model up.
The main idea is to incorporate extends of larger sizes. A vector `v8i32
mul(zext(v8i8), zext(v8i8))` will be code-generated as `zext (v8i16
mul(zext(v8i8), zext(v8i8))`, or umull+ushll+ushll2.
So the total cost should be 3ish if each instruction costs 1. Where
exactly we attribute the costs is dependable, this patch opts to sets
the cost of the extend to 0 (or the cost of the extend not included in
the mull) and the mul gets the cost of the mull+extra extends.
isWideningInstruction is split into two functions for the two types of
operands it supports. isSingleExtWideningInstruction now handles addw
instructions that extend the second operand, isBinExtWideningInstruction
is for instructions like addl that extend both operands.
getIntrinsicInstrCost should halve the cost returned by getTypeLegalizationCost
when the return type requires splitting, but we know that the whilelo
(predicate pair) instruction can be used.
When splitting is still required, the cost get_active_lane_mask should also
reflect the additional saturating add required to increment the start value.
The `masked.load`, `masked.store`, `masked.gather` and `masked.scatter`
intrinsics currently accept a separate alignment immarg. Replace this
with an `align` attribute on the pointer / vector of pointers argument.
This is the standard representation for alignment information on
intrinsics, and is already used by all other memory intrinsics. This
means the signatures now match llvm.expandload, llvm.vp.load, etc.
(Things like llvm.memcpy used to have a separate alignment argument as
well, but were already migrated a long time ago.)
It's worth noting that the masked.gather and masked.scatter intrinsics
previously accepted a zero alignment to indicate the ABI type alignment
of the element type. This special case is gone now: If the align
attribute is omitted, the implied alignment is 1, as usual. If ABI
alignment is desired, it needs to be explicitly emitted (which the
IRBuilder API already requires anyway).
Whenever someone modifies an existing test that has `undef` in it the
github code formatter will complain so it's not easy to know if it's due
to a new or old use. I figured I may as well just do a simple sed
replace of undef with poison in all the tests to clean them up.
Hopefully it makes the contribution process a bit easier.
Split out from #151300 to isolate TargetTransformInfo cost modelling for
fault-only-first loads from VPlan implementation details. This change
adds costing support for vp.load.ff independently of the VPlan work.
For now, model a vp.load.ff as cost-equivalent to a vp.load.
We need to account for AVX1 targets splitting 256-bit PMULUDQ style
patterns - we included a AVX2 cost, but missed out AVX1 despite v4i64
being a legal type
Noticed while working on #156568
This tries to add a higher cost for SVE FCM** comparison instructions
that often have a lower throughput than the Neon equivalents that can be
executed on more vector pipelines.
This patch takes the slightly unorthodox approach of using the
information in the scheduling model to compare the throughput of a
FCMEQ_PPzZZ_S (SVE) and a FCMEQv4f32 (Neon). This isn't how things will
(probably) want to work in the long run, where all the information comes
more directly from the scheduling model, but that still needs to be
proven out. The downsides of this approach of using the scheduling model
info is if the core does not have a scheduling model but wants a
different cost - then an alternative approach will be needed (but then
maybe that is a good reason to create a new scheduling model).
The alternative would either be to make a subtarget feature for the
affected cores or just always enable it.
Previously, only 128-bit "NEON" vectors were given sensible costs.
Cores with vscale>1 can use SVE's EXT instruction to perform a
fixed-length subvector extract.
This is a follow-up from the codegen patches at #152554. They show that
with the help of MOVPRFX, we can do subvector extracts with roughly one
instruction. We now at least give sensible costs for extracting 128-bit
halves from a 256-bit vector.
The cost model previously overestimating throughput costs to wide
fixed-length saturating arithmetic intrinsics when using SVE with a
fixed vscale of 2. These costs ended up much higher than for the same
operations using NEON, despite being fully legal and efficient with SVE.
This patch adjusts the cost model to avoid penalising these intrinsics
under SVE.
As fmul and fmadd are so similar, their performance characteristics tend
to be the same on most platforms, at least in terms of reciprocal
throughputs. Processors capable of performing a given number of fmul per
cycle can usually perform the same number of fma, with the extra add
being relatively simple on top. This patch makes the scores of the two
operations the same, which brings the throughput cost of a fma/fmuladd
to 2, and the latency to 3, which are the defaults for fmul.
Note that we might also want to change the throughput cost of a fmul to
1, as most processors have ample bandwidth for them, but they should
still stay in-line with one another.
LD2 is represented in IR as deinterleave-shuffle(load), and ST2 as
store(interleave-shuffle). Whilst the shuffle would be expensive in
general for MVE (it does not have zip/uzp instructions), it should be
treated as cheap when part of the LD2/ST2 pattern. This borrows some
code from the AArch64 backed to produce lower costs. (Some of which
still shows as higher than it should - that just shows how broken the
generic shuffle costs are at the moment, they would be lower if
getShuffleCost was called directly as opposed to going through
getInstructionCost).
If we have a floating point vector and no zve32f/zve64f/zve64d, we can
end up with an invalid type-legalization cost from
getTypeLegalizationCost.
Previously this triggered an assertion that the type must have been
legalized if the "legal" type is a vector, but in this case when it's
not possible to legalize the original type is spat back out.
This fixes it by just checking that the legalization cost is valid.
We don't have much testing for zve64x, so we may have other places in
the cost model with this issue.
Fixes#153008
This extracts the code for modelling a fp16 operation as
`fptrunc(fpop(fpext,fpext))` into a new function named
getFP16BF16PromoteCost so that it can be reused by the
arithmetic instructions. The function takes a lambda to
calculate the cost of the operation with the promoted type.
Now that #149310 has restricted lifetime intrinsics to only work on
allocas, we can also drop the explicit size argument. Instead, the size
is implied by the alloca.
This removes the ability to only mark a prefix of an alloca alive/dead.
We never used that capability, so we should remove the need to handle
that possibility everywhere (though many key places, including stack
coloring, did not actually respect this).
