The legacy cost model skips gather/scatters when determining the
predicated stores in useEmulatedMaskMemRefHack. Match the behavior in
the VPlan-based implementation. This fixes a cost divergence in the
attached test.
The best practices for llvm is to use the update scripts instead of
manually writing tests, so update tests to use the script, where the
change is subjectively minor. These tests are necessary to update again
for my branch to change CoroFrame to using all opaque ptradd instead of
typed gep and alloca. Currently that pass uses whatever random type the
largest non-overlapping alloca happened to have when the other alloca
were re-written to replace be replaced by it. In this future branch, it
will all just uses simple ptrgep operations. This is part of an
on-going series of PRs to fully deprecate using getAllocatedType and
replace it with getAllocationSize, because it had no semantic meaning,
just like typed pointers were deprecated for opaque pointers.
MemCopyOptimizer was merging a store instruction into a memset of an
`undef` value and emitted a large memset for the memset and store
combined.
This PR prevents MemCopyOptimizer from merging a store instruction into
a
memset of an `undef` value since it can be removed by subsequent cleanup
passes.
Helps https://github.com/rust-lang/rust/issues/152541
Directly unroll VectorEndPointerRecipe following 0636225b ([VPlan]
Directly unroll VectorPointerRecipe, #168886). It allows us to leverage
existing VPlan simplifications to optimize.
Co-authored-by: Luke Lau <luke@igalia.com>
Co-authored-by: Florian Hahn <flo@fhahn.com>
Only use a PtrToInt's source pointer as an available pointer value when
its pointer type exactly matches the inttoptr target type. This prevents
creating an invalid bitcast between differing address spaces during
foldIntegerTypedPHI.
This reverts commit 8d29d09309654541fb2861524276ada6a3ebf84c.
The underlying issue causing the revert has been fixed independently
as 301fa24671256734df6b7ee65f23ad885400108e.
Original message:
Move narrowInterleaveGroups to to general VPlan optimization stage.
To do so, narrowInterleaveGroups now has to find a suitable VF where all
interleave groups are consecutive and saturate the full vector width.
If such a VF is found, the original VPlan is split into 2:
a) a new clone which contains all VFs of Plan, except VFToOptimize, and
b) the original Plan with VFToOptimize as single VF.
The original Plan is then optimized. If a new copy for the other VFs has
been created, it is returned and the caller has to add it to the list of
candidate plans.
Together with https://github.com/llvm/llvm-project/pull/149702, this
allows to take the narrowed interleave groups into account when
computing costs to choose the best VF and interleave count.
One example where we currently miss interleaving/unrolling when
narrowing interleave groups is https://godbolt.org/z/Yz77zbacz
PR: https://github.com/llvm/llvm-project/pull/149706
Resolves#73417.
The fold described in #73417 is already present in LLVM `main`, but it
isn't tested for specifically. This PR adds a test for this fold, based
on the IR in the topmost comment of that issue.
Because in the new IR pattern for conditional traps the llvm.cond.loop
is generated late, the LowerTypeTests pass will never see it, so we no
longer need to be able to optimize it. Therefore, remove the code for
optimizing it.
This reverts commit 1de721c414407f4b92d3e2458ce9ca38563cef9c.
Reviewers: fmayer, vitalybuka
Reviewed By: fmayer
Pull Request: https://github.com/llvm/llvm-project/pull/181301
The '``llvm.looptrap``' intrinsic is equivalent to
``llvm.cond.loop(true)``, but is also considered to be ``noreturn``,
which enables certain optimizations by allowing the optimizer to
assume that a branch leading to a call to this intrinsic was not
taken. A late optimization pass will convert this intrinsic to either
``llvm.cond.loop(true)`` or ``llvm.cond.loop(pred)``, where ``pred``
is a predicate for a conditional branch leading to the intrinsic call,
if possible.
Reviewers: fmayer, vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/181299
Compute the number of predicated stores directly in VPlan instead of
using CM.useEmulatedMaskMemRefHack(), which will only account for the
number of predicated stores for the last VF the legacy cost model
considered.
Fixes https://github.com/llvm/llvm-project/issues/181183
This can come from loops like
```
for (int i = 0; i < X; ++i) {
if (i < N)
__builtin_trap();
if (i < M)
__builtin_trap();
x[i] = y[i];
}
```
Reviewers: nikic
Reviewed By: nikic
Pull Request: https://github.com/llvm/llvm-project/pull/181264
Building on top of the recent changes to introduce BranchOnTwoConds,
this patch adds support for vectorizing loops with multiple early exits,
all dominating a countable latch. The early exits must form a
dominance chain, so we can simply check which early exit has been taken
in dominance order.
Currently LoopVectorizationLegality ensures that all exits other than
the latch must be uncountable. handleUncountableEarlyExits now collects
those uncountable exits and processes each exit.
In the vector region, we compute if any exit has been taken, by taking
the OR of all early exit conditions (EarlyExitConds) and checking if
there's
any active lane.
If the early exit is taken, we exit the loop and compute which early
exit
has been taken. The first taken early exit is the one where its exit
condition is true in the first active lane of EarlyExitConds.
We create a chain of dispatch blocks outside the loop to check this for
the early exit blocks ordered by dominance.
Depends on https://github.com/llvm/llvm-project/pull/174016.
PR: https://github.com/llvm/llvm-project/pull/174864
narrowToSingleScalarRecipes' operands check is a bit too restrictive by
permitting a single user. Factor out and reuse the existing
introduces-broadcast logic to improve results.
In some cases, LV gets simplifyable IR as input. Directly apply
simplifications on the initial VPlan0 to avoid vectorization in cases
where the loop body can be folded away.
Using the end-to-end pipeline, this is relatively rare, but when
reducing test cases, the reduction often ends up with cases with trivial
folds. Rejecting those will result in more robust & realistic test
cases.
As follow-up, I also plan to add initial dead recipe removal.
Depends on https://github.com/llvm/llvm-project/pull/176795.
PR: https://github.com/llvm/llvm-project/pull/176828
If the predicate has samesign set, we could either perform the checks
with the unsigned predicate and return and unsigned invariant predicate,
or we could perform them with the signed predicate and return a signed
invariant predicate. The current implementation can end up mixing both,
using a signed predicate for one check and an unsigned one for the
other.
Avoid this by dropping the samesign flag.
Fixes https://github.com/llvm/llvm-project/issues/180870.
getTruncateExpr may not always return a SCEVAddRecExpr when truncating
loop bounds. Add a check to verify the result type before casting, and
bail out of the transformation if the cast would be invalid.
This prevents potential crashes from invalid casts when dealing with
complex loop bounds.
Co-authored by Michael Rowan
Resolves
[https://github.com/llvm/llvm-project/issues/153090](https://github.com/llvm/llvm-project/issues/153090)
Recipes can be sunk now. In those cases, the sunk recipes are outside
the loop region, but may not be uniform across VF and UF.
Update the code to only exit early if the recipe is defined before the
region. Without DT available, the easiest way to check is just if it is
in the entry/preheader block.
Fixes https://github.com/llvm/llvm-project/issues/181002.
Fixes https://github.com/llvm/llvm-project/issues/180781.
Currently, separate-const-offset-from-gep tries to combine both the
offsets on the base address and the offsets on the current GEP itself
when it tries to separate constant offsets. This results in the pass
failing to separate the offset in cases where the base address has a
large offset that would cause the total offset to be larger than what
the back-end can represent for the respective addressing mode. However,
in many cases we can still benefit from extracting the offset of the GEP
without including the offset from the base-address-GEP when the
base-address is used in multiple different places to help reduce
register pressure and recalculating the base address.
This PR addresses the above by re-trying without accumulating the offset
from the base-address GEP included if isLegalAddressingMode returns
false the first time.
Emit per-context optimization remarks, when we apply a hint
unambiguously to an allocation, including the full allocation context
hash and total profiled size. When no context size information is
available, we fall back to a single remark per allocation. While this is
somewhat redundant with the message emitted for
-memprof-report-hinted-sizes, the latter enables emitting additional
messages, and so it is left in for now.
To support this consistently, added a new internal option
-memprof-keep-context-size-info that ensures context size information
is tracked and available for these remarks even when hinted size
reporting isn't fully enabled. This also will be useful for providing
this additional information to the LTO link for remarks there (in a
subsequent PR).
Updated memprof.ll to verify both the detailed per-context remarks
(when size info is kept) and the original per-allocation remarks (when
it is not).
Switch tests from using `-debug[-only=LoopVectorize]` to
`-vplan-print-after` as that provides better control at what step in the
pipeline we want to check the VPlan (I'm using `optimize$` for now to
preserve previous state).
Then, update `-vplan-print-after*` to print what function the loop
belongs to. That enables us to simplify VPlan UTC support as the output
of the updated tests contains the VPlan dump only - no special
filtering/extraction is necessary anymore.
Add a new helper function `matchEquivZeroRHS()` that recognizes
comparisons with constants that are equivalent to comparisons with zero,
and transforms the predicate accordingly.
This handles the following transformations:
- icmp sgt X, -1 --> icmp sge X, 0
- icmp sle X, -1 --> icmp slt X, 0
- icmp [us]ge X, 1 --> icmp [us]gt X, 0
- icmp [us]lt X, 1 --> icmp [us]le X, 0
This enables more optimization opportunities in `simplifyICmpWithZero`,
such as folding icmp sgt X, -1 when X is known to be non-negative.
---
- IR Impact: https://github.com/dtcxzyw/llvm-opt-benchmark/pull/3414
We reassociate ((x && y) && z) -> (x && (y && z)) if x has more than
use, in order to allow simplifying the header mask further. However this
is somewhat unreliable as there are times when it doesn't have more than
one use, e.g. see the case we run into in
https://github.com/llvm/llvm-project/pull/173265/changes#r2769759907.
This moves it into a separate transformation that always reassociates
the header mask regardless of the number of uses, which prevents some
fragile test changes in #173265.
We need to run it before both calls to simplifyRecipes in optimize. I
considered putting it in simplifyRecipes itself but simplifyRecipes is
also called after unrolling and when the loop region is dissolved which
causes vputils::findHeaderMask to assert.
There isn't really any benefit to reassociating masks that aren't the
header mask so the existing simplification was removed.
Add a new VPlan subdirectory as common place for tests checking VPlan
printing. It contains a lit.local.cfg that only runs the tests when
assertions are enabled.
This removes the need to add explicit REQUIRES: asserts to VPlan tests.
PR: https://github.com/llvm/llvm-project/pull/180611
Added basic estimations for the external uses, when calculating the cost
of the non-profitable trees. Excluding stores/insertelement, as thay are
very good candidates for the vectorization. Also, tuned
buildvector/gather cost with minimum bitwidth analysis data.
Reviewers: hiraditya, RKSimon
Pull Request: https://github.com/llvm/llvm-project/pull/178024
The legacy cost model tries to scalarize loads that are used as
pointers. Skip if the load would need predicating when scalarized,
because that would incur very high costs, see useEmulatedMaskMemRefHack.
Fixes https://github.com/llvm/llvm-project/issues/180780.
If the gathered loads nodes are deleted for deletion, need to actually
deleted them from tree. Also, if the remaining tree is too short
(buildvector + gather node), need to skip such trees to avoid hanging.
Fixes#180846
Unify the ad-hoc use of whitespace in `LLVM_DEBUG()` messages.
This approach should also make it easier to see which loop debug
messages correspond to and which part of the loop unrolling heuristics
each message corresponds to.
Update LowerMatrixIntrinsics to use tiled loops automatically in for
larger matrixes. The fully unrolled codegen creates a huge amount of
code, which performs noticably worse then the tiled loop nest variant.
We new try to estimate the number of instructions needed for the
multiply, and if it is too large, tiled loops are used. The current
threshold is anything roughly larger than 6x6x6 double multiply.
Eventually I think we want to only generate tiled loops. This patch is a
first step, trying to opt in for cases where we know it is beneficial.
Checked on AArch64, but should help on other architectures similarly,
and also drastically reduce binary size + compile time.
PR: https://github.com/llvm/llvm-project/pull/179325
Add support for expanding fptosi.sat and fptoui.sat via IR expansions.
Similar to fptosi/fptoui we would get legalization errors otherwise.
The previous expansion for fptosi/fptoui was already saturating -- but
those instructions do not actually require saturation, and the
implementation of the saturation was incorrect in lots of ways. What
this PR does is:
* For fptosi, remove the unnecessary saturation handling.
* For fptoui, remove the unnecessary saturation handling and sign
multiplication.
* For fptosi, use the previous saturation handling with fixes: We need
to map NaNs to 0 and the saturation condition on the exponent was
incorrect. (I'm performing the NaN check via fcmp -- there's no
requirement to do everything bitwise here.)
* For fptoui use a variation of the signed saturation handling: Negative
values need to go to zero and we saturate to unsigned max.
Proofs: https://alive2.llvm.org/ce/z/Xv9FNd
If the instructions are compatible but non-matching (zext-select pair as
example), no need to perform operands analysis, just return that they
are matching.
The checks created by LAA only compute a pointer difference and do not
need to capture provenance. Use SCEVPtrToAddr instead of SCEVPtrToInt
for computations.
To avoid regressions while parts of SCEV are migrated to use PtrToAddr
this adds logic to rewrite all PtrToInt to PtrToAddr if possible in the
created expressions. This is needed to avoid regressions.
Similarly, if in the original IR we have a PtrToInt, SCEVExpander tries
to re-use it if possible when expanding PtrToAddr.
Depends on https://github.com/llvm/llvm-project/pull/178727.
Fixes https://github.com/llvm/llvm-project/issues/156978.
PR: https://github.com/llvm/llvm-project/pull/178861
When vectorising calls to math intrinsics such as llvm.pow we
correctly detect and generate calls to the corresponding vector
math variant. However, we don't pick up and use the calling
convention for the vector math function. This matters for veclibs
such as ArmPL where the aarch64_vector_pcs calling convention
can improve codegen by reducing the number of registers that
need saving across calls.
getTruncateExpr may not always return a SCEVAddRecExpr when truncating
loop bounds. Add a check to verify the result type before casting, and
bail out of the transformation if the cast would be invalid.
This prevents potential crashes from invalid casts when dealing with
complex loop bounds.
Co-authored by Michael Rowan
Resolves [#153090](https://github.com/llvm/llvm-project/issues/153090)
