Given a set of pointers, check if they can be rearranged as follows (%s is a constant):
%b + 0 * %s + 0
%b + 0 * %s + 1
%b + 0 * %s + 2
...
%b + 0 * %s + w
%b + 1 * %s + 0
%b + 1 * %s + 1
%b + 1 * %s + 2
...
%b + 1 * %s + w
...
If the pointers can be rearanged in the above pattern, it means that the
memory can be accessed with a strided loads of width `w` and stride `%s`.
Do not consider loops with a zero backedge taken count as candidates for
interchange. This seems like a sensible thing because it suggests the loop
doesn't execute and there is no point in interchanging. As a bonus, this
seems to avoid triggering an assert about phis and their uses from source
code, so this is a partial fix for #163954 but it needs more work to properly
fix that.
This patch replaces the delinearization function used in DA, switching
from one that depends on type information in GEPs to one that does not.
There are three types of changes in regression tests: improvements,
degradations, and degradations but the related features will be
removed. Since there were very few cases that are classified into the
second category, I believe the impact of this change should be
practically insignificant.
Consider skipping epilogue scalable VF when they are greater than
RemainingIterations same as fixed VF.
And skip scalable RemainingIterations from that comparison because
SCEV ATM can't evaluate non-canonical vscale-based expressions.
We don't have enough information to infer the probability of a weak function pointer being nullptr or not (open question if we could propagate this from the linker)
Issue #147390
FCmp instructions have both a predicate and fast-math flags. Introduce a
new FCmp kind, that combines both to model this correctly in the current
system.
This should be NFC modulo VPlan printing which now includes the correct
fast-math flags.
Follow up on a cse OpType-mismatch crash reported due to ef023cae388d
(Reland [VPlan] Expand WidenInt inductions with nuw/nsw), setting the
OpType correctly when returning from getFlagsFromIndDesc.
Identity masks can be treated as free when scalable vectorization is
possible making the check agnostic of the vectorization policy
fixed/scalable, This allows for aggressive vector combines for identity
shuffle masks.
https://alive2.llvm.org/ce/z/YGT5SNhttps://alive2.llvm.org/ce/z/PVDxCwhttps://alive2.llvm.org/ce/z/8buR2N
This is tricky because with positive numbers, we only go up, so we can
in fact always hit the signed_max boundary. This is important because
the intrinsic we use has the behavior of going the OTHER way, aka clamp
to INT_MIN if it goes in that direction.
And the range checking we do only works for positive numbers.
Because of this issue, we can only do this for constants as well.
Update VPlan to populate VPIRFlags during VPInstruction construction and
use it when creating widened recipes, instead of constructing VPIRFlags
from the underlying IR instruction each time. The VPRecipeWithIRFlags
constructor taking an underlying instruction and setting the flags based
on it has been removed.
This centralizes initial VPIRFlags creation and ensures flags are
consistently available throughout VPlan transformations and makes sure
we don't accidentally re-add flags from the underlying instruction that
already got dropped during transformations.
Follow-up to https://github.com/llvm/llvm-project/pull/167253, which did
the same for VPIRMetadata.
Should be NFC w.r.t. to the generated IR.
PR: https://github.com/llvm/llvm-project/pull/168450
This patch introduces preliminary support for additional memory
locations.
They are: target_mem0 and target_mem1 and they model memory locations
that cannot be represented with existing memory locations.
It was a solution suggested in :
https://discourse.llvm.org/t/rfc-improving-fpmr-handling-for-fp8-intrinsics-in-llvm/86868/6
Currently, these locations are not yet target-specific. The goal is to
enable the compiler to express read/write effects on these resources.
This patch implements a transform to hoists single-scalar replicated
loads with invariant addresses out of the vector loop to the preheader
when scoped noalias metadata proves they cannot alias with any stores in
the loop.
This enables hosting of loads we can prove do not alias any stores in
the loop due to memory runtime checks added during vectorization.
PR: https://github.com/llvm/llvm-project/pull/166247
This is a small code size optimization that lets us avoid both shifting
and comparing to a constant if we need the shifted value anyway. On most
architectures the zero comparison is cheaper than a constant comparison
(or free if the shift sets flags).
Although this change appears to remove the optimization entirely, we
continue to do this transform if there is one use because of the code
below the removed code that transforms the shift into an and, followed
by the PR10267 case in InstCombinerImpl::foldICmpAndConstConst that
transforms the and into a ult/ugt. Added a test case to verify this
explicitly.
Per [1] reduces clang .text size by 0.09% and dynamic instruction count
by 0.01%.
[1] https://llvm-compile-time-tracker.com/compare.php?from=1f38d49ebe96417e368a567efa4d650b8a9ac30f&to=0873787a12b8f2eab019d8211ace4bccc1807343&stat=size-text
Reviewers: nikic, dtcxzyw
Reviewed By: dtcxzyw
Pull Request: https://github.com/llvm/llvm-project/pull/168007
We build the callsite graph by first adding nodes and edges for all
allocation contexts, then match the interior callsite nodes onto actual
calls (IR or summary), which due to inlining may result in the
generation of new nodes representing the inlined context sequence. We
attempt to update edges correctly during this process, but in the case
of recursion this becomes impossible to always get correct.
Specifically, when creating new inlined sequence nodes for stack ids on
recursive cycles we can't always update correctly, because we have lost
the original ordering of the context.
This PR introduces a mechanism, guarded by -memprof-top-n-important=
flag, to keep track of extra information for the largest N cold
contexts. Another flag -memprof-fixup-important (enabled by default)
will perform more expensive fixup of the edges for those largest N cold
contexts, by saving and walking the original ordered list of stack ids
from the context.
Changes: The previous patch had to be reverted to a mismatching-OpType
assert in cse. The reduced-test has now been added corresponding to a
RVV pointer-induction, and the pointer-induction case has been updated
to use createOverflowingBinaryOp.
While at it, record VPIRFlags in VPWidenInductionRecipe.
For a scalar only VPlan with tail folding, if it has a phi live out then
legalizeAndOptimizeInductions will scalarize the widened canonical IV
feeding into the header mask:
<x1> vector loop: {
vector.body:
EMIT vp<%4> = CANONICAL-INDUCTION ir<0>, vp<%index.next>
vp<%5> = SCALAR-STEPS vp<%4>, ir<1>, vp<%0>
EMIT vp<%6> = icmp ule vp<%5>, vp<%3>
EMIT vp<%index.next> = add nuw vp<%4>, vp<%1>
EMIT branch-on-count vp<%index.next>, vp<%2>
No successors
}
Successor(s): middle.block
middle.block:
EMIT vp<%8> = last-active-lane vp<%6>
EMIT vp<%9> = extract-lane vp<%8>, vp<%5>
Successor(s): ir-bb<exit>
The verifier complains about this but this should still generate the
correct last active lane, so this fixes the assert by handling this case
in isHeaderMask. There is a similar pattern already there for
ActiveLaneMask, which also expects a VPScalarIVSteps recipe.
Fixes#167813
- Introduce the -aarch64-force-unroll-threshold option; when a loop’s
cost is below this value we set UP.Force = true (default 0 keeps current
behaviour)
- Add an AArch64 loop-unroll regression test that runs once at the
default threshold and once with the flag raised, confirming forced
unrolling
The compiler should not consider split vectorize nodes, when checking
for non-schedulable PHI-based parent nodes. Only pure PHI nodes must be
considered, they only can be considered as explicit users, split nodes
are not.
Fixes#168268
In https://github.com/llvm/llvm-project/pull/165748 constant expressions
were allowed in `collectPossibleValues` because we are still using
insertelement + shufflevector idioms to represent a scalable vector
splat. However, it also accepts some unresolved constants like ptrtoint
of globals or pointer difference between two globals. Absolutely we can
ask the user to check this case with the constant folding API. However,
since we don't observe the real-world usefulness of handling constant
expressions, I decide to be more conservative and only handle immediate
constants in the helper function. With this patch, we don't need to
touch the SimplifyCFG part, as the values can only be either ConstantInt
or undef/poison values (NB: switch on undef condition is UB).
Fix the miscompilation reported by
https://github.com/llvm/llvm-project/pull/165748#issuecomment-3532245218
Extend willNotFreeBetween to perform simple checking across blocks to
support the case where CtxI is in a successor of the block that contains
the assume, but the assume's parent is the single predecessor of CtxI's
block.
This enables using _builtin_assume_dereferenceable to vectorize
std::find_if and co in practice.
End-to-end reproducer: https://godbolt.org/z/6jbsd4EjT
PR: https://github.com/llvm/llvm-project/pull/167965
When simplifying min/max intrinsics with fixed-size vector constants,
InstructionSimplify attempts to optimize element-wise. However,
getAggregateElement() can return null for certain constant expressions
like bitcasts, leading to a null pointer dereference.
This patch adds a check to bail out of the optimization when
getAggregateElement() returns null, preventing the crash while
maintaining correct behavior for normal constant vectors.
Fixes crash with patterns like:
call <2 x half> @llvm.minnum.v2f16(<2 x half> %x,
<2 x half> bitcast (<1 x i32> <i32 N> to <2 x half>))
Patch adds support for sub instructions as main instruction in copyables
elements. Also, adds a check if the base instruction is not profitable
for the selection if at least one instruction with the main opcode is
used as an immediate operand.
Reviewers: RKSimon, hiraditya
Reviewed By: RKSimon
Pull Request: https://github.com/llvm/llvm-project/pull/163231
Propagate branch weights in `mergeComparisons` : the probability of reaching the common "exit" BB (`bb_phi` in the description in `processPhi`)doesn't change, and is a disjunction over the probabilities of doing that from the blocks performing comparisons which are now being merged
Issue #147390
Addresses the issues found on the review of
https://github.com/llvm/llvm-project/pull/150267/files#r2356936355
Currently when collecting the users of an IFunc symbol to determine the
callers, we incorrectly mix versions of different functions together,
alongside non-FMV callers all in the same bag. That is problematic
because we incorrectly deduce which features are unavailable as we
iterate the callers.
I have updated the unit tests to require a resolver function for the
callers and regenerated the resolvers since some FMV features have been
removed making the detection bitmasks different. I've replaced the
deleted FMV feature ls64 with cssc. I've added a new test to cover
unrelated callers.
The IR verifier will carsh if there is any instructions located before
phi-node. The `infer-address-spaces` pass would like to insert
`addrspacecast` before phi-node in some corner cases. Indeed, since the
operand pointer(phi-node's incoming value) has been determined to
`NewAS` by the pass, it is safe to `addrspacecast` it immediately after
the position where defined it.
Co-authored-by: Kerang Mao <krmao@birentech.com>
