If the user node of the SExt/ZExt node is a bitcast to a float point
type, the node itself should not be considered legal to demote, since
still the casting is required to match the size of the float point type.
Fixes#157277
Technically this could happen with vector units that can't handle all legal scalar widths - but its good enough to use a generic crash test without a suitable target
Fixes#157335
Remove instcombine, simplifycfg and dce from some tests, as they make it
a bit more difficult to see the codegen coming out of LV and most
simplifications are already done on the VPlan-level.
Also modernizes some check lines.
foldCmpLoadFromIndexedGlobal() currently checks that the global type,
the GEP type and the load type match in certain ways. Replace this with
generic logic based on offsets.
This is a reboot of https://github.com/llvm/llvm-project/pull/67093.
This PR is less ambitious by requiring that the constant offset is
smaller than the stride, which avoids the additional complexity of that
PR.
In #157322 we crash because we try to infer a type for a VPReplicate
switch recipe.
My understanding was that these switches should be removed by
VPlanPredicator, but this switch survived through it because it was
unconditional, i.e. had no cases other than the default case.
This fixes#157322 by not emitting any recipes for unconditional
switches to begin with, similar to how we treat unconditional branches.
If a standalone schedule data relates to a vectorized instruction, still
need to schedule it as a part of pseudo-bundle to correctly handle
dependencies between its child nodes.
If a standalone schedule data relates to a vectorized instruction, still
need to schedule it as a part of pseudo-bundle to correctly handle
dependencies between its child nodes.
Look through extractvalue to simplify umul_with_overflow where one of
the operands is 1.
This removes some redundant instructions when expanding SCEVs, which in
turn makes the runtime check cost estimate more accurate, reducing the
minimum iterations for which vectorization is profitable.
PR: https://github.com/llvm/llvm-project/pull/157307
The SCEV predicate in the existing tests for optimizing for size is
known to be false. Add additional test with a predicate that cannot be
proven true/false.
Also generate checks with latest version of script.
Remove instcombine, simplifycfg and dce from some tests, as they make it
a bit more difficult to see the codegen coming out of LV and most
simplifications are already done on the VPlan-level.
Also modernizes some check lines.
Apply hints even if the attribute is the default "notcold" or
"ambiguous", to enable better tracking through the allocator.
Add an option to control the ambiguous allocation hint value.
Update hasChecks to use getSCEV/MemRuntimeChecks(), which automatically
handles checking for known-false checks.
This improves a few cases where we previously did not add metadata to
disable runtime unrolling, due to runtime checks, even though no runtime
checks are needed.
The second operand when using a safe divisor will always be a select in
the loop, so won't be invariant; don't treat it as such.
This fixes a divergence with legacy and VPlan based cost model.
Fixes https://github.com/llvm/llvm-project/issues/156066.
Fixes an issue where bits next to the sign bit were not constant-folded
when squaring a sign- or zero-extended small integer. Added logic to
detect when both operands of a multiplication are the same extended
value, allowing InstCombine to mark bits above the maximum possible
square as known zero. This enables correct folding of (x * x) & (1 << N)
to 0 when N is out of range.
Proof: https://alive2.llvm.org/ce/z/YGou44Fixes#152061
As observed by @mikaelholmen, PR #130988
"[AMDGPU] Implement IR expansion for frem instruction" introduced a
regression on x86. Its changes led to the pass being skipped on
functions with the optnone attribute. @bjope also noted that a check
concerning the optnone handling is wrong.
This patch fixes both issues which together fixes the regression. During
the review it was observed that, even before PR #130988, the pass would
not run on optnone functions with the new pass manager. This is also
fixed.
The 256-bit maximum vector register size control was removed from AVX10
whitepaper, ref: https://cdrdv2.intel.com/v1/dl/getContent/784343
We have warned these options in LLVM21 through #132542. This patch
removes underlying implementations in LLVM22.
This commit resolves a defect in the creation of DILabels for suspend
points. While a previous change introduced support for inlined
locations, it did not consider that a location might originate from
multiple inlinings. Therefore, we extend this to walk iteratively
through all `inlinedAt` locations to get to the root.
This refactors fixSuccessorPhis from
LoopIdiomVectorize::transformByteCompare and uses it in
LoopIdiomVectorize::expandFindFirstByte to ensure that all successor
Phis have incoming values from the vector basic blocks.
Fixes#156588.
---------
Co-authored-by: Ricardo Jesus <rjj@nvidia.com>
Optimize CRC loops using a Sarwate table-lookup by using the results of
HashRecognize in LoopIdiomRecognize. The optimization is checked for
correctness using the SingleSource/UnitTests/HashRecognize tests in
llvm-test-suite.
When estimating the cost to avoid exponential unswitches of non-trivial
invariant conditions, also consider the parent loop basic blocks size,
ensuring this does not grow unexpectedly.
Fixes: https://github.com/llvm/llvm-project/issues/138509.
Following up from #150368, this moves folding common edge masks into
simplifyBlends.
One test in uniform-blend.ll ended up regressing but after looking at it
closely, it came from a weird (x && !x) edge mask. So I've just included
a simplifcation in this PR to fold that to false.
In #149056 VF pruning was changed so that it only pruned VFs that
stemmed from MaxBandwidth being enabled.
However we always compute register pressure regardless of whether or not
max bandwidth is permitted for any VFs (via
`MaxPermissibleVFWithoutMaxBW`).
This skips the computation if not needed and renames the method for
clarity.
The diff in reg-usage.ll is due to the scalable VPlan not actually
having any maxbandwidth VFs, so I've changed it to check the
fixed-length VF instead, which is affected by maxbandwidth.
This patch splits out the legality checks from PR #151300, following the
landing of PR #128593.
It is a step toward supporting vectorization of early-exit loops that
contain potentially faulting loads.
In this commit, an early-exit loop is considered legal for vectorization
if it satisfies the following criteria:
1. it is a read-only loop.
2. all potentially faulting loads are unit-stride, which is the only
type currently supported by vp.load.ff.
The branch weights capture probability. The probability has everything to do with the (SSA) value the condition is predicated on, and nothing to do with the position in the CFG.
This patch consolidates updating loop metadata and profile info for both
the remainder and vector loops in a single place. This is NFC, modulo
consistently applying vectorization specific metadata also in the
experimental VPlan-native path.
Split off from https://github.com/llvm/llvm-project/pull/154510.
Commutable instruction can be reordering during tree building, and if
the parent node is not scheduled, its ScheduleData elements are
considered independent and compiler do not looks for reordered operands.
Need to cancel scheduling of copyables in this case.
When partially or runtime unrolling loops with reductions, currently the
reductions are performed in-order in the loop, negating most benefits
from unrolling such loops.
This patch extends unrolling code-gen to keep a parallel reduction phi
per unrolled iteration and combining the final result after the loop.
For out-of-order CPUs, this allows executing mutliple reduction chains
in parallel.
For now, the initial transformation is restricted to cases where we
unroll a small number of iterations (hard-coded to 4, but should maybe
be capped by TTI depending on the execution units), to avoid introducing
an excessive amount of parallel phis.
It also requires single block loops for now, where the unrolled
iterations are known to not exit the loop (either due to runtime
unrolling or partial unrolling). This ensures that the unrolled loop
will have a single basic block, with a single exit block where we can
place the final reduction value computation.
The initial implementation also only supports parallelizing loops with a
single reduction and only integer reductions. Those restrictions are
just to keep the initial implementation simpler, and can easily be
lifted as follow-ups.
With corresponding TTI to the AArch64 unrolling preferences which I will
also share soon, this triggers in ~300 loops across a wide range of
workloads, including LLVM itself, ffmgep, av1aom, sqlite, blender,
brotli, zstd and more.
PR: https://github.com/llvm/llvm-project/pull/149470
In case of equal costs Prefer epilogue with fixed-width over scalable VF.
That is helpful in cases like post-LTO vectorization where epilogue with
fixed-width VF can be removed when we eventually know that the trip count
is less than the epilogue iterations.
Update evaluatePtrAddrecAtMaxBTCWillNotWrap to support non-constant
sizes in dereferenceable assumptions.
Apply loop-guards in a few places needed to reason about expressions
involving trip counts of the from (BTC - 1).
PR: https://github.com/llvm/llvm-project/pull/156758
