If the function dependencesAllowFusion returns false, in fuseCandidates
the reportLoopFusion function is used to increment InvalidDependencies
and to emit a OptimizationRemarkMissed. If both dependencesAllowFusion
and reportLoopFusion increment InvalidDependencies, statistics will
appear duplicated
`LoopVectorizationCostModel::getPredBlockCostDivisor(...)` may return
large `uint64_t` values that get coerced to an `unsigned` by
`VPCostContext::getPredBlockCostDivisor(...)`, which can cause division
by zero.
Fixes#187584
Initially, the reduction root was detected using the last member of the UserIgnoreList set, which is unordered. Better to use the reduction root explicitly to avoid non-determinism in the reduction parent block, which may cause incorrect scale factor estimation for the reduction cost.
If the const values have more active bits, than requested by the another
operand of the compare, such constants should not be trunced to avoid
miscompilation
Added two patterns for IR pattern matching, `m_IToFP` and `m_FPToI`
which are basically shortcuts of `m_CombinedOr(..., ...)`
> if there isn't already one, PatternMatch should have an m_ItoFP which
covers both
_Originally posted by @arsenm in
https://github.com/llvm/llvm-project/pull/185826#discussion_r2967473936_
/cc @arsenm
If the const values have more active bits, than requested by the another
operand of the compare, such constants should not be trunced to avoid
miscompilation
Resolves: https://github.com/llvm/llvm-project/issues/174933
The issue goes into a case where fetch_sub(n) is properly optimized but
fetch_add(neg(n)) is not optimized to the same code.
Although the issue is tagged for x86 I assumed this be best handled
outside of the backends so I put this in InstCombine.
This patch fixes the update of the DAGNode UnscheduledSucc counter when
a use edge is modified. This is the result of a setOperand() or a RAUW
(and friends) operation.
Before this patch we would not check if the User (i.e., the consumer of
the use-def edge) is scheduled and we would update the definition's
UnscheduledSucc counter, resulting in counting errors.
For example, consider the following IR:
```
%A = ...
%B = ...
%U = %A ; scheduled
```
Note that %U's DAGNode is marked as "scheduled" while %A and %B are not.
If we change %U's operand from %A to %B then we should not attempt to
update %A's or %B's UnscheduledSuccs because %U is scheduled so it
should not get counted as an "unscheduled" successor.
Need to update matching between the original reduced values and their
vectorized matches after ordered reduction vectorization to avoid
a compiler crash
Patch models ordered reductions as a series of extractelements for the
cases which cannot be modeled as unordered reductions.
Fixes#50590
Reviewers: RKSimon, hiraditya
Pull Request: https://github.com/llvm/llvm-project/pull/182644
If the instructions state is alternate and/or contains non-directly
matching instructions, need to check if it is better to represent such
operations as non-alternate with copyables.
To do this, we need to compare operands between the instructions in their
different representations and choose the best one for optimal
vectorization.
Reviewers: RKSimon, hiraditya
Pull Request: https://github.com/llvm/llvm-project/pull/183777
When narrowing interleave groups, the main vector loop processes IC
iterations instead of VF * IC. Update selectEpilogueVectorizationFactor
to use the effective VF, checking if the canonical IV controlling the
loop now steps by UF instead of VFxUF.
This avoids epilogue vectorization with dead epilogue vector loops and
also prevents crashes in cases where we can prove both the epilogue and
scalar loop are dead.
Fixes https://github.com/llvm/llvm-project/issues/186846
PR: https://github.com/llvm/llvm-project/pull/187016
Simplify exactly as InstCombine does. A follow-up would include
simplifying add x, (sub 0, y) -> sub x, y.
Alive2 proof: https://alive2.llvm.org/ce/z/Af7QiD
Most loop transformations, like unrolling and vectorization, expect the
latch branch to be countable. Allow rotation, if it turns the latch from
uncountable to countable.
This use SCEV to check for countable exits, if CheckExitCount set.
Currently it is not set for the LPM1 run (where SCEV is not used by
other passes), only in LPM.
With that compile-time impact is mostly neutral
https://llvm-compile-time-tracker.com/compare.php?from=eba342d0ba930a404a026c80aada51c43974f0db&to=2e676337b45fae63ce9498116d8e6e43772363c5&stat=instructions:u
ClamAV is consistently slower (~+0.15%) and 7zip faster in most cases
(~-0.13%)
Across a large test set based on C/C++ workloads, this rotates ~0.8%
more loops with ~2.68M rotated loops.
For the test set, ~2.7% more loops are runtime-unrolled and +6.36% more
early exit loops vectorized on ARM64 macOS.
This fixes a regression where std::ranges::find_last loops stopped
being runtime-unrolled after
5f648c370e
which changed the loop
structure so we stopped rotating.
https://clang.godbolt.org/z/6baeE1av6
Based on https://github.com/llvm/llvm-project/pull/162654.
Co-authored-by: Marek Sedláček <mr.mareksedlacek@gmail.com>
PR: https://github.com/llvm/llvm-project/pull/187483
This updates `matchExtendedReductionOperand` so the simple case of
`UpdateR(PrevValue, ext(...))` is matched first as an early exit. The
binop matching is then flattened to remove the extra layer of the
`MatchExtends` lambda.
I was very puzzled the other day when it showed that VF 8 had a cost of
X and VF 16 had a cost of X/2, yet it still choose VF 8. This PR adds
some extra debug output to explain why this happens.
Part 2/4: Implement HALO for coroutines that flow off final suspend.
Parent PR approved in https://github.com/llvm/llvm-project/pull/185336,
with no change since then
Since `coro.id` is unavailable in resumers, Elide `coro.free` based on
frame instead of `coro.id`
Instead of checking dereferenceability early during
LoopVectorizationLegality, defer the check to VPlan construction via
areAllLoadsDereferenceable.
This in preparation for supporting early exit vectorization of
non-dereferencable loads, e.g. via speculative loads
(https://discourse.llvm.org/t/rfc-provide-intrinsics-for-speculative-loads/89692)
or first-faulting loads. Detection in VPlan allows easily replacing
potentially non-deref loads with other loads as needed.
PR: https://github.com/llvm/llvm-project/pull/185323
This adds a new function-level `flatten` LLVM IR attribute and
implements support for it in the AlwaysInliner pass, bringing LLVM's
behavior in line with GCC.
Previously, the `flatten` attribute only existed as a Clang attribute,
which was lowered to `alwaysinline` on individual call sites. Per the
RFC discussion [1], the consensus was to match GCC semantics:
recursively inline the entire call tree into the
flattened function, rather than just immediate call sites.
This PR:
- Adds the `flatten` function attribute to LLVM IR
- Implements recursive inlining of all viable callees in AlwaysInliner
- Uses inline history tracking to detect and stop at recursive call
cycles
- Emits optimization remarks when inlining is skipped due to recursion
A follow-up patch will update Clang to emit the LLVM `flatten` attribute
on
functions instead of marking individual call sites with `alwaysinline`.
[1]
https://discourse.llvm.org/t/rfc-function-level-flatten-depth-attribute-for-depth-limited-inlining
shl-based reduced values in many cases serve as a bitcast/bswap-based
transfromation root, but need to improve analysis for better matching.
This patch merges reduction candidates into a single reduced value
array, if there are only 2 different candidate arrays, one of them has
only single element, the second is a list of shl instructions. Also,
sorts these shl instructions by their shift amount and merges with the
single candidate, if it is profitable to have a copyable reduction.
This patch implements the unconditional and conditional branch
instructions mirroring the newly added LLVM IR instructions.
So now we have two new classes UncondBrInst and CondBrInst inheriting
from BranchInst.
The original Br opcode has been removed in favor of UncondBr and CondBr.
When two GEPs A & B with variable indices have source element types of
different sizes (e.g. i8 vs i32), their offsets cannot be constants.
Currently the `getConstantOffsetComplexAddrs` does not check this and
simply use the stride of GEP A assuming they are equal, resulting in
miscompile.
Add a check that the source element types are the same, otherwise bail
out.
Fixes#186327
The original support for copyables leads to a regression in x264 in
RISCV, this patch improves detection of the copyable candidates by more
precise checking of the profitability and adds and extra check for
splitnode reduction, if it is profitable.
Fixes#184313
Reviewers: hiraditya, RKSimon
Pull Request: https://github.com/llvm/llvm-project/pull/185697
Summary:
The changes in https://www.github.com/llvm/llvm-project/pull/185552
allowed us to
start building the standard `libclang_rt.profile.a` for GPU targets.
This PR expands this by adding an optimized GPU routine for counter
increment and removing the special-case handling of these functions in
the OpenMP runtime.
Vast majority of these functions are boilerplate, but we should be able
to do more interesting things with this in the future, like value or
memory profiling.
Recursively splitting out some work from #183318; this covers
the enums for early exit loop type (none, readonly, readwrite)
and the style used (just readonly and
masked-handle-ee-in-scalar-tail for now) and refactoring for
basic use of those enums.
This reverts commit 91b928f919364b29e241821fc639b9ef56dab1a5.
This complicates some analysis that need the happen on the scalar VPlan,
before regions have been created, e.g.
https://github.com/llvm/llvm-project/pull/185323/.
Alive2 proof:
https://alive2.llvm.org/ce/z/bK93Cn
I've implemented a fold in `InstCombineAndOrXor.cpp` to canonicalize `~x
| (x - 1)` to `~(x & -x)` which enables the CodeGen to emit the `blsi`
instruction.
I've also added a test in `CodeGen/X86`.
Fixes#184055
---------
Co-authored-by: Tim Gymnich <tim@gymni.ch>
If current buildvector node is part of the combined nodes of the
matching candidate node, this matching candidate must be considered as
non-matching to prevent wrong def-use chain
Reviewers:
Pull Request: https://github.com/llvm/llvm-project/pull/187491
When matching scalar steps of the canonical IV, also match a derived IV
of the canonical IV if the derivation is essentially a no-op. Fixes a
failure in the mve-reg-pressure-spills.ll test when expensive checks are
enabled.
Replace the DenseMap from blocks to their innermost loop a vector
indexed by block numbers, when possible. Supporting number updates is
not trivial as we don't store a list of basic blocks, so this is not
implemented.
NB: I'm generally not happy with the way loops are stored. As I think
that there's room for improvement, I don't want to touch the
representation at this point.
Pull Request: https://github.com/llvm/llvm-project/pull/103400
This patch implements the auxiliary pass argument, a secondary argument
that can be passed to Sandbox IR passes.
It also updates the SeedCollection pass to accept the auxiliary argument
"enable-diff-types", which enables collection of seeds of different
types. This will be used in a follow-up patch.
This patch replace the remaining LogicalAnd to vp.merge in the second
pass to not break the `m_RemoveMask` pattern in the optimizeMaskToEVL.
Also skip cost model comparison when the plan contains `vp_merge` which
won't be calculated by the legacy model.
This can help to remove header mask for FindLast reduction (CSA) loops.
Original PR: https://github.com/llvm/llvm-project/pull/184068
Original built-bot failure:
https://lab.llvm.org/buildbot/#/builders/213/builds/2497
Currently, SLP vectorizer do not care about loops and their trip count.
It may lead to inefficient vectorization in some cases. Patch adds loop
nest-aware tree building and cost estimation.
When it comes to tree building, it now checks that tree do not span
across different loop nests. The nodes from other loop nests are
immediate buildvector nodes.
The cost model adds the knowledge about loop trip count. If it is
unknown, the default value is used, controlled by the
-slp-cost-loop-min-trip-count=<value> option. The cost of the vector
nodes in the loop is multiplied by the number of iteration (trip count),
because each vector node will be executed the trip count number of
times. This allows better cost estimation.
Original Reviewers:
jdenny-ornl, vporpo, hiraditya, RKSimon
Original PR: https://github.com/llvm/llvm-project/pull/150450
Recommit after revert in c7bd3062f1dac975cf9b706f457b3c55b4bf57ff and in 4e500bd0015042b0cd4b7c87b81caeea06072d24
Reviewers:
Pull Request: https://github.com/llvm/llvm-project/pull/187391
masked_cond is used to combine early-exit conditions with masks from
predicate. The early-exit condition should only be evaluated if the mask
is true. Emit the mask first, to avoid incorrect poison propagation.
Fixes https://github.com/llvm/llvm-project/issues/187061.
IVUsers records ephemeral values used only by `llvm.assume` as IV
operands in the Processed set. As a result, `CollectChains` picks them
up and builds unnecessary increment chains. Fix this by checking
`IVUsers::isEphemeral` before collecting the chains.
Fixes#187270
Now that we can vectorize loops with multiple early exits, we emit
dispatch blocks after the middle block to go to a specific exit or
continue in the dispatch chain.
With that, we need to be a bit more careful when it comes to picking the
loop the dispatch block belongs to. The dispatch block will belong to
the innermost loop of all exit blocks reachable from the current block.
Fixes https://github.com/llvm/llvm-project/issues/185362
PR: https://github.com/llvm/llvm-project/pull/185618
