We are bringing a new algorithm for function layout (reordering) based on the
call graph (extracted from a profile data). The algorithm is an improvement of
top of a known heuristic, C^3. It tries to co-locate hot and frequently executed
together functions in the resulting ordering. Unlike C^3, it explores a larger
search space and have an objective closely tied to the performance of
instruction and i-TLB caches. Hence, the name CDS = Cache-Directed Sort.
The algorithm can be used at the linking or post-linking (e.g., BOLT) stage.
The algorithm shares some similarities with C^3 and an approach for basic block
reordering (ext-tsp). It works with chains (ordered lists)
of functions. Initially all chains are isolated functions. On every iteration,
we pick a pair of chains whose merging yields the biggest increase in the
objective, which is a weighted combination of frequency-based and distance-based
locality. That is, we try to co-locate hot functions together (so they can share
the cache lines) and functions frequently executed together. The merging process
stops when there is only one chain left, or when merging does not improve the
objective. In the latter case, the remaining chains are sorted by density in the
decreasing order.
**Complexity**
We regularly apply the algorithm for large data-center binaries containing 10K+
(hot) functions, and the algorithm takes only a few seconds. For some extreme
cases with 100K-1M nodes, the runtime is within minutes.
**Perf-impact**
We extensively tested the implementation extensively on a benchmark of isolated
binaries and prod services. The impact is measurable for "larger" binaries that
are front-end bound: the cpu time improvement (on top of C^3) is in the range
of [0% .. 1%], which is a result of a reduced i-TLB miss rate (by up to 20%) and
i-cache miss rate (up to 5%).
Reviewed By: rahmanl
Differential Revision: https://reviews.llvm.org/D152834
The debug output of replaceDominatedUsesWith() prints incorrect
information, and the user is left confused about what exactly was
replaced. Fix this.
Differential Revision: https://reviews.llvm.org/D156318
Guard FoldBranchToCommonDest in SimplifyCFG with the SpeculateBlocks
flag as it can also speculate instructions.
This was split out of D155997.
Differential Revision: https://reviews.llvm.org/D156194
Refactor to use BasicBlockUtils functions and make life easier for
a subsequent patch for updating the dominator tree.
Reviewed By: dmgreen
Differential Revision: https://reviews.llvm.org/D154053
Add a more "flexible" `SplitBlockAndInsertIfThenElse` function
and re-implement some others on top of it.
Reviewed By: dmgreen
Differential Revision: https://reviews.llvm.org/D154052
Block dispositions of values defined inside the loop may change
during peeling, so clear them. We already do this for other kinds
of unrolling.
Differential Revision: https://reviews.llvm.org/D153762
replaceCongruentIVs analysis is based on ScalarEvolution; this makes
comparing different PHIs and performing the replacement straightforward.
However, it can have some side-effects: it isn't aware whether an
induction variable is in canonical form, so it can perform replacements
which obscure the meaning of the IR.
In test22 in widen-loop-comp.ll, the resulting loop can't be analyzed by
ScalarEvolution at all.
My attempted solution is to restrict the transform: don't try to replace
induction variables using PHI nodes that don't represent simple
induction variables.
I'm not sure if this is the best solution; suggestions welcome.
Differential Revision: https://reviews.llvm.org/D121950
/Users/jiefu/llvm-project/llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp:293:13: error: function 'FactorOutConstant' is not needed and will not be emitted [-Werror,-Wunneeded-internal-declaration]
static bool FactorOutConstant(const SCEV *&S, const SCEV *&Remainder,
^
1 error generated.
Instead of checking the pointer type, check the element type of
the GEP.
Previously we ended up reusing GEP increments that were not in
expanded form, thus not respecting LSRs choice of representation.
The change in 2011-10-06-ReusePhi.ll recovers a regression that
appeared when converting that test to opaque pointers.
Changes in various Thumb tests now compute the step outside the
loop instead of using add.w inside the loop, which is LSR's
preferred representation for this target.
Ideally the normal fadd/fmin/fmax this was creating would fail the verifier.
It's probably also necessary to force off FP exception handlers in the cmpxchg
loop but we don't have a generic way to do that now.
Note strictfp builder is broken in the minnum/maxnum case
https://reviews.llvm.org/D154993
In preparation for removing the `#include "llvm/ADT/StringExtras.h"`
from the header to source file of `llvm/Support/Error.h`, first add in
all the missing includes that were previously included transitively
through this header.
This is fixing all files missed in b0abd4893fa1.
Differential Revision: https://reviews.llvm.org/D154543
We are brining a new algorithm for function layout (reordering) based on the
call graph (extracted from a profile data). The algorithm is an improvement of
top of a known heuristic, C^3. It tries to co-locate hot and frequently executed
together functions in the resulting ordering. Unlike C^3, it explores a larger
search space and have an objective closely tied to the performance of
instruction and i-TLB caches. Hence, the name CDS = Cache-Directed Sort.
The algorithm can be used at the linking or post-linking (e.g., BOLT) stage.
This diff modifies the existing data structures to facilitate the implementation
(down the stack). This is a no-op change.
Reviewed By: hoy
Differential Revision: https://reviews.llvm.org/D152833
When new assumption is created it should be registered in assumption cache
or cache should be invalidated.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D154601
Create a quiet floating-point comparision if function has strictfp attribute.
Avoid unexpected FP exception raised during libcall domain error checking.
It raises an FP exception only in case where an input is a signaling NaN.
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D152776
When normalizing a SCEV expression during expansion, there should be
no need for it to be invertible, as it will only be used for code
generation. This fixes a crash after 7f5b15ad150e.
Fixes https://github.com/llvm/llvm-project/issues/63678.
Loop unrolling tends to produce chains of
`%x1 = add %x0, 1; %x2 = add %x1, 1; ...` with one add per unrolled
iteration. This patch simplifies these adds to `%xN = add %x0, N`
directly during unrolling, rather than waiting for InstCombine to do so.
The motivation for this is that having a single add (rather than
an add chain) on the induction variable makes it a simple recurrence,
which we specially recognize in a number of places. This allows
InstCombine to directly perform folds with that knowledge, instead
of first folding the add chains, and then doing other folds in another
InstCombine iteration.
Due to the reduced number of InstCombine iterations, this also
results in a small compile-time improvement.
Differential Revision: https://reviews.llvm.org/D153540
6f9e743b91ad6ac1f333c introduced a mode which renames only instructions in
the function. This change updates that mode to skip instructions that are already named.
This serves the original purpose of the mode (rename-only-inst) which is:
1. Modify IR without failing verifier with serially ordered number
requirement (%1, %2, %3 required in order).
2. Give meaningful names to instructions.
This reverts commit 20f0c68fd83a0147a8ec1722bd2e848180610288.
https://reviews.llvm.org/D153966#4464594 reports an optimization
regression in Rust.
Additionally this change has caused an unexpected 0.3% compile-time
regression.
This is useful when needing to modify IR and test some optimizations on
them, while keeping BB names and function names intact. If
the IR uses ordered number naming (%1, %2, %3 etc), then we cannot just
remove or reorder specific instructions since the verifier expects the
numbers to be in order.
Explicitly inserting undef is overly defensive. Any values computed
nside the loop that are referenced by dbg.values should naturally
become undef when the loop is deleted, and all other values that
are loop invariant must be preserved.
Reviewed By: scott.linder
Differential Revision: https://reviews.llvm.org/D153539
Currently, to use PSI->isFunctionHotInCallGraph, we first need to
calculate BPI->BFI, which is expensive. Instead, we can implement this
directly with MBFI. Also as @wenlei mentioned in another patch review,
that MachineSizeOpts already has isFunctionColdInCallGraph,
isFunctionHotInCallGraphNthPercentile, etc implemented. These can be
refactored and so they can be reused across MachineFunctionSplitting
and MachineSizeOpts passes.
This CL does this - it refactors out those internal static functions
into PSI as templated functions, so they can be accessed easily.
Differential Revision: https://reviews.llvm.org/D153927
Partial progress towards removing in-tree uses of `Type::getPointerTo`,
before we can deprecate the API.
If the API is used solely to support an unnecessary bitcast, get rid of
the bitcast as well.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D153933
This reverts commit 4d8cf2ae6804e0d3f2b668dbec0f5c1983358328.
There is a library layering violation. LLVMAnalysis cannot depend on LLVMCodeGen.
```
llvm/include/llvm/Analysis/ProfileSummaryInfo.h:19:10: fatal error: 'llvm/CodeGen/MachineFunction.h' file not found
19 | #include "llvm/CodeGen/MachineFunction.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
In D152399, we calculate BPI->BFI in MachineFunctionSplit pass just to
use PSI->isFunctionHotInCallGraph, which is expensive. Instead, we can
implement this directly with MBFI.
Reviewer mentioned in the comment, that machine_size_opts already has
isFunctionColdInCallGraph, isFunctionHotInCallGraphNthPercentile, etc
implemented. These can be refactored and reused across MFS and machine
size opts.
This CL does this - it refactors out those internal static functions
into PSI as templated functions, so they can be accessed easily.
Differential Revision: https://reviews.llvm.org/D153927
Move `AttributeMask` out of `llvm/IR/Attributes.h` to a new file
`llvm/IR/AttributeMask.h`. After doing this we can remove the
`#include <bitset>` and `#include <set>` directives from `Attributes.h`.
Since there are many headers including `Attributes.h`, but not needing
the definition of `AttributeMask`, this causes unnecessary bloating of
the translation units and slows down compilation.
This commit adds in the include directive for `llvm/IR/AttributeMask.h`
to the handful of source files that need to see the definition.
This reduces the total number of preprocessing tokens across the LLVM
source files in lib from (roughly) 1,917,509,187 to 1,902,982,273 - a
reduction of ~0.76%. This should result in a small improvement in
compilation time.
Differential Revision: https://reviews.llvm.org/D153728
Add two new functions `createCond` and `createOrCond` that accept extra
arguments Arg and Arg/Arg2 respectively.
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D153253
In D152399, we calculate BPI->BFI in MachineFunctionSplit pass just to
use PSI->isFunctionHotInCallGraph, which is expensive. Instead, we can
implement this directly with MBFI.
Reviewer mentioned in the comment, that machine_size_opts already has
isFunctionColdInCallGraph, isFunctionHotInCallGraphNthPercentile, etc
implemented. These can be refactored and reused across MFS and machine
size opts.
This CL does this - it refactors out those internal static functions
into PSI as templated functions, so they can be accessed easily.
Differential Revision: https://reviews.llvm.org/D152758
The backstory is that the LCSSA invalidation we perform here is not
really necessary from a SCEV perspective. However, other code may
rely on the fact that invalidating only LCSSA phi nodes is sufficient
for transforms like loop peeling
(see https://reviews.llvm.org/D149331#4398582 for more details).
However, performing invalidation during LCSSA construction also
means that SCEV expansion (which may need to construct LCSSA) can
invalidate SCEV, which is somewhat unexpected and code may not be
prepared to deal with it (see the added test case, reported at
https://reviews.llvm.org/D149435#4428219).
Instead of invalidating SCEV, ensure that the LCSSA phi node also
has cached SCEV if the original instruction did. This means that
later invalidation of LCSSA phi nodes will work as expected. This
should avoid both the above issues and be more efficient.
Differential Revision: https://reviews.llvm.org/D153145
In preparation for removing the `#include "llvm/ADT/StringExtras.h"`
from the header to source file of `llvm/Support/Error.h`, first add in
all the missing includes that were previously included transitively
through this header.
Partial progress towards replacing in-tree uses of
`Type::getPointerTo()`.
If `getPointerTo()` is used solely to support an unnecessary bitcast,
remove the bitcast.
Reviewed By: barannikov88, nikic
Differential Revision: https://reviews.llvm.org/D153307
If a value is already the last element of the worklist, then I think that we don't have to add it again, it is not needed to process it repeatedly.
For some long Triton-generated LLVM IR, this can cause a ~100x speedup.
Differential Revision: https://reviews.llvm.org/D153561
{mini|maxi}mum intrinsics are different from {min|max}num intrinsics in
the propagation of NaN and signed zero. Also, the minnum/maxnum
intrinsics require the presence of nsz flags to be valid reductions in
vectorizer. In this regard, we introduce a new recurrence kind and also
add support for identifying reduction patterns using these intrinsics.
The reduction intrinsics and lowering was introduced here: 26bfbec5d2.
There are tests added which show how this interacts across chains of
min/max patterns.
Differential Revision: https://reviews.llvm.org/D151482
SplitBlockAndInsertIfThen utility creates two new blocks,
they're called ThenBlock and Tail (true and false destinations of a conditional
branch correspondingly). The function has a bool parameter Unreachable,
and if it's set, then ThenBlock is terminated with an unreachable.
At the end of the function the new blocks are added to the loop of the split
block. However, in case ThenBlock is terminated with an unreachable,
it cannot belong to any loop.
Differential Revision: https://reviews.llvm.org/D152434
The ConstantRange specifies the range of the scalar elements in the
vector. When converting into a Constant, we need to create a vector
splat with the correct type. For that purpose, pass in the expected
type for the constant.
Fixes https://github.com/llvm/llvm-project/issues/63380.
