Applies ThinLTO cloning decisions made during the thin link and
recorded in the summary index to the IR during the ThinLTO backend.
Depends on D141077.
Differential Revision: https://reviews.llvm.org/D149117
This reverts commit 8b8466fd31e5a194fd8ba7a73a0f23d32f164318.
This is causing size regressions with -Oz and FullLTO. Revert while I
come up with a repro.
This reverts commit f09807ca9dda2f588298d8733e89a81105c88120, restoring
bfe7205975a63a605ff3faacd97fe4c1bf4c19b3 and follow on fix
e3e6bc699574550f2ed1de07f4e5bcdddaa65557, now that the nondeterminism
has been addressed by D149924.
Differential Revision: https://reviews.llvm.org/D141077
Introduce processBuildVector as a next step to generalize code for cost
estimation and code emission for gather/buildvector nodes.
Differential Revision: https://reviews.llvm.org/D149973
Multiple cases of instability in the cloning behavior occurred due to
iteration of maps indexed by pointers. Fix by changing the maps to
MapVector. This necessitated adding DenseMapInfo specializations for the
structure types used in the keys.
These were found while trying to commit patch 3 of the cloning
(bfe7205975a63a605ff3faacd97fe4c1bf4c19b3), but the second one turned
out to be in code committed in patch 2, but just exposed by a new test
added with patch 3. Specifically, the iteration in identifyClones().
Added the portion of the new test cases from patch 3 that only relied on
the already committed changes and exposed the issue.
Differential Revision: https://reviews.llvm.org/D149924
To generate cast instructions, the result type is needed. To allow
creating widened casts without underlying instruction, introduce a new
VPWidenCastRecipe that also holds the result type.
This functionality will be used in a follow-up patch to
implement truncateToMinimalBitwidths as VPlan-to-VPlan transform.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D149081
Mitigate https://github.com/llvm/llvm-project/issues/62348
The root cause for the above issue is that we used a textbook dataflow
analysis for the cross suspend point information. The analysis is
powerful but not scaling.
It is not easy to improve the current algorithm and the patch tries to
prune some branches to mitigate the problems.
Before the patch:
```
n: 20000
real 0m11.081s
user 0m10.597s
sys 0m0.320s
n: 40000
real 0m32.927s
user 0m31.403s
sys 0m1.043s
n: 60000
real 1m2.145s
user 0m58.903s
sys 0m2.268s
n: 80000
real 1m47.143s
user 1m41.630s
sys 0m3.857s
n: 100000
real 2m34.758s
user 2m26.587s
sys 0m5.922s
```
After the patch:
```
n: 20000
real 0m10.418s
user 0m9.945s
sys 0m0.311s
n: 40000
real 0m27.884s
user 0m26.430s
sys 0m1.036s
n: 60000
real 0m52.420s
user 0m49.321s
sys 0m2.267s
n: 80000
real 1m25.389s
user 1m20.247s
sys 0m3.856s
n: 100000
real 2m4.275s
user 1m56.405s
sys 0m5.975s
```
This patch intended to be a NFC patch.
There are a few inaccuracies with how FuncSpec handles global
variables.
When specialisation on non-const global variables is disabled (the
default) the pass could nevertheless perform some specializations,
e.g. on a constant GEP expression, or on a SSA variable, for which the
Solver has determined it has the value of a global variable.
When specialisation on non-const global variables is enabled, the pass
would skip non-scalars, e.g. a global array, but this should be
completely inconsequential, a pointer is a pointer.
Reviewed By: SjoerdMeijer
Differential Revision: https://reviews.llvm.org/D149476
Change-Id: Ic73051b2f8602587306760bf2ec552e5860f8d39
We are simplifying the loop and all its children. Each time, we
invalidate the top-most loop. The top-most loop is going to be
the same every time. The cost of SCEV invalidation is largely
independent from how data about the loop is actually cached, so
we should avoid redundant invalidations.
Fix bug constants and sub instructions
When finding constants in a chain starting with the RHS operator of
sub instructions, we were negating the constant before zero extending
it, which is incorrect.
Unfortunately, I was unable to find a simple way to implement this
transformation correctly, so for now I just disabled this optimization
for constants that feed into the RHS of a sub.
Resolves#62379
Transformation from alive2.llvm.org:
define i16 @src(i8 %a, i8 %b, i8 %c) {
entry:
%0 = sub nuw nsw i8 %c, %a
%1 = sub nuw nsw i8 %b, %0
%2 = zext i8 %1 to i16
ret i16 %2
}
Before/Bad:
define i16 @tgt(i8 %a, i8 %b, i8 %c) {
entry:
%0 = zext i8 %a to i16
%1 = zext i8 %b to i16
%c_neg = sub i8 0, %c
%c_zext = zext i8 %c_neg to i16
%2 = sub i16 0, %0
%3 = sub i16 %1, %2
%4 = add i16 %3, %c_zext
ret i16 %4
}
Correct:
define i16 @tgt(i8 %a, i8 %b, i8 %c) {
entry:
%0 = zext i8 %a to i16
%1 = zext i8 %b to i16
%c_zext = zext i8 %c to i16
%c_neg = sub i16 0, %c_zext
%2 = sub i16 0, %0
%3 = sub i16 %1, %2
%4 = add i16 %3, %c_neg
ret i16 %4
}
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D149507
The pass should not try to revectorize instructions with constant
operands, which were not folded by the IRBuilder. It prevents the
non-terminating loop in the SLP vectorizer for non foldable constant
operations.
insts.
If the vectorizable GEP node is built, which should not be scheduled,
and at least one node is a non-gep instruction, need to insert the
vectorized instructions before the last instruction in the list, not
before the first one, otherwise the instructions may be emitted in the
wrong order.
This reverts commit bfe7205975a63a605ff3faacd97fe4c1bf4c19b3, and follow
on fix e3e6bc699574550f2ed1de07f4e5bcdddaa65557, due to some remaining
instability exposed by the bot enabling expensive checks:
https://lab.llvm.org/buildbot/#/builders/42/builds/9842
The asm in a naked function may reasonably expect the argument registers and the
return address register (if present) to be live.
When using -pg and -finstrument-functions, functions are instrumented by adding
a function call to `_mcount/__cyg_profile_func_enter/__cyg_profile_func_enter_bare`/etc,
which will clobber these registers. If the return address register is clobbered,
the function will be unable to return to the caller, possibly causing an
infinite loop.
```
__attribute__((naked)) void g() {
#if defined(__arm__)
__asm__("bx lr");
#else
__asm__("ret");
#endif
}
int main() { g(); }
```
It seems that the only one reasonable way to handle the combination is to
disable instrumenting for naked functions.
GCC PR: https://gcc.gnu.org/PR109707
Close https://github.com/llvm/llvm-project/issues/62504
Reviewed By: hans
Differential Revision: https://reviews.llvm.org/D149721
This reverts commit 6fbf022908c104a380fd1854fb96eafc64509366, restoring
commit bf6ff4fd4b735afffc65f92a4a79f6610e7174c3 with a fix for a bot
failure due to a previously unstable iteration order.
Differential Revision: https://reviews.llvm.org/D141077
This patch adds a new preheader block the VPlan to place SCEV expansions
expansions like the trip count. This preheader block is disconnected
at the moment, as the bypass blocks of the skeleton are not yet modeled
in VPlan.
The preheader block is executed before skeleton creation, so the SCEV
expansion results can be used during skeleton creation. At the moment,
the trip count expression and induction steps are expanded in the new
preheader. The remainder of SCEV expansions will be moved gradually in
the future.
D147965 will update skeleton creation to use the steps expanded in the
pre-header to fix#58811.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D147964
The step is already expanded in the VPlan. Use this expansion instead.
This is a step towards modeling fixing up IV users in VPlan.
It also fixes a crash casued by SCEV-expanding the Step expression in
fixupIVUsers, where the IR is in an incomplete state
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D147963
We use a map of Argument->AllocaInst when mapping Arguments to the
AllocaInst created for them.
Said map is declared from "Value" and called a "DbgPtrCache". This
commit:
* replaces Value to the more specialized Argument class, to reflect the
intent better (i.e. we are _always_ mapping Arguments).
* replaces the name "DbgPtrCache" with the more explicit "ArgToAllocaMap",
as it is not clear reading the code what a "DbgPtr" is.
Differential Revision: https://reviews.llvm.org/D149748
This reverts commit bf6ff4fd4b735afffc65f92a4a79f6610e7174c3.
There is a bot failure where we are getting the correct remarks output
but in a different order. I'll need to investigate to see where we are
having nondeterministic behavior.
Applies cloning decisions to the IR, cloning functions and updating
calls. For Regular LTO, the IR is updated directly during function
assignment, whereas for ThinLTO it is recorded in the summary index
(a subsequent patch will apply to the IR via the index during the
ThinLTO backend.
The function assignment and cloning proceeds greedily, and we create new
clones as needed when we find an incompatible assignment of function
clones to callsite clones (i.e. when different callers need to invoke
different combinations of callsite clones).
Depends on D140949.
Differential Revision: https://reviews.llvm.org/D141077
This includes a couple of changes:
1. Moves the code that changes the root node out of the `TryToReduce` lambda and out of the traversal loop.
2. Since that code moved, there isn't much left in `TryToReduce` so the code was inlined.
3. The phi node variable `P` was also being used as a flag that turns on/off the exploration of operands as new seeds. This patch uses a new variable `TryOperandsAsNewSeeds` for this.
4. Simplifies the code executed when vectorization fails.
The logic of the code should be identical to the original, but I may be missing something not caught by tests.
Differential Revision: https://reviews.llvm.org/D149627
Added basic implementation of ShuffleCostBuilder class in
ShuffleCostEstimator and generalized BaseShuffleAnalysis::createShuffle
function to support emission of Value */InstructionCost for the
vectorization/cost estimation.
Differential Revision: https://reviews.llvm.org/D149171
Argument promotion mostly works on functions with more than one caller (otherwise the function would be inlined or is dead), so there's a good chance that performing this increases code size since we introduce loads at every call site. If any caller is marked minsize, bail.
We could compare the number of loads/stores removed from the function with the number of loads introduced in callers, but that's TODO.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D149768
Now that the runtime tracks the lifetime access density directly, we can
use that directly in the threshold checks instead of less accurately
computing from other statistics.
Differential Revision: https://reviews.llvm.org/D149684
After importing variables, we do some checking to ensure that variables
marked read or write only, which have been marked exported (e.g.
because a referencing function has been exported), are on at least one
module's imports list. This is because the read or write only variables
will be internalized, so we need a copy any any module that references
it.
This checking is overly conservative in the case of linkonce_odr or
other linkage types where there can already be a duplicate copy in
existence in the importing module, which therefore wouldn't need to
import it. Loosen up the checking for these linkage types.
Fixes https://github.com/llvm/llvm-project/issues/62468.
Differential Revision: https://reviews.llvm.org/D149630
There is no need to combine metadata if we're performing store to
load forwarding. In that case we would end up combining metadata
on an unrelated load instruction.
This commit removes potential UB in the PGO instrumentation passes that
was caused by casting away constness and then potentially modifying the
object.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D148903
LCSSA currently invalidates SCEV for all instructions for which
LCSSA phi nodes are created. This used to be necessary, because
SCEV has historically tried to maintain LCSSA form as well. As
such, some SCEV uses of the value would have to become uses of
the phi node instead.
However, nowadays SCEV itself no longer maintains LCSSA form.
The SCEV of on LCSSA phi node will be the same as the SCEV of its
argument. LCSSA is instead maintained in the SCEVExpander.
As such, I believe it is no longer necessary to perform any SCEV
invalidation during LCSSA construction.
After this patch the ScalarEvolution argument to the LCSSA utilities
is no longer necessary -- I'll remove it in a followup NFC patch to
keep this patch more concise.
Differential Revision: https://reviews.llvm.org/D149435
D146349 Introduced the ability to use the information from the
`select` condition to deduce constants as we folded a binop into
select. I.e if the `select` cond was `icmp eq %A, 10`, then in the
true-arm of `select`, we would be able to replace usage of `A` with
`10`.
This is broken for vectors that contain `undef` elements. I.e with
`icmp eq %A, <10, undef>`, subsituting `<10, undef>` for `A` can
result in creating a more undefined result than we otherwise would
have.
We fix the issue with simply checking if the candidate constant for
substituting may contain `undef` elements and don't do it in that
case.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D149592
This patch reverts rGae739aefd7473517d3f08b5c8d08a66c7f469198 to address performance regressions reported by our [CI](https://github.com/dtcxzyw/llvm-ci/issues/137) after rG2ec1d0f427c7822540352c0c14d057e7bfe4f77b.
For example:
```
define ptr @const_gep_chain(ptr %p, i64 %a) {
%p1 = getelementptr inbounds i8, ptr %p, i64 %a
%p2 = getelementptr inbounds i8, ptr %p1, i64 1
%p3 = getelementptr inbounds i8, ptr %p2, i64 2
%p4 = getelementptr inbounds i8, ptr %p3, i64 3
ret ptr %p4
}
```
The last three GEPs will not be folded since rG2ec1d0f427c7822540352c0c14d057e7bfe4f77b.
I think it is appropriate to remove this code because there is no compile-time regression reported in our benchmarks.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D149240
