As part of RemoveDIs, we need instruction insertion to be done with
iterators rather than instruction pointers, so that we can communicate
some debug-info facts about the position. This patch is an entirely
mechanical replacement of Instruction * with BasicBlock::iterator, plus
using insertBefore to insert some instructions because we don't have
iterator-taking constructors yet.
Sadly it's not NFC because it causes dbg.value intrinsics / their
DPValue equivalents to shift location.
Because we're storing some extra debug-info information in the iterator
class, we need to insert new LICM-created stores using such iterators.
Switch LICM to storing iterators instead of pointers when it promotes
variables in loops, add a test for the desired behaviour, and enable
RemoveDIs instrumentation on a variety of other LICM tests for good
measure.
(This would appear to be the only pass in LLVM that needs to store
iterators on the heap).
This adds a writable attribute, which in conjunction with
dereferenceable(N) states that a spurious store of N bytes is
introduced on function entry. This implies that this many bytes
are writable without trapping or introducing data races. See
https://llvm.org/docs/Atomics.html#optimization-outside-atomic for
why the second point is important.
This attribute can be added to sret arguments. I believe Rust will
also be able to use it for by-value (moved) arguments. Rust likely
won't be able to use it for &mut arguments (tree borrows does not
appear to allow spurious stores).
In this patch the new attribute is only used by LICM scalar promotion.
However, the actual motivation for this is to fix a correctness issue
in call slot optimization, which needs this attribute to avoid
optimization regressions.
Followup to the discussion on D157499.
Differential Revision: https://reviews.llvm.org/D158081
Since we no longer support typed pointers in LLVM IR, the PtrASXTy
in isLoadInvariantInLoop was set to be equal to Addr->getType() (an
opaque ptr in the same address space). That made the loop looking
through bitcasts redundant.
Continuing the patch series to get rid of debug intrinsics [0], instruction
insertion needs to be done with iterators rather than instruction pointers,
so that we can communicate information in the iterator class. This patch
adds an iterator-taking insertBefore method and converts various call sites
to take iterators. These are all sites where such debug-info needs to be
preserved so that a stage2 clang can be built identically; it's likely that
many more will need to be changed in the future.
At this stage, this is just changing the spelling of a few operations,
which will eventually become signifiant once the debug-info bearing
iterator is used.
[0] https://discourse.llvm.org/t/rfc-instruction-api-changes-needed-to-eliminate-debug-intrinsics-from-ir/68939
Differential Revision: https://reviews.llvm.org/D152537
This matches the check done by the Reassociate pass that we're
trying to reverse.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D158042
Consider the following piece of code:
```
void innermost_loop(int i, double d1, double d2, double delta, int n, double cells[n])
{
int j;
const double d1d = d1 * delta;
const double d2d = d2 * delta;
for (j = 0; j <= i; j++)
cells[j] = d1d * cells[j + 1] + d2d * cells[j];
}
```
When compiling at -Ofast level, after the "Reassociate expressions"
pass, this code is transformed into an equivalent of:
```
int j;
for (j = 0; j <= i; j++)
cells[j] = (d1 * cells[j + 1] + d2 * cells[j]) * delta;
```
Effectively, the computation of those loop invariants isn't done
before the loop anymore, we have one extra multiplication on each
loop iteration instead. Sadly, this results in a significant
performance hit.
Similarly, specifically crafted user code will also experience
inability to hoist those invariants.
This patch is solving this issue by adding the ability to undo such
reassociation into the LICM pass. Note that for doing such
transformation this pass requires the same conditions as the
"Reassociate expressions" pass, namely, the involved binary operators
must have the reassociations allowed (e.g. by specifying the `fast`
attribute) and they must have single use only.
Some parts of this patch were suggested by Nikita Popov.
Reviewed By: huntergr, nikic, paulwalker-arm
Differential Revision: https://reviews.llvm.org/D152281
Building the given test case with 'clang -O2 -g' the call to
'getInOrder' is sunk out of the loop by LICM, but the source
location is not dropped.
Reviewed By: aprantl, fdeazeve
Differential Revision: https://reviews.llvm.org/D152691
The last use was removed by:
commit d623b2f95fd559901f008a0588dddd0949a8db01
Author: Arthur Eubanks <aeubanks@google.com>
Date: Fri Mar 10 17:24:19 2023 -0800
LICM could reassociate mixed variant/invariant comparison/arithmetic operations
and hoist invariant parts out of loop if it can prove that they can be computed
without overflow. Motivating example here:
```
INV1 - VAR1 < INV2
```
can be turned into
```
VAR > INV1 - INV2
```
if we can prove no-signed-overflow here. Then `INV1 - INV2` can be computed
out of loop, so we save one arithmetic operation in-loop.
Reviewed By: skatkov
Differential Revision: https://reviews.llvm.org/D148001
This patch allows LICM to reassociate and hoist following expressions:
```
loop:
%sum = add nsw %iv, %C1
%cmp = icmp <signed pred> %sum, C2
```
where `C1` and `C2` are loop invariants. The reassociated version looks like
```
preheader:
%inv_sum = C2 - C1
...
loop:
%cmp = icmp <signed pred> %iv, %inv_sum
```
In order to prove legality, we need both initial addition and the newly created subtraction
to happen without overflow.
Differential Revision: https://reviews.llvm.org/D149132
Reviewed By: skatkov
This commit removes constness from DILocation::getMergedLocation and
fixes all its users accordingly.
Having constness on the parameters forced the return type to be const
as well, which does force usage of `const_cast` when the location needs
to be used in metadata nodes.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D149942
D37076 makes LICM duplicate instructions into exit blocks if the
instruction is free. For GEPs, the motivation appears to be that
this allows the GEP to be folded into addressing modes, while
non-foldable users outside the loop might prevent this. TBH I don't
think LICM is the place to do this (why doesn't CGP apply this
heuristic itself?) but at least I understand the motivation.
However, the transform is also applied to all other "free"
instructions, which are just that (removed during lowering and not
"folded" in some way). For such instructions, this transform seems
somewhere between useless, counter-productive (undoing CSE/GVN) and
actively incorrect. For example, this transform can duplicate freeze
instructions, which is illegal.
This patch limits the transform to just foldable GEPs, though we
might want to drop it from LICM entirely as a followup.
This is a small compile-time improvement, because querying TTI cost
model for every single instruction is expensive.
Differential Revision: https://reviews.llvm.org/D149136
This was introduced in 030f02021b6359ec5641622cf1aa63d873ecf55a as
an alleged compile-time optimization. In reality, trying to constant
fold instructions is more expensive than just hoisting them. In a
standard pipeline, LICM tends to run either after a run of
LoopInstSimplify or InstCombine, so LICM doesn't really see constant
foldable instructions in the first place, and the attempted fold
is futile.
This makes for a very minor compile-time improvement.
Differential Revision: https://reviews.llvm.org/D149134
As we are moving the instruction without changing its value, it
is sufficient to only invalidate the loop/block dispositions.
This is the same we do in LoopSink.
This exposed another miscompile in GVN, which was fixed by
20e9b31f88149a1d5ef78c0be50051e345098e41.
-----
After D141386, violation of nonnull, range and align metadata
results in poison rather than immediate undefined behavior,
which means that these are now safe to retain when speculating.
We only need to remove UB-implying metadata like noundef.
This is done by adding a dropUBImplyingAttrsAndMetadata() helper,
which lists the metadata which is known safe to retain on speculation.
Differential Revision: https://reviews.llvm.org/D146629
This exposed a miscompile in GVN, which was fixed by D148129.
-----
After D141386, violation of nonnull, range and align metadata
results in poison rather than immediate undefined behavior,
which means that these are now safe to retain when speculating.
We only need to remove UB-implying metadata like noundef.
This is done by adding a dropUBImplyingAttrsAndMetadata() helper,
which lists the metadata which is known safe to retain on speculation.
Differential Revision: https://reviews.llvm.org/D146629
Avoid divergence b/w different kinds of hoisting with reassociation.
Make them all collect general stat NumHoisted and also specific stats
for each particular transform.
In this case the source GEP might not be hoisted even though it
has invariant operands. For now just bail out, but we might need
additional checks for AllowSpeculation in these special-case
reassociation folds.
Reassociate gep (gep ptr, idx1), idx2 to gep (gep ptr, idx2), idx1
if this would make the inner GEP loop invariant and thus hoistable.
This is intended to replace an InstCombine fold that does this (in
04f61fb73d/llvm/lib/Transforms/InstCombine/InstructionCombining.cpp (L2006)).
The problem with the InstCombine fold is that LoopInfo is an optional
dependency, so it is not performed reliably.
Differential Revision: https://reviews.llvm.org/D146813
LICM currently requests optimized use MSSA form. This is wasteful,
because LICM doesn't actually care about most uses, only those of
invariant pointers in loops. Everything else doesn't need to be
optimized.
LICM already uses the clobber walker in most places. This patch
adjusts one place that was using getDefiningAccess() to use it as
well, so we no longer have a dependence on pre-optimized uses.
This change is not NFC in that the fallback on the defining access
when there are too many clobber calls may now fall back to an
unoptimized use. In practice, I've not seen any problems with this
though. If desired, we could also increase licm-mssa-optimization-cap
to a higher value (increasing this from 100 to 200 has no impact on
average compile-time -- but also doesn't appear to have any impact
on LICM quality either).
This makes for a 0.9% geomean compile-time improvement on CTMark.
Differential Revision: https://reviews.llvm.org/D147437
After D141386, violation of nonnull, range and align metadata
results in poison rather than immediate undefined behavior,
which means that these are now safe to retain when speculating.
We only need to remove UB-implying metadata like noundef.
This is done by adding a dropUBImplyingAttrsAndMetadata() helper,
which lists the metadata which is known safe to retain on speculation.
Differential Revision: https://reviews.llvm.org/D146629
Even if there are no thread-safety concerns, we should not promote
(not guaranteed-to-execute) stores to globals without further
analysis: While the global may be writable, we may not have
provenance to perform the write. The @promote_global_noalias test
case illustrates a miscompile in the presence of a noalias pointer
to the global.
Worth noting that the load-only promotion may also not be well-defined
depending on precise semantics (we don't specify whether load
violating noalias is poison or UB -- though I believe the general
inclination is to make it poison, and only stores UB), but that's
a more general issue.
This is inspired by https://github.com/llvm/llvm-project/issues/60860,
which is a related issue with TBAA metadata.
Differential Revision: https://reviews.llvm.org/D146233
Extract a helper that does the clobber walk while taking into
account the cap. Slightly reflow things to check this first in
the store case, before we start walking over all accesses in the
loop.
Despite the fact that it is legal, it is not profitable. It may prevent
Loop Guard Widening to happen. Because of bug described at
https://github.com/llvm/llvm-project/issues/60234, now the guard widening is
only possible when condtion we want to add is available at the point of the
widenable_condition() of dominating guard. It means that, if all such calls are
hoisted out of loop, and the loop conditions depend on loop-variants, we cannot
widen. Hoisting is otherwise not helpful, because it does not introduce any
optimization opportunities.
Differential Revision: https://reviews.llvm.org/D146274
Reviewed By: apilipenko
Only fetch preheader once we want to actually hoist. It turns out
that calculating the preheader is expensive enough to affect
overall compile-time if you do it for every single instruction.
Addresses the compile-time regression from D143726.
We can handle logical AND/OR in the same way as arithmetic AND/OR, it only
takes us freezing `RHS2` for which we may introduce a new use which didn't
exist before dynamically.
Differential Revision: https://reviews.llvm.org/D145771
Reviewed By: nikic
We don't do this transform in InstCombine in general case for arbitrary values, because cost of
AND and 2 ICMP's isn't higher than of MIN and ICMP. However, LICM also has a notion
about the loop structure. This transform becomes profitable if `A` and `B` are loop-invariant and
`X` is not: by doing this, we can compute min outside the loop.
Differential Revision: https://reviews.llvm.org/D143726
Reviewed By: nikic
Invariant.group's are not sufficiently handled by LICM. Specifically,
if a given invariant.group loaded pointer is not overwritten between
the start of a loop, and its use in the load, it can be hoisted.
The invariant.group (on an already invariant pointer operand) ensures
the result is the same. If it is not overwritten between the start
of the loop and the load, it is therefore legal to hoist.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D144053
