In the 0e0ff8573de69286536e4f49098226eda0c4c7f5 was introduced
inconsistency between condition widening and checking if it's possible
to widen. We check the possibility to hoist checks parsed from the
condition, but hoist entire condition.
This patch returns testing that a condition can be hoisted rather than
the checks parsed from that condition.
Co-authored-by: Aleksander Popov <apopov@azul.com>
Currently we hoist conditions from widenable branch which are joined to
the widenable_condition by And operation. E.g if we have
br(WC && (c1 && c2)) we will operate with (c1 && c2) unsplitted.
This patch adds more flexibility to that mechanism by supporting work
with the list of checks parsed from the widenable branch.
On that stage patch doesn't change the logic of checks hoisting. In the
example above we will either hoist both checks [c1, c2] or none of them.
But in the future we would improve that logic analyzing each check
separately.
Reviewed By: anna
Differential Revision: https://reviews.llvm.org/D157689
When we replace poison with freeze poison it might appear
that user of poison is a constant (for example vector constant).
In this case we will get that constant will get non-constant operand.
Moreover replacing poison and GlobalValue everywhere in module seems
to be overkill. So the solution will be just make a replacement
only in instructions we visited (contributing to hoisted condition).
Moreover if user of posion is constant, this constant also should need
a freeze and it does not make sense to replace poison with frozen version,
just freeze another constant.
Reviewed By: mkazantsev
Differential Revision: https://reviews.llvm.org/D147429
Non-determenism is fixed.
Guard widening optimization is able to move the condition from one
guard to the previous one. As a result if the condition is poison
and orginal second guard is never executed but the first one does,
we introduce undefined behavior which was not observed in original
program.
To resolve the issue we must freeze the condition we are moving.
However optimization itself does not know how to work with freeze.
Additionally optimization is written in incremental way.
For example we have three guards
G1(base + 8 < L)
G2(base + 16 < L)
G3(base + 24 < L)
On the first step GW will combine G1 and G2 as
G1(base + 8 < L && freeze(base + 16 < L))
G2(true)
G3(base + 24 < L)
while combining G1 and G3 base appears to be different.
To keep optimization enabled after freezing the moving condition, the
freeze instruction is pushed as much as possible and later all uses
of freezed values are replaced with frozen version.
This is similar what instruction combining does but more aggressevely.
This reverts commit f4b2360cecd4c92e85bccb1443f2ef425fc6a77b.
The patch has no specific order in adding freeze instruction in the
entry basic block. It causes failure of CHECK like unit tests.
Guard widening optimization is able to move the condition from one
guard to the previous one. As a result if the condition is poison
and orginal second guard is never executed but the first one does,
we introduce undefined behavior which was not observed in original
program.
To resolve the issue we must freeze the condition we are moving.
However optimization itself does not know how to work with freeze.
Additionally optimization is written in incremental way.
For example we have three guards
G1(base + 8 < L)
G2(base + 16 < L)
G3(base + 24 < L)
On the first step GW will combine G1 and G2 as
G1(base + 8 < L && freeze(base + 16 < L))
G2(true)
G3(base + 24 < L)
while combining G1 and G3 base appears to be different.
To keep optimization enabled after freezing the moving condition, the
freeze instruction is pushed as much as possible and later all uses
of freezed values are replaced with frozen version.
This is similar what instruction combining does but more aggressevely.
Reviewed By: mkazantsev
Differential Revision: https://reviews.llvm.org/D146699
Guard Widening is ignorant about blocks frequency. As result, it may
end up widening conditions from cold/effectively dead code into some
much hotter place, harming average performance.
The initial version was reverted because it looped infinitely if the likely successor
isn't properly dominated by the predecessor. In practice it means that we went up the
CFG through backedge and looped infinitely.
I also added some paranoid assertion checks to make sure that every other invariant
holds. I also found a hypothetical situation when we may go past the dominated block
while following the likely successors (it means that in fact the dominated block is
dynamically not reachable from dominating block) and explicitly prohibited this, though
I don't have a motivating test showing that it's a real problem.
https://reviews.llvm.org/D146276
This reverts commit 8d2885c2ef98b81927c1f816691ec4e77cfc7f3f.
I accidentally introduced an infinite loop in this patch, will return when this is fixed.
There is a piece of logic in GuardWidening which is very limited, and it happens
to ignore implicit control flow, therefore it "works fine" with guards expressed as
intrinsic calls. However, when the guards are represented as branches, its limitations
create a lot of trouble.
The intent here is to make sure that we do not widen code across complex CFG,
so that it can end up being in hotter code than it used to be. The old logic was limited
to unique immediate successor and that's it.
This patch changes the logic there to work the following way: when we need to check
if we can widen from `DominatedBlock` into `DominatingBlock`, we first try to find the
lowest (by CFG) transitive successor of `DominatingBlock` which is guaranteed to not
be significantly colder than the `DominatingBlock`. It means that every time we move
to either:
- Unique successor of the current block, if it only has one successor;
- The only taken successor, if the current block ends with `br(const)`;
- If one of successors ends with deopt, another one is taken;
If the lowest block we can find this way is the `DominatedBlock`, then it is safe to assume
that this widening won't move the code into a hotter location.
I did not touch the existing check with PDT. It looks fishy to me (post-dominance doesn't
really guarantee anything about hotness), but let's keep it as is and maybe fix later.
With this patch, Guard Widening can widen explicitly expressed branches across more than one
dominating guard if it's profitable.
Differential Revision: https://reviews.llvm.org/D146276
Reviewed By: skatkov
Despite the fact that it is legal, it is not profitable. It may prevent
Loop Guard Widening to happen. Because of bug described at
https://github.com/llvm/llvm-project/issues/60234, now the guard widening is
only possible when condtion we want to add is available at the point of the
widenable_condition() of dominating guard. It means that, if all such calls are
hoisted out of loop, and the loop conditions depend on loop-variants, we cannot
widen. Hoisting is otherwise not helpful, because it does not introduce any
optimization opportunities.
Differential Revision: https://reviews.llvm.org/D146274
Reviewed By: apilipenko
Guards can be represented differently (as intrinsic calls and as explicit branches),
and the expectation is that Guard Widening should do the same things for them, no
matter what form it was suggested. Currently it seems to be not so.
We also don't want LICM to prevent guard widening.
This fixes a possible issue when we could hoist an instruction up to a widenable
condition intrinsic call without making sure its operands are available at
hoisiting point.
Recently insertion point finding algorithm changed a bit, so this availability
check became necessary.
Verifier would crash after we handled the following special case:
L >u C0 && L >u C1 -> L >u max(C0, C1),
Previously we would insert the new condition right before the widenable condition
branch where all L operands were available.
Now we may choose the widenable condition intrinsic call as insertion point and it may
happen so that the L operands are computed after the call, so we have to make sure that
L operands are available at the point we want to insert it.
Differential Revision: https://reviews.llvm.org/D144944
There was a crash because there was inconsistency between 'isAvailableAt'
and 'makeAvailableAt' queries. 'makeAvailableAt' is called on conditions
of both guards (dominating and dominated) and 'isAvailableAt' is called
only for dominated guard's condition. Before this patch, it didn't matter
because insertion point always matched the dominating guard. Now, because
they are different, this inconsistency leads to incorrect transforms which
are caught by assert.
The fix is to check 'isAvailableAt' for both conditions.
Differential Revision: https://reviews.llvm.org/D142693
When guards are represented as widenable branches, there is a tricky
situation when the branch stays in loop but widenable condition doesn't.
It means that the widenable condition is loop-invariant, and some other
optimizations could have done changes using this fact.
If widening is allowed to create widened condition inside this loop,
and join the loop-invariant wc with some non-invariant facts, it can
cause miscompile. See example of this at https://github.com/llvm/llvm-project/issues/60234.
The solution is to adjust the point of generationg the wide condition,
and therefore of hoisting all related parts there. It should not be before
the branch, but before the widenable_condition call. The fact that `wc()`
and the wide condition are in the same block guarantees that they will
not violate the invariance property for any loop.
Differential Revision: https://reviews.llvm.org/D142693
Reviewed By: apilipenko
The first test shows that combineRangeChecks may choose to keep only two poison conditions.
And we cannot do simple arithmetic or logical and in guard.
The second test shows that keeping two poison conditions in the widened guard may allow
execution of side-effect instruction even if just freeze these conditions.
The third test shows that even in simple test we can hoist a poison and even logical and does not help here.
Reviewed By: mkazantsev
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D128779
When we hoist instructions over guard we must clear flags due to these flags
might be implied using this guard, so they make sense only after the guard.
As an example of the bug due to current behavior.
L is known to be in range say [0, 100)
c1 = x u< L
guard (c1)
x1 = add x, 1
c2 = x1 u< L
guard(c2)
basing on guard(c1) we can say that x1 = add nuw nsw x, 1
after guard widening we get
c1 = x u< L
x1 = add nuw nsw x, 1
c2 = x1 u< L
c = and c1, c2
guard(c)
now, basing on fact that x + 1 < L and x >= 0 due to x + 1 is nuw
we can prove that x + 1 u< L implies that x u< L, so we can just remove c1
x1 = add nuw nsw x, 1
c2 = x1 u< L
guard(c2)
But that is not correct due to we will pass x == -1 value.
Reviewed By: mkazantsev
Subscribers: llvm-commits, nikic
Differential Revision: https://reviews.llvm.org/D126354
As reported on https://bugs.llvm.org/show_bug.cgi?id=51020, the
guard widening pass doesn't preserve MemorySSA, so it can no
longer be scheduled in the same loop pass manager as LICM. However,
the loop-schedule.ll test indicates that this is supposed to work.
Fix this by preserving MemorySSA if available, as this seems to be
trivial in this case (we only need to drop the memory access for
the removed guards).
Differential Revision: https://reviews.llvm.org/D108386
As a reminder, a "widenable branch" is the pattern "br i1 (and i1 X, WC()), label %taken, label %untaken" where "WC" is the widenable condition intrinsics. The semantics of such a branch (derived from the semantics of WC) is that a new condition can be added into the condition arbitrarily without violating legality.
Broaden the definition in two ways:
Allow swapped operands to the br (and X, WC()) form
Allow widenable branch w/trivial condition (i.e. true) which takes form of br i1 WC()
The former is just general robustness (e.g. for X = non-instruction this is what instcombine produces). The later is specifically important as partial unswitching of a widenable range check produces exactly this form above the loop.
Differential Revision: https://reviews.llvm.org/D70502
This code has never been enabled. While it is tested, it's complicating some refactoring. If we decide to re-implement this, doing it in SimplifyCFG would probably make more sense anyways.
We had a subtle, but nasty bug in our definition of a widenable branch, and thus in the transforms which used that utility. Specifically, we returned true for any branch which included a widenable condition within it's condition, regardless of whether that widenable condition also had other uses.
The problem is that the result of the WC() call is defined to be one particular value. As such, all users must agree as to what that value is. If we widen a branch without also updating *all other users* of the WC in the same way, we have broken the required semantics.
Most of the textual diff is updating existing transforms not to leave dead uses hanging around. They're largely NFC as the dead instructions would be immediately deleted by other passes. The reason to make these changes is so that the transforms preserve the widenable branch form.
In practice, we don't get bitten by this only because it isn't profitable to CSE WC() calls and the lowering pass from guards uses distinct WC calls per branch.
Differential Revision: https://reviews.llvm.org/D69916
As it's causing some bot failures (and per request from kbarton).
This reverts commit r358543/ab70da07286e618016e78247e4a24fcb84077fda.
llvm-svn: 358546
This patch adds support of guards expressed in explicit form via
`widenable_condition` in Guard Widening pass.
Differential Revision: https://reviews.llvm.org/D56075
Reviewed By: reames
llvm-svn: 353932
Guard widening should not spend efforts on dealing with guards with trivial true/false conditions.
Such guards can easily be eliminated by any further cleanup pass like instcombine. However we
should not unconditionally delete them because it may be profitable to widen other conditions
into such guards.
Differential Revision: https://reviews.llvm.org/D50247
Reviewed By: fedor.sergeev
llvm-svn: 340381
This patch teaches LICM to hoist guards from the loop if they are guaranteed to execute and
if there are no side effects that could prevent that.
Differential Revision: https://reviews.llvm.org/D50501
Reviewed By: reames
llvm-svn: 340256
This is a second part of D49974 that handles widening of conditional branches that
have very likely `false` branch.
Differential Revision: https://reviews.llvm.org/D50040
Reviewed By: reames
llvm-svn: 339537
If there is a frequently taken branch dominated by a guard, and its condition is available
at the point of the guard, we can widen guard with condition of this branch and convert
the branch into unconditional:
guard(cond1)
if (cond2) {
// taken in 99.9% cases
// do something
} else {
// do something else
}
Converts to
guard(cond1 && cond2)
// do something
Differential Revision: https://reviews.llvm.org/D49974
Reviewed By: reames
llvm-svn: 338988
The effect of doing so is not disrupting the LoopPassManager when mixing this pass with other loop passes. This should help locality of access substaintially and avoids the cost of computing PostDom.
The assumption here is that the full GuardWidening (which does use PostDom) is run as a canonicalization before loop opts and that this version is just catching cases exposed by other loop passes. (i.e. LoopPredication, IndVarSimplify, LoopUnswitch, etc..)
llvm-svn: 331094
The idea is to have a pass which performs the same transformation as GuardWidening, but can be run within a loop pass manager without disrupting the pass manager structure. As demonstrated by the test case, this doesn't quite get there because of issues with post dom, but it gives a good step in the right direction. the motivation is purely to reduce compile time since we can now preserve locality during the loop walk.
This patch only includes a legacy pass. A follow up will add a new style pass as well.
llvm-svn: 331060
I had used `std::remove_if` under the assumption that it moves the
predicate matching elements to the end, but actaully the elements
remaining towards the end (after the iterator returned by
`std::remove_if`) are indeterminate. Fix the bug (and make the code
more straightforward) by using a temporary SmallVector, and add a test
case demonstrating the issue.
llvm-svn: 270306
Sequences of range checks expressed using guards, like
guard((I - 2) u< L)
guard((I - 1) u< L)
guard((I + 0) u< L)
guard((I + 1) u< L)
guard((I + 2) u< L)
can sometimes be combined into a smaller sequence:
guard((I - 2) u< L AND (I + 2) u< L)
if we can prove that (I - 2) u< L AND (I + 2) u< L implies all of checks
expressed in the previous sequence.
This change teaches GuardWidening to do this kind of merging when
feasible.
llvm-svn: 270151
Summary:
Implement guard widening in LLVM. Description from GuardWidening.cpp:
The semantics of the `@llvm.experimental.guard` intrinsic lets LLVM
transform it so that it fails more often that it did before the
transform. This optimization is called "widening" and can be used hoist
and common runtime checks in situations like these:
```
%cmp0 = 7 u< Length
call @llvm.experimental.guard(i1 %cmp0) [ "deopt"(...) ]
call @unknown_side_effects()
%cmp1 = 9 u< Length
call @llvm.experimental.guard(i1 %cmp1) [ "deopt"(...) ]
...
```
to
```
%cmp0 = 9 u< Length
call @llvm.experimental.guard(i1 %cmp0) [ "deopt"(...) ]
call @unknown_side_effects()
...
```
If `%cmp0` is false, `@llvm.experimental.guard` will "deoptimize" back
to a generic implementation of the same function, which will have the
correct semantics from that point onward. It is always _legal_ to
deoptimize (so replacing `%cmp0` with false is "correct"), though it may
not always be profitable to do so.
NB! This pass is a work in progress. It hasn't been tuned to be
"production ready" yet. It is known to have quadriatic running time and
will not scale to large numbers of guards
Reviewers: reames, atrick, bogner, apilipenko, nlewycky
Subscribers: mcrosier, llvm-commits
Differential Revision: http://reviews.llvm.org/D20143
llvm-svn: 269997
