This picks up from #166028, making the `Function` argument optional:
most cases don't need to provide it, but in e.g. InstCombine's case,
where the instruction (select, branch) is not attached to a function
yet, the function needs to be passed explicitly.
Co-authored-by: Florian Hahn <flo@fhahn.com>
Eventually this should be generated by tablegen for all functions.
For now add a manually implementation for sincos_stret, which I
have an immediate use for. This will allow pulling repeated code
across targets into shared call sequence code.
Also add sqrt just to make sure we can handle adding return attributes
on the declaration.
PR #159163's probability computation for epilogue loops does not handle
the possibility of an original loop probability of one. Runtime loop
unrolling does not make sense for such an infinite loop, and a division
by zero results. This patch works around that case.
Issue #165998.
ConstantRange uses `[-1, -1)` as the canonical form of a full set.
Therefore, the `for (APInt I = Lower; I != Upper; ++I)` idiom doesn't
work for full ranges. This patch fixes the value enumeration in
`ConstantComparesGatherer` to prevent missing values for full sets.
Closes https://github.com/llvm/llvm-project/issues/166369.
Also add a corresponding intrinsic property that can be used to mark
intrinsics that do not introduce poison, for example simple arithmetic
intrinsics that propagate poison just like a simple arithmetic
instruction.
As a smoke test this patch adds the new property to
llvm.amdgcn.fmul.legacy.
Previously, sign-extending a 1-bit boolean operand in `#DBG_VALUE` would
convert `true` to -1 (i.e., 0xffffffffffffffff). However, DWARF treats
booleans as unsigned values, so this resulted in the attribute
`DW_AT_const_value(0xffffffffffffffff)` being emitted. As a result, the
debugger would display the value as `255` instead of `true`.
This change modifies the behavior to use zero-extension for 1-bit values
instead, ensuring that `true` is represented as 1. Consequently, the
DWARF attribute emitted is now `DW_AT_const_value(1)`, which allows the
debugger to correctly display the boolean as `true`.
PR #161000 introduced a bug whereby the IR would become invalid by having an unconditional branch have `!prof`attached to it. This only became evident in PR #165744, because the IR of `test/Transforms/SimplifyCFG/pr165301.ll`was simple enough to both (1) introduce the unconditional branch, and (2) survive in that fashion until the end of the pass (simplifycfg) and thus trip the verifier.
`simplifySwitchOfPowersOfTwo` converts (when applicable, see `00f5a1e30b`) a switch to a conditional branch. Its false case goes to the `default` target of the former switch, and the true case goes to a BB performing a `cttz`. We can calculate the branch weights from the branch weights of the old switch.
Issue #147390
As another step in issue #135812, this patch fixes block frequencies for
partial loop unrolling with an epilogue remainder loop. It does not
fully handle the case when the epilogue loop itself is unrolled. That
will be handled in the next patch.
For the guard and latch of each of the unrolled loop and epilogue loop,
this patch sets branch weights derived directly from the original loop
latch branch weights. The total frequency of the original loop body,
summed across all its occurrences in the unrolled loop and epilogue
loop, is the same as in the original loop. This patch also sets
`llvm.loop.estimated_trip_count` for the epilogue loop instead of
relying on the epilogue's latch branch weights to imply it.
This patch fixes branch weights in tests that PR #157754 adversely
affected.
This patch implements the LoopUnroll changes discussed in [[RFC] Fix
Loop Transformations to Preserve Block
Frequencies](https://discourse.llvm.org/t/rfc-fix-loop-transformations-to-preserve-block-frequencies/85785)
and is thus another step in addressing issue #135812.
In summary, for the case of partial loop unrolling without a remainder
loop, this patch changes LoopUnroll to:
- Maintain branch weights consistently with the original loop for the
sake of preserving the total frequency of the original loop body.
- Store the new estimated trip count in the
`llvm.loop.estimated_trip_count` metadata, introduced by PR #148758.
- Correct the new estimated trip count (e.g., 3 instead of 2) when the
original estimated trip count (e.g., 10) divided by the unroll count
(e.g., 4) leaves a remainder (e.g., 2).
There are loop unrolling cases this patch does not fully fix, such as
partial unrolling with a remainder loop and complete unrolling, and
there are two associated tests whose branch weights this patch adversely
affects. They will be addressed in future patches that should land with
this patch.
Followup metadata for remainder loops is handled by two implementations,
both added by 7244852557ca6:
1. `tryToUnrollLoop` in `LoopUnrollPass.cpp`.
2. `CloneLoopBlocks` in `LoopUnrollRuntime.cpp`.
As far as I can tell, 2 is useless: I added `assert(!NewLoopID)` for the
`NewLoopID` returned by the `makeFollowupLoopID` call, and it never
fails throughout check-all for my build.
Moreover, if 2 were useful, it appears it would have a bug caused by
7cd826a321d9. That commit skips adding loop metadata to a new remainder
loop if the remainder loop itself is to be completely unrolled because
it will then no longer be a loop. However, that commit incorrectly
assumes that `UnrollRemainder` dictates complete unrolling of a
remainder loop, and thus it skips adding loop metadata even if the
remainder loop will be only partially unrolled.
To avoid further confusion here, this patch removes 2. check-all
continues to pass for my build. If 2 actually is useful, please advise
so we can create a test that covers that usage.
Near 2, this patch retains the `UnrollRemainder` guard on the
`setLoopAlreadyUnrolled` call, which adds `llvm.loop.unroll.disable` to
the remainder loop. That behavior exists both before and after
7cd826a321d9. The logic appears to be that remainder loop unrolling
(whether complete or partial) is opt-in. That is, unless
`UnrollRemainder` is true, `UnrollRuntimeLoopRemainder` skips running
remainder loop unrolling, and `llvm.loop.unroll.disable` suppresses any
later attempt at it.
This patch also extends testing of remainder loop followup metadata to
be sure remainder loop partial unrolling is handled correctly by 1.
A recent commit 00f5a1e30b modified simplifySwitchOfPowersOfTwo to
generate a branch to handle the non-power-of-2 case when appropriate,
but does not set a DebugLoc on the new branch instruction; this patch
propagates the switch's DebugLoc to the new branch, as for the other
instructions generated in the same block.
Found using #107279.
In `simplifyBranchOnICmpChain`, if we can merge the comparisons into a
range check, use a conditional branch instead. This change also breaks
the cycle found in https://github.com/llvm/llvm-project/issues/165088.
Closes https://github.com/llvm/llvm-project/issues/165088.
Detailed description of the cycle:
```
define void @pr165088_cycle_1(i8 %x) {
entry:
%switch = icmp uge i8 %x, 2
%cond1 = icmp ugt i8 %x, 1
%or.cond = and i1 %switch, %cond1
br i1 %or.cond, label %block3, label %block2
block1:
%cond2 = icmp ugt i8 %x, 1
br i1 %cond2, label %block3, label %block2
block2:
br label %block3
block3:
%cond3 = icmp eq i8 %x, 0
br i1 %cond3, label %exit, label %block1
exit:
ret void
}
```
`simplifyBranchOnICmpChain` folds the branch in `entry` to a switch.
Then we get:
```
entry:
switch i8 %x, label %block3 [
i8 1, label %block2
i8 0, label %block2
]
...
```
`performValueComparisonIntoPredecessorFolding` redirects the default
target `block3` into `block1` because `%x` is never zero.
```
entry:
switch i8 %x, label %block1 [
i8 1, label %block2
i8 0, label %block2
]
...
```
Then `turnSwitchRangeIntoICmp` will convert the switch back into a
branch on icmp.
```
entry:
%switch = icmp ult i8 %x, 2
br i1 %switch, label %block2, label %block1
...
```
Since `block1` and `block2` share the same successor `block3`,
`performBranchToCommonDestFolding` merges the conditions of `entry` and
`block1`, resulting in the original pattern again.
Favour a `cttz`-indexed table lookup over an indirect jump table when
the default switch case is reachable, by branching non-power-of-two
inputs to the default case.
Proofs: https://alive2.llvm.org/ce/z/HeRAtf.
This patch fixes a missed optimization issue: predicate infos might be
lost in phi-use scenarios.
Due to the existence of and-chains, a phi-use might be associated with
multiple LN_Last predicate infos.
E.g.,
```cpp
// TWO LN_Last Predicate Info defs:
// 1. a >= 1
// 2. a < 2
if ( a < 1 || a >= 2) {
a = 1;
}
// PHI use of `a`
use(a)
```
However, previously, `popStackUntilDFSScope` reserved only ONE LN_Last
def for a phi use (i.e., reserve only one of `a >= 1` / `a < 2`),
although there might be multiple LN_Last defs for the same phi use.
This patch reserves the adjacent LN_Last defs if they are designated for
the same phi use.
This patch adds support for constant propagation of individual structure
members through Phi nodes. Each member is handled independently,
allowing optimization opportunities when specific members become
constant.
Also, update the testcase since we are able to optimize the call
parameter now.
[LLVM][CppCompile] Fix unused variable warning after 0e025418
Fixes unused variable warning for libTransform with non debug builds.
Folds the unused variable into the assert.
- Use explicit types instead of auto when type is not obvious.
- Move local function out of anonymous namespace and make them static.
- Use structured bindings in some range for loops.
- Simplify PHI handling code a bit.
This optimization is performed as a separate pass over newly inserted PHI nodes to simplify and deduplicate them. By processing PHIs separately, we avoid the complexity of tracking reference bookkeeping needed to update BBValueInfo structures during insertion.
- Make instruction arguments to the `misexpect` functions const and drop
const for `ArrayRef` arguments.
- Remove namespace llvm/misexpect surrounding all the code in .cpp file
and instead use `using namespace`.
- Change static helper functions to also use const references or
pointers when possible.
GVN currently has two different implementation of equality propagation.
One of them is used for branch conditions (dominating an edge), which
performs replacements across multiple blocks. This is also used for
assumes to handle uses outside the current block.
However, uses inside the block are handled using a completely separate
implementation, which involves populating a replacement map and then
checking it for individual instructions during normal GVN. While this
approach generally makes sense, it is kind of pointless if we already do
a use walk to handle the cross-block case anyway.
This PR generalizes propagateEquality() to accept either a
BasicBlockEdge or an Instruction* and replace dominated users. This
removes the need for special handling of uses in the same block for
assumes, as they're covered by instruction dominance, and ensures that
both implementations do not go out of sync.
This patch sets up `DICompileUnit` to support the DWARFv6
`DW_AT_language_name` and `DW_AT_language_version` attributes (which are
set to replace `DW_AT_language`). This patch changes the
`DICompileUnit::SourceLanguage` field type to a `DISourceLanguageName`
that encapsulates the notion of "versioned vs. unversioned name". A
"versioned" name is one that has an associated version stored separately
in `DISourceLanguageName::Version`.
This patch just changes all the clients of the `getSourceLanguage` API
to the expect a `DISourceLanguageName`. Currently they all just `assert`
(via `DISourceLanguageName::getUnversionedName`) that we're dealing with
"unversioned names" (i.e., the pre-DWARFv6 language codes). In follow-up
patches (e.g., draft is at
https://github.com/llvm/llvm-project/pull/162261), when we start
emitting versioned language codes, the `getUnversionedName` calls can
then be adjusted to `getName`.
**Implementation considerations**
* We could have added a new member to `DICompileUnit` alongside the
existing `SourceLanguage` field. I don't think this would have made the
transition any simpler (clients would still need to be aware of
"versioned" vs. "unversioned" language names). I felt that encapsulating
this inside a `DISourceLanguageName` was easier to reason about for
maintainers.
* Currently DISourceLanguageName is a `12` byte structure. We could
probably pack all the info inside a `uint64_t` (16-bits for the name,
32-bits for the version, 1-bit for answering the `hasVersionedName`).
Just to keep the prototype simple I used a `std::optional`. But since
the guts of the structure are hidden, we can always change the layout to
a more compact representation instead.
**How to review**
* The new `DISourceLanguageName` structure is defined in
`DebugInfoMetadata.h`. All the other changes fall out from changing the
`DICompileUnit::SourceLanguage` from `unsigned` to
`DISourceLanguageName`.
This patch removes canRotateDeoptimizingLatchExit check from loop roate
and connected multi rotation option for loop roate (-loop-rotate-multi
option).
The heuristic in canRotateDeoptimizingLatchExit returns true if any of
the loop exits are non-deoptimizing. This means if the loop has multiple
deopt exits, then without multi-rotate, we may still end up having deopt
exit at the latch. This multi-rotate option is introduced but it is off
by default.
We have not seen any improvements downstream as well, where
we have frequent number of deoptimizations in IR. Since the original
heuristic without the multi-rotate effectively ends up being "rotate
under some conditions", but the loop may still not be in the form we
want, we should remove the heuristic and multi-rotate framework entirely
Note that the diff is big, but it's just removal of while loop and
indentation change.
After this patch I would like to continue here and propose adding a
computability check for exit count, but that will be in a separate PR.
Requests for review: @annamthomas @fhahn @davemgreen
Co-authored-by: Marek Sedlacek <msedlacek@azul.com>
While we do not want to form actual lookup tables early, we do want to
perform some optimizations, as they may enable inlining of the much
simpler form.
Builds on https://github.com/llvm/llvm-project/pull/156477, which
originally included this change as well. This PR makes two changes on
top of it:
* Do not perform the optimization early if it requires adding a mask
check. These make the resulting IR less analyzable.
* Add a new SimplifyCFG option that controls switch-to-arithmetic
conversion separately from switch-to-lookup conversion. Enable the new
flag at the end of the function simplification pipeline. This means that
we attempt the arithmetic conversion before inlining, but avoid it in
the early pipeline, where it may lose information.
The original loop (OL) that serves as input to LoopUnroll has basic
blocks that are arranged as follows:
```
OLPreHeader
OLHeader <-.
... |
OLLatch ---'
OLExit
```
In this depiction, every block has an implicit edge to the next block
below, so any explicit edge indicates a conditional branch.
Given OL and unroll count N, LoopUnroll sometimes creates an unrolled
loop (UL) with a remainder loop (RL) epilogue arranged like this:
```
,-- ULGuard
| ULPreHeader
| ULHeader <-.
| ... |
| ULLatch ---'
| ULExit
`-> RLGuard -----.
RLPreHeader |
,-> RLHeader |
| ... |
`-- RLLatch |
RLExit |
OLExit <-----'
```
Each UL iteration executes N OL iterations, but each RL iteration
executes 1 OL iteration. ULGuard or RLGuard checks whether the first
iteration of UL or RL should execute, respectively. If so, ULLatch or
RLLatch checks whether to execute each subsequent iteration.
Once reached, OL always executes its first iteration but not necessarily
the next N-1 iterations. Thus, ULGuard is always required before the
first UL iteration. However, when control flows from ULGuard directly to
RLGuard, the first OL iteration has yet to execute, so RLGuard is then
redundant before the first RL iteration.
Thus, this patch makes the following changes:
- Adjust ULGuard to branch to RLPreHeader instead of RLGuard, thus
eliminating RLGuard's unnecessary branch instruction for that path.
- Eliminate the creation of RLGuard phi node poison values. Without this
patch, RLGuard has such a phi node for each value that is defined by any
OL iteration and used in OLExit. The poison value is required where
ULGuard is the predecessor. The poison value indicates that control flow
from ULGuard to RLGuard to Exit has no counterpart in OL because the
first OL iteration must execute either in UL or RL.
- Simplify the CFG by not splitting ULExit and RLGuard because, without
the ULGuard predecessor, the single block can now be a dedicated UL
exit.
- To RLPreHeader, add an `llvm.assume` call that asserts the RL trip
count is non-zero. Without this patch, RLPreHeader is reachable only
when RLGuard guarantees that assertion is true. With this patch, RLGuard
guarantees it only when RLGuard is the predecessor, and the OL structure
guarantees it when ULGuard is the predecessor. If RL itself is unrolled
later, this guarantee somehow prevents ScalarEvolution from giving up
when trying to compute a maximum trip count for RL. That maximum trip
count enables the branch instruction in the final unrolled instance of
RLLatch to be eliminated. Without the `llvm.assume` call, some existing
unroll tests start to fail because that instruction is not eliminated.
The original motivation for this patch is to facilitate later patches
that fix LoopUnroll's computation of branch weights so that they
maintain the block frequency of OL's body (see #135812). Specifically,
this patch ensures RLGuard's branch weights do not affect RL's
contribution to the block frequency of OL's body in the case that
ULGuard skips UL.
Handle branch weights for `indirectbr` simplification: if we drop branches that aren't taken, we just need to remove the corresponding branch weight (which is presumably 0).
Issue #147390
Fixes#157113.
Take the following IR as an example; we know the destination of the `[1,
3]` cases is `%else`.
```llvm
define i32 @src(i8 range(i8 0, 6) %arg) {
switch i8 %arg, label %else [
i8 0, label %if
i8 4, label %if
i8 5, label %if
]
if:
ret i32 0
else:
ret i32 1
}
```
We can first try the non-wrapping range for both destinations, but I
don't see how that would be any better.
Proof: https://alive2.llvm.org/ce/z/acdWD4.
The switch becomes a conditional branch, one edge going to what was the default target of the switch, the other to a BB that performs a lookup in a table. The branch weights are accurately determinable from the ones of the switch.
Issue #147390
We cannot calculate the weights of the switch precisely, but we do know the probability of the default branch. We then split equally the remaining probability over the rest of the cases. If we did nothing, the static estimation could be considerably poorer.
Issue #147390
[LoopPeel] Fix branch weights' effect on block frequencies
This patch implements the LoopPeel changes discussed in [[RFC] Fix Loop
Transformations to Preserve Block
Frequencies](https://discourse.llvm.org/t/rfc-fix-loop-transformations-to-preserve-block-frequencies/85785).
In summary, a loop's latch block can have branch weight metadata that
encodes an estimated trip count that is derived from application profile
data. Initially, the loop body's block frequencies agree with the
estimated trip count, as expected. However, sometimes loop
transformations adjust those branch weights in a way that correctly
maintains the estimated trip count but that corrupts the block
frequencies. This patch addresses that problem in LoopPeel, which it
changes to:
- Maintain branch weights consistently with the original loop for the
sake of preserving the total frequency of the original loop body.
- Store the new estimated trip count in the
`llvm.loop.estimated_trip_count` metadata, introduced by PR #148758.
