Unify the ad-hoc use of whitespace in `LLVM_DEBUG()` messages.
This approach should also make it easier to see which loop debug
messages correspond to and which part of the loop unrolling heuristics
each message corresponds to.
Update LowerMatrixIntrinsics to use tiled loops automatically in for
larger matrixes. The fully unrolled codegen creates a huge amount of
code, which performs noticably worse then the tiled loop nest variant.
We new try to estimate the number of instructions needed for the
multiply, and if it is too large, tiled loops are used. The current
threshold is anything roughly larger than 6x6x6 double multiply.
Eventually I think we want to only generate tiled loops. This patch is a
first step, trying to opt in for cases where we know it is beneficial.
Checked on AArch64, but should help on other architectures similarly,
and also drastically reduce binary size + compile time.
PR: https://github.com/llvm/llvm-project/pull/179325
getTruncateExpr may not always return a SCEVAddRecExpr when truncating
loop bounds. Add a check to verify the result type before casting, and
bail out of the transformation if the cast would be invalid.
This prevents potential crashes from invalid casts when dealing with
complex loop bounds.
Co-authored by Michael Rowan
Resolves [#153090](https://github.com/llvm/llvm-project/issues/153090)
urem x, n: result < n (remainder is always less than divisor)
urem x, n: result <= x (remainder is at most the dividend)
udiv x, n: result <= x (quotient is at most the dividend)
https://alive2.llvm.org/ce/z/ezzsjQ
With a sized dead_on_return, we need to not eliminate stores if there
are to a pointer with a variable offset from the underlying object
marked dead_on_return. This manifested as an assertion failure as
BaseValue/V ended up not being equal. It's possible we could do a range
analysis to try and prove the variable offset stays within bounds, but
this case seems to come up relatively rarely (only reproducible with a
UBSan build of LLVM) and is probably not worth the compile time.
Fixes#180361.
Fixes#171890
If the pointer operands of an instruction are all constants with generic
AS, we always infer the AS of the instruction as uninitialized finally.
And the rewrite process will skip cloning the instruction, producing
invalid IR.
This patch fixes it by inferring the AS of this kind of instruction as
flat. Maybe we can fold the operator with all constants to get better
performance, but I think this case is rare in the real world.
Allow SCEVPtrToAddr as cast in assertion in SCEVDbgValueBuilder.
SCEVPtrToAddr is handled similarly to SCEVPtrToInt.
Fixes a crash with debug info after bd40d1de9c9ee, which started to
generate ptrtoaddr instead of ptrtoint expressions.
In our internal (ByteDance) builds we frequently hit very large
`DeadPhiWeb`s that cause serious compile-time slowdowns, especially in
some auto-generated code where a single file can take 20+ minutes to
compile. There were previous attempts to reduce `DeadPhiWeb` in
`InstCombine` (e.g. llvm/llvm-project#108876 and
llvm/llvm-project#158057), but in our workload we still see a lot of
time spent later in the pipeline (notably `JumpThreading` and
`CorrelatedValuePropagation`).
After digging into our cases, a big chunk of the `DeadPhiWeb` comes from
SROA rewriting `memset`s. We often end up with patterns like:
```
%.sroa.xxx.oldload = load <ty>, ptr %.sroa.xxx
%unused = ptrtoint ptr %.sroa.xxx.oldload to i64 ; or a bitcast-like use
store <ty> <new_value>, ptr %.sroa.xxx
```
Even if `%unused` is cleaned up by later DCE-style passes, the
load/store shape can still make `PromoteMem2Reg` conservatively treat
many blocks as live-in when computing IDF. With cyclic CFGs this can
easily create large, sticky dead phi webs, and the rest of the pipeline
pays for it.
The core issue is that `visitMemSetInst` was using the slice’s original
offsets (`BeginOffset`/`EndOffset`) when deciding whether it needs to
merge with an `.oldload` to preserve bytes not written by the `memset`.
First, there was a typo in the original condition (`EndOffset !=
NewAllocaBeginOffset` instead of `EndOffset != NewAllocaEndOffset`),
which effectively made the check always true and forced the merge path
in most cases. Second, even if the typo is fixed, comparing the original
slice range against the partition bounds is still too strict: cases
where the `memset` contains the partition (e.g. a large `memset` over
the whole alloca while the partition is just a subrange) would still be
misclassified as requiring an `.oldload`. Both issues lead to many
redundant loads and downstream dead phi webs.
This change switches the check to use the already-computed intersection
offsets (`NewBeginOffset`/`NewEndOffset`) against the partition bounds,
so we only generate `.oldload` when the `memset` actually writes only
part of the partition:
```diff
- if (IntTy && (BeginOffset != NewAllocaBeginOffset ||
- EndOffset != NewAllocaBeginOffset)) {
+ if (IntTy && (NewBeginOffset != NewAllocaBeginOffset ||
+ NewEndOffset != NewAllocaEndOffset)) {
; emit oldload + insertInteger merge
}
```
In our workload this cuts down a lot of pointless `.oldload`s and helps
reduce the size of dead phi webs seen after `mem2reg`, improving compile
time without changing semantics (partial overwrites still merge, full
overwrites don’t).
Three asserts/checks in SimpleLoopUnswitchPass are quite expensive on IR
containing many deeply nested loop nests.
Their cost is not linear with the number of loop nests and these asserts
quickly become the most significant cost of the whole compilation.
This patch move some of the asserts under `EXPENSIVE_CHECKS` and leaves the one
line 1080 unguarded.
This problem was exposed with flang because of Fortran
multidimensional arrays and array expressions. In some programs, half of the end to
end compilation time was spent in those asserts.
The AMDGPU promote alloca pass is missing a conversion link when casting
between vectors of pointers and pointers or vectors of pointers with
different number of elements. This causes codegen to crash due to
invalid casts being generated. To address this, this commit adds the
missing conversion link.
In addition to this, the commit moves the common load/store cast logic
into a new function `createLoadStoreCastChain`.
---------
Signed-off-by: Steffen Holst Larsen <HolstLarsen.Steffen@amd.com>
Co-authored-by: Steffen Holst Larsen <HolstLarsen.Steffen@amd.com>
We never need to use this conditionally (and it doesn't really make
sense, as the behavior is substantially different). Force the use of
separate APIs instead of a boolean argument.
The note the comment is making should be obvious based on the structure
of the pass. Additionally, it is grammatically incorrect and has
spelling errors.
a9b0776a81
added an assertion to avoid infinite loops. However, the limit seems
arbitrary, there is no justification for it neither in the code nor in
the commit message, so I think this can be increased.
While doing memset-to-memcpy forwarding, take into account memset
that covers memory regions from a given offset, and the leading
bytes of such a region are undef.
Fixes: https://github.com/llvm/llvm-project/issues/172326.
Replace repeated calls with getAllocationSize() and cached NewAllocaTy.
In AllocaSliceRewriter, the allocated type is already stored in the
NewAllocaTy member variable and now passed directly there. This change
replaces the remaining direct calls to `NewAI.getAllocatedType()` with
the cached NewAllocaTy to simplify and DRY the code.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The allocas in RewriteStatepointsForGC are created with
LiveValue->getType(), so we can use Def->getType() (which equals the
alloca's type) instead of querying getAllocatedType().
Changes:
- Load instructions now use Def->getType() directly
- The ToClobber vector now stores (Type*, AllocaInst*) pairs to track
the original value's type for creating null constants
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This reverts commit fdb05bbf62b8d4fc8dc7ce1f1cfa570f3265a8ae.
This was causing failures in release-mode builds because the
GetPointerBaseWithConstantOffset call would never be run which leads to
ValueOffset being uninitialized and thus the behavior of the function is
unpredictable.
Remove the variable definition and move the function call directly into
the assert statement. Otherwise builds with -Werror that don't use
asserts would fail.
dead_on_return is made optionally sized in #171712. This patch adds
handling in DSE so that we can actually eliminate stores to pointer
parameters marked with a sized dead_on_return attribute. We do not
eliminate stores where the store may overlap with bytes that are not
known to be dead after return.
Reviewers: nikic, antoniofrighetto, alinas, aeubanks
Pull Request: https://github.com/llvm/llvm-project/pull/173694
This patch fixes the issue #115279. After the fusion, some of the cached
SCEV values such as the induction variable may not be valid anymore and
need to be forgotten.
Fixed issue #165087.
When we sink phis from the 2nd loop preheader to the exit block, we
optimize it a bit further, i.e., propagate the uses of each phi node with
its incoming value and optimize away the phis. Deleted `fixPHINodes()`
too because the phis are already optimized away and there is no point
processing `fixPHINodes()`.
This adds the analogous metadata to the nofpclass attribute
to assert values are not a certain set of floating-point classes.
This allows the same information to be expressed if a function
argument is passed indirectly. This matches the bitmask encoding
of nofpclass.
I also think this should be allowed for stores to symmetrically handle
sret, but leave that for later.
Alternatively we could add a more expressive !fprange metadata,
but that would be much more complex. It's useful to match the attribute,
and more annotations can always be added.
Fixes#133560
Returns uint64_t to simplify callers. The goal is eventually replace
getValueType with this query, which should return the known minimum
reference-able size, as provided (instead of a Type) during create.
Additionally the common isSized query would be replaced with an
isExactKnownSize query to test if that size is an exact definition.
Following our
[discussion](https://discourse.llvm.org/t/rfc-plan-to-improve-loopinterchange-by-undoing-simple-reductions/89071),
I ported GCC’s undo_simple_reduction into LLVM.
**Key changes**
- Implement an Reduction2Memory step in LoopInterchange to support the
reduction in the inner loop.
- The feature is behind an option `-loop-interchange-reduction-to-mem`
and is OFF by default. With the feature off, the pass behaves as before
(minimal impact).
- Add a regression test.
**Validation & performance**
- No compile or semantic errors observed on SPEC2006 and SPEC2017 with
the new feature enabled for validation.
- With options: `-da-disable-delinearization-checks` and
`-loop-interchange-reduction-to-mem`
- SPEC2006 410.bwaves on x86 (Intel i9-11900K, Rocket Lake): **+6%**
- SPEC2017 603.bwaves_s on x86 (Intel i9-11900K, Rocket Lake): **+6%**
- SPEC2006 410.bwaves on SpacemiT Key Stone K1: **+29%**
- SPEC2006 410.bwaves on KMH RTL: **+56%**
- SPEC2017 603.bwaves_s on KMH RTL: **+24%**
**Note**
- Reduction2Memory only runs when legality and profitability checks
indicate the interchange will actually be performed. If interchange is
illegal or not profitable, no reduction2mem is applied.
---------
Co-authored-by: ict-ql <168183727+ict-ql@users.noreply.github.com>
Co-authored-by: Lin Wang <wanglulin@ict.ac.cn>
Co-authored-by: Ryotaro Kasuga <kasuga.ryotaro@fujitsu.com>
This patch makes the dead_on_return parameter attribute optionally require a number
of bytes to be passed in to specify the number of bytes known to be dead
upon function return/unwind. This is aimed at enabling annotating the
this pointer in C++ destructors with dead_on_return in clang. We need
this to handle cases like the following:
```
struct X {
int n;
~X() {
this[n].n = 0;
}
};
void f() {
X xs[] = {42, -1};
}
```
Where we only certain that sizeof(X) bytes are dead upon return of ~X.
Otherwise DSE would be able to eliminate the store in ~X which would not
be correct.
This patch only does the wiring within IR. Future patches will make
clang emit correct sizing information and update DSE to only delete
stores to objects marked dead_on_return that are provably in bounds of
the number of bytes specified to be dead_on_return.
Reviewers: nikic, alinas, antoniofrighetto
Pull Request: https://github.com/llvm/llvm-project/pull/171712
In particular, support offset of src, since offset of dest will be a
followup change when dest is allowed to be not full-sized with copy.
Extracted from https://github.com/llvm/llvm-project/pull/150792
Replace patterns that manually compute allocation sizes by multiplying
getTypeAllocSize(getAllocatedType()) by the array size with calls to the
getAllocationSize(DL) API, which handles this correctly and concisely,
returning nullopt for VLAs.
This fixes several places that were not accounting for array allocations
when computing sizes, simplifies code that was doing this manually, and
adds some explicit isFixed checks where implied convert was being used.
This PR is because now that we have opaque pointers, I hate that some
AllocaInst still has type information being consumed by some passes
instead of just using the size, since passes rarely handle that type
information well or correctly. I hope this will grow into a sequence of
commits to slowly eliminate uses of getAllocatedType from AllocaInst.
And similarly later to remove type information from GlobalValue too (it
can be replaced with just dereferenceable bytes, similar to arguments).
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Reduce the size of a DomTreeNodeBase from 80 to 56 bytes by not storing
the children in a SmallVector. Instead, store children as forward-linked
list. This also avoids extra allocations for nodes with many children.
Additionally, DomTreeNodeBase is now trivially destructible.
A lot of code depends on the order of nodes in the dominator tree, so
make sure that the order is the same when inserting nodes. (Not having
to do this would save 8 bytes per node.)
NewGVN uses the order of nodes in the dominator tree in a way that is
not entirely clear to me (https://reviews.llvm.org/D28129). I kept the
semantics as, but now this is the only external user of
addChild/removeChild, which actually should be private.
https://llvm-compile-time-tracker.com/compare.php?from=263802c56b4db3fc9b6ed9fd313499cb03ca44da&to=43e0c0c5b663b3a4067252fc0addbaccefd0014d&stat=instructions:u
Rather than test dominator of every use, just check which of src or dest
is first, and use that insert location. This minimizes unnecessary
dominator queries while also helping to preserve the order of allocas
(for better code readability / diff).
Extracted from PR optimization improvement series at
https://github.com/llvm/llvm-project/pull/150792
Resize the alloca if needed to a common size, as long as the dest was
still fully initialized by the copy.
Extracted from PR optimization improvement series at
https://github.com/llvm/llvm-project/pull/150792 (included all tests
additions from there as well)
In case a ptrmask cannot be converted to the new address space due to an
unknown mask value, this needs to be detcted and an addrspacecast is
needed to not hinder a future use of the unconverted return value of
ptrmask. Otherwise, users of this value will become invalid by receiving
a nullptr as an operand.
This LLVM defect was identified via the AMD Fuzzing project.
(See https://reviews.llvm.org/D80129 for an explanation of why some
ptrmasks are impossible to convert to other addrspaces.)
A patch from May 2013, commit 1e211913b56f390, changed SROA into using a
select instruction to perform vector blend. Idea was that using a select
was the canonical form, and that we optimize select better than
shufflevector.
This patch is changing SROA back into using shufflevector instead of
select when doing the blend (inserting a smaller vector into a larger
vector).
Motivation:
Nowadays InstCombine is canonicalizing this kind of vector blends, using
vector select instructions, into a shufflevector instruction. So it is
assumed that shufflevector is the canonical form now. It is also assumed
that we are better at optimizing shufflevector today, compared to back
in 2013.
Commit f26710d97d9c272be8a55 includes links to a discussion from 2016
(https://discourse.llvm.org/t/ir-canonicalization-vector-select-or-shufflevector/42257/6)
about picking shufflevector as the canonical form.
Corrected various spelling mistakes such as 'occurred', 'receiver',
'initialized', 'length', and others in comments, variable names,
function names, and documentation throughout the project. These
changes improve code readability and maintain consistency in naming
and documentation.
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
In general, GEP offset calculations are allowed to overflow (if no
poison flags are set). Using int64_t for this purpose can result in C
level signed integer overflow, which is UB. It also means that we
incorrectly model whether some offsets are zero, and thus generate
redundant zero-index GEPs.
Change the code to track offsets in APInts of the pointer index size,
like we do in other places (like accumulateConstantOffset etc).
It's okay if the offset calculation overflows and we have to
truncate.
However, this should really be doing all the offset calculations on
correctly-sized APInts. For the case where the overflow occurs
on 64-bit indices, this would trigger signed integer overflow UB.
Fixes issue reported at:
https://github.com/llvm/llvm-project/pull/171456#issuecomment-3741522625
