To aid in debugging, (optionally) dump the dot graph immediately after
the stack update phase (which matches nodes to interior callsites) and
before we cleanup mismatched callee edges (either via tail call fixup,
indirect call fixup, or nulling otherwise).
In the LowerTypeTests pass we used to create IR like this:
%3 = zext i8 ptrtoint (ptr @__typeid_allones7_align to i8) to i64
%4 = lshr i64 %2, %3
%5 = zext i8 sub (i8 64, i8 ptrtoint (ptr @__typeid_allones7_align to i8)) to i64
%6 = shl i64 %2, %5
%7 = or i64 %4, %6
This is because when this code was originally written there were no
funnel shifts and as I recall it was necessary to create an i8 and zext
to pointer width (instead of just having a ptrtoint of pointer width)
in order for the shl/shr/or to be pattern matched to ror. At the time
this caused no problems because there existed a zext ConstantExpr. But
after zext ConstantExpr was removed in #71040, the newly present zext
instruction can prevent pattern matching the rotate, for example if
the zext gets hoisted to a loop preheader or common ancestor of the
check. LowerTypeTests was made to use fshr in #141735 so now we can
ptrtoint to pointer width and stop creating the zext.
Reviewers: fmayer, nikic
Reviewed By: nikic
Pull Request: https://github.com/llvm/llvm-project/pull/142886
This pass figures out whether inlining has exposed a constant address to
a lowered type test, and remove the test if so and the address is known
to pass the test. Unfortunately this pass ends up needing to reverse
engineer what LowerTypeTests did; this is currently inherent to the design
of ThinLTO importing where LowerTypeTests needs to run at the start.
Reviewers: teresajohnson
Reviewed By: teresajohnson
Pull Request: https://github.com/llvm/llvm-project/pull/141327
This option complements -funique-source-file-names and allows the user
to use a different unique identifier than the source file path.
Reviewers: teresajohnson
Reviewed By: teresajohnson
Pull Request: https://github.com/llvm/llvm-project/pull/142901
This patch fixes the handling of a confused `Dependence` object. Such an
object doesn’t contain any information about dependencies, so we must
process it conservatively. However, it was converted into a direction
vector like `[I I ... I]`. As a result, it was treated as if there are
no loop-carried dependencies, which can lead to illegal loop exchanges.
Fixes#140238
The existing way of managing clustered nodes was done through adding
weak edges between the neighbouring cluster nodes, which is a sort of
ordered queue. And this will be later recorded as `NextClusterPred` or
`NextClusterSucc` in `ScheduleDAGMI`.
But actually the instruction may be picked not in the exact order of the
queue. For example, we have a queue of cluster nodes A B C. But during
scheduling, node B might be picked first, then it will be very likely
that we only cluster B and C for Top-Down scheduling (leaving A alone).
Another issue is:
```
if (!ReorderWhileClustering && SUa->NodeNum > SUb->NodeNum)
std::swap(SUa, SUb);
if (!DAG->addEdge(SUb, SDep(SUa, SDep::Cluster)))
```
may break the cluster queue.
For example, we want to cluster nodes (order as in `MemOpRecords`): 1 3
2. 1(SUa) will be pred of 3(SUb) normally. But when it comes to (3, 2),
As 3(SUa) > 2(SUb), we would reorder the two nodes, which makes 2 be
pred of 3. This makes both 1 and 2 become preds of 3, but there is no
edge between 1 and 2. Thus we get a broken cluster chain.
To fix both issues, we introduce an unordered set in the change. This
could help improve clustering in some hard case.
One key reason the change causes so many test check changes is: As the
cluster candidates are not ordered now, the candidates might be picked
in different order from before.
The most affected targets are: AMDGPU, AArch64, RISCV.
For RISCV, it seems to me most are just minor instruction reorder, don't
see obvious regression.
For AArch64, there were some combining of ldr into ldp being affected.
With two cases being regressed and two being improved. This has more
deeper reason that machine scheduler cannot cluster them well both
before and after the change, and the load combine algorithm later is
also not smart enough.
For AMDGPU, some cases have more v_dual instructions used while some are
regressed. It seems less critical. Seems like test `v_vselect_v32bf16`
gets more buffer_load being claused.
This patch implements a simple pass that tries to de-duplicate packs. If
there are two packing patterns inserting the exact same values in the
exact same order, then we will keep the top-most one of them. Even
though such patterns may be optimized away by subsequent passes it is
still useful to do this within the vectorizer because otherwise the cost
estimation may be off, making the vectorizer over conservative.
Reapply PR142507 with fix for test: add in the same x86_64-linux
requirement as other tests as the stack ids are currently computed
differently on big endian systems. This will be investigated separately.
In order to allow selective reporting of context hinting during the LTO
link, and in the future to allow selective more aggressive cloning, add
an option to specify a minimum percent of the max cold size in the
profile summary. Contexts that meet that threshold will get context size
info metadata (and ThinLTO summary information) on the associated
allocations.
Specifying -memprof-report-hinted-sizes during the pre-LTO compile step
will continue to cause all contexts to receive this metadata. But
specifying -memprof-report-hinted-sizes only during the LTO link will
cause only those that meet the new threshold and have the metadata to
get reported.
To support this, because the alloc info summary and associated bitcode
requires the context size information to be in the same order as the
other context information, 0s are inserted for contexts without this
metadata. The bitcode writer uses a more compact format for the context
ids to allow better compression of the 0s.
As part of this change several helper methods are added to query whether
metadata contains context size info on any or all contexts.
This was an oversight. GlobalAliases aren't `Functions`, so `getCalledFunction` would return `nullptr` and the callsite would be deemed as uninstrumentable.
We'd hit an assertion checking proper alignment for an i8 when building
chromium because we used the prefered alignment (which is 4 bytes)
instead of the ABI alignment (which is 1 byte). The ABI alignment should
be used because that's the actual alignment needed to load a constant
from the vtable.
This also updates the two `virtual-const-prop-small-alignment-*` to
explicitly give ABI alignments for i64s.
Currently if there's any memory access that AccessAnalysis couldn't
analyze then all of the runtime pointer check results are discarded.
This patch makes this able to be controlled with the AllowPartial
option, which makes it so we generate the runtime check information
for those pointers that we could analyze, as transformations may still
be able to make use of the partial information.
Of the transformations that use LoopAccessAnalysis, only
LoopVersioningLICM changes behaviour as a result of this change. This is
because the others either:
* Check canVectorizeMemory, which will return false when we have partial
pointer information as analyzeLoop() will return false.
* Examine the dependencies returned by getDepChecker(), which will be
empty as we exit analyzeLoop if we have partial pointer information
before calling areDepsSafe(), which is what fills in the dependency
information.
Before this patch, InstCombine hung because it replaced a value with a
more complex one:
```
%sel = select i1 %cmp, i32 %smax, i32 0 ->
%sel = select i1 %cmp, i32 %masked, i32 0 ->
%sel = select i1 %cmp, i32 %smax, i32 0 ->
...
```
This patch makes this replacement more conservative. It only performs
the replacement iff the new value is one of the operands of the original
value.
Closes https://github.com/llvm/llvm-project/issues/142405.
- Consider inlining recursive function of depth 1 only when
the caller is the function itself instead of inlining it
for each callsite so that we avoid redundant work.
- Use CondContext instead of DomTree for better compilation time.
Consider the following case:
```
define i1 @src(i8 %x) {
%cmp = icmp slt i8 %x, -1
%not1 = xor i1 %cmp, true
%or = or i1 %cmp, %not1
%not2 = xor i1 %or, true
ret i1 %not2
}
```
`sinkNotIntoLogicalOp(%or)` calls `freelyInvert(%cmp,
/*IgnoredUser=*/%or)` first. However, as `%cmp` is also used by `Op1 =
%not1`, the RHS of `%or` is set to `%cmp.not = xor i1 %cmp, true`. Thus
`Op1` is out of date in the second call to `freelyInvert`. Similarly,
the second call may change `Op0`. According to the analysis above, I
decided to avoid this fold when one of the operands is also a user of
the other.
Closes https://github.com/llvm/llvm-project/issues/142518.
ShapeInfo for the store operand may be dropped, e.g. because the operand
got folded by transpose optimizations to another instruction w/o shape
info. This was exposed by the assertion added in
https://github.com/llvm/llvm-project/pull/142416.
This updates VisitStore to use the shape-info directly from the
instruction, which is in line with the other Visit* functions and
ensures that we won't lose shape info.
PR: https://github.com/llvm/llvm-project/pull/142664
Add a `alloc-variant-zeroed` function attribute which can be used to
inform folding allocation+memset. This addresses
https://github.com/rust-lang/rust/issues/104847, where LLVM does not
know how to perform this transformation for non-C languages.
Co-authored-by: Jamie <jamie@osec.io>
Commutative intrinsics go through a separate code path, which did not
check for attribute compatibility, resulting in a later assertion
failure.
Fixes https://github.com/llvm/llvm-project/issues/142462.
In order to allow selective reporting of context hinting during the LTO
link, and in the future to allow selective more aggressive cloning, add
an option to specify a minimum percent of the max cold size in the
profile summary. Contexts that meet that threshold will get context size
info metadata (and ThinLTO summary information) on the associated
allocations.
Specifying -memprof-report-hinted-sizes during the pre-LTO compile step
will continue to cause all contexts to receive this metadata. But
specifying -memprof-report-hinted-sizes only during the LTO link will
cause only those that meet the new threshold and have the metadata to
get reported.
To support this, because the alloc info summary and associated bitcode
requires the context size information to be in the same order as the
other context information, 0s are inserted for contexts without this
metadata. The bitcode writer uses a more compact format for the context
ids to allow better compression of the 0s.
As part of this change several helper methods are added to query whether
metadata contains context size info on any or all contexts.
If the input has known zero bits, InstCombine may have simplied one
of the expected And masks. Teach AggressiveInstCombine to use
MaskedValueIsZero to recover these missing bits.
Fixes#142042.
Just for consistency, to avoid confusing conditions.
`reverse` helps to avoid tests updates as nothing is
changing for for successors count <=2.
For #142461
Over in 6a45fce, this flag (experimental-debuginfo-iterators) was
switched to do nothing, to flush out anything that depended on the
debug-intrinsics way of doing things. It's been a month and nothing's
super-broken, so we'll start to rip things out.
This commit deletes MergeFunc's debuginfo-iterators test: in d2942a86d7
it's documented that that test is specifically because of differences
between intrinsic/non-intrinsic data structures, and we're deleting the
possibility of that difference.
Move VPlan-based calculateRegisterUsage from LoopVectorize
to VPlanAnalysis.cpp. It is a VPlan-based analysis and this helps
to reduce the size of LoopVectorize.
PR: https://github.com/llvm/llvm-project/pull/135673
With:
commit 2425626d803002027cbf71c39df80cb7b56db0fb
Author: Kazu Hirata <kazu@google.com>
Date: Sun Jun 1 08:09:58 2025 -0700
we print out a lot of duplicate alloc site matches.
This patch partially reverts the patch above. The core idea of using
a map to deduplicate entries remains the same, but details are
different. Specifically:
- This PR uses the [FullStackID, MatchLength] as the key, where
MatchLength is the length of an alloc site match.
- AllocMatchInfo in this PR no longer has Matched because we always
report matches.
- AllocMatchInfo in this PR no longer has NumFramesMatched because it
has become part of the key.
This deduplication roughly halves the amount of messages printed out.
5f39be5 ([VPlan] Use InstSimplifyFolder instead of TargetFolder) updated
simplifyRecipe to fold live-ins to Values that are not necessarily
Constant, but forgot to update the corresponding PredPHI folder, which
still folds PredPHI constant -> constant. Update it to fold PredPHI
LiveIn -> LiveIn.
Fixes#141968.
https://github.com/llvm/llvm-project/pull/79993 assumes that a reachable
case must be contained by `CR`. However, it doesn't hold for some edge
cases. This patch adds additional checks to ensure `ReachableCaseCount`
is correct.
Note: Similar optimization in SCCP isn't affected by this bug because it
uses `CR` to compute `ReachableCaseCount`.
Closes https://github.com/llvm/llvm-project/issues/142286.
This change makes the C++ noop coroutine to be built with attributes
based on the module flags. Among other things, this adds function
attributes related to the Pointer Authentication and Branch Target
Enforcement module flags, which are set by command-line options.
Before this patch, C++ programs built with either PAC-RET or BTI that
had the noop coroutine emitted could have a runtime error because this
function wasn't compatible with the rest of the program.
Without this patch, we buffer alloc site matches in
FullStackIdToAllocMatchInfo and then print them out at the end of
MemProfUsePass.
This practice is problematic when we have multiple matches per alloc
site. Consider:
char *f1() { return new char[3]; }
char *f2() { return f1(); }
__attribute__((noinline)) char *f3() { return f2(); }
In this example, f1 contains an alloc site, of course, but so do f2
and f3 via inlining. When something like this happens,
FullStackIdToAllocMatchInfo gets updated multiple times for the same
full stack ID at:
FullStackIdToAllocMatchInfo[FullStackId] = { ... };
with different InlinedCallStack.size() each time.
This patch changes the behavior by immediately printing out alloc site
matches, potentially printing out multiple matches for the same
FullStackId. It is up to the consumer of the message to figure out
the length of the longest matches for example.
For the test, this test adjusts an existing one,
memprof-dump-matched-alloc-site.ll. Specifically, this patch
"restores" the IR and corresponding profile for f2 and f1 so that the
compiler generates a "MemProf notcold" message for each of f1, f2, and
f3.
