Avoid wastefully setting CanVecMem in several places in analyzeLoop,
complicating the logic, to get the function to return a bool, and set
CanVecMem in the caller.
In some modules, e.g. Kotlin-generated IR, we end up with a huge RefSCC
and the call graph updates done as a result of the inliner take a long
time. This is due to RefSCC::removeInternalRefEdges() getting called
many times, each time removing one function from the RefSCC, but each
call to removeInternalRefEdges() is proportional to the size of the
RefSCC.
There are two places that call removeInternalRefEdges(), in
updateCGAndAnalysisManagerForPass() and
LazyCallGraph::removeDeadFunction().
1) Since LazyCallGraph can deal with spurious (edges that exist in the
graph but not in the IR) ref edges, we can simply not call
removeInternalRefEdges() in updateCGAndAnalysisManagerForPass().
2) LazyCallGraph::removeDeadFunction() still ends up taking the brunt of
compile time with the above change for the original reason. So instead
we batch all the dead function removals so we can call
removeInternalRefEdges() just once. This requires some changes to
callers of removeDeadFunction() to not actually erase the function from
the module, but defer it to when we batch delete dead functions at the
end of the CGSCC run, leaving the function body as "unreachable" in the
meantime. We still need to ensure that call edges are accurate. I had
also tried deleting dead functions after visiting a RefSCC, but deleting
them all at once at the end was simpler.
Many test changes are due to not performing unnecessary revisits of an
SCC (the CGSCC infrastructure deems ref edge refinements as unimportant
when it comes to revisiting SCCs, although that seems to not be
consistently true given these changes) because we don't remove some ref
edges. Specifically for devirt-invalidated.ll this seems to expose an
inlining order issue with the inliner. Probably unimportant for this
type of intentionally weird call graph.
Compile time:
https://llvm-compile-time-tracker.com/compare.php?from=6f2c61071c274a1b5e212e6ad4114641ec7c7fc3&to=b08c90d05e290dd065755ea776ceaf1420680224&stat=instructions:u
Fragments are allocated with `operator new` and stored in an ilist with
Prev/Next/Parent pointers. A more efficient representation would be an
array of fragments without the overhead of Prev/Next pointers.
As the first step, replace ilist with singly-linked lists.
* `getPrevNode` uses have been eliminated by previous changes.
* The last use of the `Prev` pointer remains: for each subsection, there is an insertion point and
the current insertion point is stored at `CurInsertionPoint`.
* `HexagonAsmBackend::finishLayout` needs a backward iterator. Save all
fragments within `Frags`. Hexagon programs are usually small, and the
performance does not matter that much.
To eliminate `Prev`, change the subsection representation to
singly-linked lists for subsections and a pointer to the active
singly-linked list. The fragments from all subsections will be chained
together at layout time.
Since fragment lists are disconnected before layout time, we can remove
`MCFragment::SubsectionNumber` (https://reviews.llvm.org/D69411). The
current implementation of `AttemptToFoldSymbolOffsetDifference` requires
future improvement for robustness.
Pull Request: https://github.com/llvm/llvm-project/pull/95077
Prepare for new pass manager version of `MachineDominatorTreeAnalysis`.
We may need a machine dominator tree version of `DomTreeUpdater` to
handle `SplitCriticalEdge` in some CodeGen passes.
"[Flang] Update test to not check for tail calls on debug intrinsics" &
"Reapply#3 "[RemoveDIs] Load into new debug info format by default in LLVM (#89799)"
Recent updates to flang have added debug info generation via MLIR, a path
which currently does not support debug records. The patch that enables
debug records by default (and a small followup patch) are thus being
reverted until the MLIR path has been fixed.
This reverts commits:
21396be865b4640abf6afa0b05de6708a1a996e0
c5aeca732d1ff6769b0659efebd1cfb5f60487e4
This avoids std::map, which is slow, and uses a StringMap. Section name,
group name, linked-to name and unique id are encoded into the key for
fast lookup.
This gives a measurable performance boost for applications that compile
many small object files (e.g., functions in JIT compilers).
---
Now also the second case works properly. That's what happens when you do
that last refactoring without re-running all tests... sorry.
Lower global references to ptrauth constants into `@AUTH` `MCExpr`'s.
The logic is common for MachO and ELF - test both.
---------
Co-authored-by: Ahmed Bougacha <ahmed@bougacha.org>
Relanding this PR now that
https://github.com/llvm/llvm-project/pull/90503 has merged. with `FTAN`
landing in
[TargetLoweringBase.cpp:L1021](https://github.com/llvm/llvm-project/blob/main/llvm/lib/CodeGen/TargetLoweringBase.cpp#L1020C23-L1021C63
) There is now a llvm tan intrinsic 32\64\128 Expand case for all llvm
backends.
In LLVM, the `llvm.experimental.constrained.cos` and
`llvm.experimental.constrained.sin` intrinsics are used for performing
cosine and sine calculations with additional constraints on
floating-point operations. This behavior is expected for all
floating-point math intrinsics. This change adds these constraints for
the `tan` intrinsic.
- `Builtins.td` - replace TanF128 with F16F128MathTemplate
- `CGBuiltin.cpp` - map existing tan builtins to `tan` and
`constrained_tan` intrinsic
- `ConstrainedOps.def` map tan and constrained_tan to an ISDOpcode.
resolves #91421
---------
Co-authored-by: Farzon Lotfi <farzon@farzon.com>
This reverts commit 2cf14398c9341feddb419e7ff9c8c5623a3da3db since it
broke the llvm test suite:
SingleSource/UnitTests/AArch64/acle-fmv-features.c:59:9:
error: instruction requires: altnzcv
SingleSource/UnitTests/AArch64/acle-fmv-features.c:117:10:
error: instruction requires: aes
...
Looks like the FMV dependencies were used in the target attribute and
now features that are FMVOnly (have AEK_NONE) cannot be expanded in
parseTargetAttr using the ExtensionSet.
This suggests that either the tests are wrong (they are using an FMVOnly
feature in a target attribute), or that we need to turn the FMVOnly
features into Extensions (these two are tablegen classes).
There's only one way to create unnamed symbols (createTempSymbol).
Previously, the name was evaluated unconditionally, but often
unnecessarily. Avoid this.
Also the parameter names in the header were wrong, fix these.
Reverts llvm/llvm-project#95006
Seems like there's some bug where the section name is empty in the `if
(!Section.isSingleStringRef())`. Revert for now to get builds back to
green.
This avoid std::map, which is slow, and uses a StringMap. Section name,
group name, linked-to name and unique id are encoded into the key for
fast lookup.
This gives a measurable performance boost (>3%) for applications that
compile many small object files (e.g., functions in JIT compilers).
This patch implements the changes to LLVM IR discussed in
https://discourse.llvm.org/t/rfc-update-branch-weights-metadata-to-allow-tracking-branch-weight-origins/75032
In this patch, we add an optional field to MD_prof metadata nodes for
branch weights, which can be used to distinguish weights added from
`llvm.expect*` intrinsics from those added via other methods, e.g.
from profiles or inserted by the compiler.
One of the major motivations, is for use with MisExpect diagnostics,
which need to know if branch_weight metadata originates from an
llvm.expect intrinsic. Without that information, we end up checking
branch weights multiple times in the case if ThinLTO + SampleProfiling,
leading to some inaccuracy in how we report MisExpect related
diagnostics to users.
Since we change the format of MD_prof metadata in a fundamental way, we
need to update code handling branch weights in a number of places.
We also update the lang ref for branch weights to reflect the change.
This adds IntrWillReturn, IntrNoSync, IntrNoFree, and IntrNoCallback.
The IntrWillReturn is needed to make the intrinsic eligible for being
dead code eliminated if they are unused.
We already use this for most intrinsics. I wonder if the rounding mode
intrinsics were in review when the other intrinsics were changed to
DefaultAttrsIntrinsic.
The patch didn't consistently clean up `#ifdef LLVM_ENABLE_ABI_BREAKING_CHECKS` and '#if defined(LLVM_ENABLE_ABI_BREAKING_CHECKS)' paths, causing a lot of build failures
The categories are primarily meant for OpenMP, where the spec assigns a
category to each directive. It's one of declarative, executable,
informational, meta, subsidiary, and utility.
These will be used in clang to avoid listing directives belonging to
certain categories by hand.
---------
Co-authored-by: Valentin Clement (バレンタイン クレメン) <clementval@gmail.com>
The dependency expansion step which was introduced by FMV has been
erroneously used for non-FMV features, for example when parsing the
target attribute. The PR #93695 has rectified most of the tests which
were relying on dependency expansion of target features specified on the
-cc1 command line. In this patch I am decoupling the dependency
expansion of features specified on the target attribute from FMV.
To do that first I am expanding FMV dependencies before passing the list
of target features to initFeatureMap(). Similarly when parsing the
target attribute I am reconstructing an ExtensionSet from the list of
target features which was created during the command line option
parsing. The attribute parsing may toggle bits of that ExtensionSet and
at the end it is converted to a list of target features. Those are
passed to initFeatureMap(), which no longer requires an override.
A side effect of this refactoring is that features specified on the
target_version attribute now supersede the command line options, which
is what should be happening in the first place.
`LLVM_ENABLE_ABI_BREAKING_CHECKS` is always defined:
72c901f5e5/llvm/include/llvm/Config/abi-breaking.h.cmake (L16C2-L16C15)
It uses `cmakedefine01` rather than `cmakedefine`, so
`LLVM_ENABLE_ABI_BREAKING_CHECKS` is always defined,
so the preprocessed code is probably not what the author wanted.
Reapplies commit 91446e2, which was reverted due to a downstream error,
discussed on the pull request. The error could not be reproduced
upstream, and cannot be reproduced downstream as-of current main, so
until the error can be confirmed to still exist this patch should
return.
This reverts commit 23f8fac745bdde70ed4f9c585d19c4913734f1b8.
Add `LLVMPositionBuilderBeforeDbgRecords` and
`LLVMPositionBuilderBeforeInstrAndDbgRecords` to `llvm/include/llvm-c/Core.h`
which behave the same as `LLVMPositionBuilder` and `LVMPositionBuilderBefore`
except that the position is set before debug records attached to the target
instruction (the existing functions set the insertion point to after any
attached debug records).
More info on debug records and the migration towards using them can be found
here: https://llvm.org/docs/RemoveDIsDebugInfo.html
The distinction is important in some situations. An important example is when
inserting a phi before another instruction which has debug records attached to
it (these come "before" the instruction). Inserting before the instruction but
after the debug records would result in having debug records before a phi, which
is illegal. That results in an assertion failure:
`llvm/lib/IR/Instruction.cpp:166: Assertion '!isa<PHINode>(this) && "Inserting
PHI after debug-records!"' failed.`
In llvm (C++) we've added bit to instruction iterators that carries around the
extra information. Adding dedicated functions seemed like the least invasive and
least suprising way to update the C API.
Update llvm/tools/llvm-c-test/debuginfo.c to test this functionality.
Update the OCaml bindings, the migration docs and release notes.
Lazy relaxation caused hash table lookups (`getFragmentOffset`) and
complex use/compute interdependencies. Some expressions involding
forward declared symbols (e.g. `subsection-if.s`) cannot be computed.
Recursion detection requires complex `IsBeingLaidOut`
(https://reviews.llvm.org/D79570).
D76114's `invalidateFragmentsFrom` makes lazy relaxation even less
useful.
Switch to eager relaxation to greatly simplify code and resolve these
issues. This change also removes a `getPrevNode` use, which makes it
more feasible to replace the fragment representation, which might yield
a large peak RSS win.
Minor downsides: The number of section relaxations may increase (offset
by avoiding the hash table lookup). For relax-recompute-align.s, the
computed layout is not optimal.
BinaryIdsStart and BinaryIdsSize in IndexedInstrProfReader are always
used together, so this patch packages them into an ArrayRef<uint8_t>.
For now, readBinaryIdsInternal immediately unpacks ArrayRef into its
constituents to avoid touching the rest of readBinaryIdsInternal.
MCInst is primarily used in local variables and MCRelaxableFragment
(mostly JMP/JCC for x86). Reducing the inline element count can make
MCRelaxableFragment smaller, potentially leading to a lower peak RSS.
When compiling sqlite3.c, x86-64 has the largest maximum numOperands.
aarch64: 5; ppc64: 6; riscv64: 3; s390x: 6; x86-64: 8
Here is the frequency table for x86-64:
max getNumOperands: 8
0: 676
1: 37892
2: 84046
3: 26767
4: 1640
5: 1222
6: 80794
7: 768
8: 22
Pull Request: https://github.com/llvm/llvm-project/pull/94913
VTableNamePtr and CompressedVTableNamesLen are always used together to
create a StringRef in getSymtab.
We can create the StringRef ahead of time in readHeader. This way,
IndexedInstrProfReader becomes a tiny bit simpler with fewer member
variables. Also, StringRef default-constructs itself with its Data
and Length set to nullptr and 0, respectively, which is exactly what
we need.
This moves the combine of fdiv by constant to fmul out of an
'if (Options.UnsafeFPMath || Flags.hasAllowReciprocal()' block,
so that it triggers if the divide is exact. An extra check for
Recip.isDenormal() is added as multiple places make reference
to it being unsafe or slow on certain platforms.
`BlockT *LoopBase<BlockT, LoopT>::getLoopPreheader()` was changed in
7243607867393a2b8ccd477e95e6f62d00f3206f to use `llvm::size` rather than
the checking that `child_begin() + 1 == child_end()`. `llvm::size`
requires that `std::distance` be O(1) and hence that clients support
random access. Use `llvm::hasSingleElement` instead.
While we are at it, this patch changes the type of ValueCounts to
std:array<double, ...> so that we can use std::array:fill.
Identified with modernize-use-default-member-init.
We call llvm::sort in a couple of places in the V3 encoding:
- We sort Frames by FrameIds for stability of the output.
- We sort call stacks in the dictionary order to maximize the length
of the common prefix between adjacent call stacks.
It turns out that we can improve the deserialization performance by
modifying the comparison functions -- without changing the format at
all. Both places take advantage of the histogram of Frames -- how
many times each Frame occurs in the call stacks.
- Frames: We serialize popular Frames in the descending order of
popularity for improved cache locality. For two equally popular
Frames, we break a tie by serializing one that tends to appear
earlier in call stacks. Here, "earlier" means a smaller index
within llvm::SmallVector<FrameId>.
- Call Stacks: We sort the call stacks to reduce the number of times
we follow pointers to parents during deserialization. Specifically,
instead of comparing two call stacks in the strcmp style -- integer
comparisons of FrameIds, we compare two FrameIds F1 and F2 with
Histogram[F1] < Histogram[F2] at respective indexes. Since we
encode from the end of the sorted list of call stacks, we tend to
encode popular call stacks first.
Since the two places use the same histogram, we compute it once and
share it in the two places.
Sorting the call stacks reduces the number of "jumps" by 74% when we
deserialize all MemProfRecords. The cycle and instruction counts go
down by 10% and 1.5%, respectively.
If we sort the Frames in addition to the call stacks, then the cycle
and instruction counts go down by 14% and 1.6%, respectively, relative
to the same baseline (that is, without this patch).
To facilitate sharing LoopIdiomTransform between AArch64 and RISC-V,
this first patch moves AArch64LoopIdiomTransform from lib/Target/AArch64
to lib/Transforms/Vectorize and renames it to LoopIdiomVectorize. The
following patch (#94082) will teach LoopIdiomVectorize how to generate VP
intrinsics (in addition to the current masked vector style) in favor of
RVV.
