Profile data for instructions (e.g., branch weights) is automatically
preserved via `splice()` which moves the basic blocks along with their
instruction metadata. However, entry count is stored as function
metadata, which was dropped when creating merged function and thunks.
The fix is to explicitly set entry count for both merged function (.Tgm)
and thunks via `setEntryCount()`.
This is an attempt to reland #151660 by including a missing STL header
found by a buildbot failure.
The stable function map could be huge for a large application. Fully
loading it is slow and consumes a significant amount of memory, which is
unnecessary and drastically slows down compilation especially for
non-LTO and distributed-ThinLTO setups. This patch introduces an opt-in
lazy loading support for the stable function map. The detailed changes
are:
- `StableFunctionMap`
- The map now stores entries in an `EntryStorage` struct, which includes
offsets for serialized entries and a `std::once_flag` for thread-safe
lazy loading.
- The underlying map type is changed from `DenseMap` to
`std::unordered_map` for compatibility with `std::once_flag`.
- `contains()`, `size()` and `at()` are implemented to only load
requested entries on demand.
- Lazy Loading Mechanism
- When reading indexed codegen data, if the newly-introduced
`-indexed-codegen-data-lazy-loading` flag is set, the stable function
map is not fully deserialized up front. The binary format for the stable
function map now includes offsets and sizes to support lazy loading.
- The safety of lazy loading is guarded by the once flag per function
hash. This guarantees that even in a multi-threaded environment, the
deserialization for a given function hash will happen exactly once. The
first thread to request it performs the load, and subsequent threads
will wait for it to complete before using the data. For single-threaded
builds, the overhead is negligible (a single check on the once flag).
For multi-threaded scenarios, users can omit the flag to retain the
previous eager-loading behavior.
The stable function map could be huge for a large application. Fully
loading it is slow and consumes a significant amount of memory, which is
unnecessary and drastically slows down compilation especially for
non-LTO and distributed-ThinLTO setups. This patch introduces an opt-in
lazy loading support for the stable function map. The detailed changes
are:
- `StableFunctionMap`
- The map now stores entries in an `EntryStorage` struct, which includes
offsets for serialized entries and a `std::once_flag` for thread-safe
lazy loading.
- The underlying map type is changed from `DenseMap` to
`std::unordered_map` for compatibility with `std::once_flag`.
- `contains()`, `size()` and `at()` are implemented to only load
requested entries on demand.
- Lazy Loading Mechanism
- When reading indexed codegen data, if the newly-introduced
`-indexed-codegen-data-lazy-loading` flag is set, the stable function
map is not fully deserialized up front. The binary format for the stable
function map now includes offsets and sizes to support lazy loading.
- The safety of lazy loading is guarded by the once flag per function
hash. This guarantees that even in a multi-threaded environment, the
deserialization for a given function hash will happen exactly once. The
first thread to request it performs the load, and subsequent threads
will wait for it to complete before using the data. For single-threaded
builds, the overhead is negligible (a single check on the once flag).
For multi-threaded scenarios, users can omit the flag to retain the
previous eager-loading behavior.
## Purpose
This patch is one in a series of code-mods that annotate LLVM’s public
interface for export. This patch annotates the remaining LLVM CodeGen
and CodeGenTypes library interfaces that were missed in, or modified
since, previous patches. The annotations currently have no meaningful
impact on the LLVM build; however, they are a prerequisite to support an
LLVM Windows DLL (shared library) build.
## Background
This effort is tracked in #109483. Additional context is provided in
[this
discourse](https://discourse.llvm.org/t/psa-annotating-llvm-public-interface/85307),
and documentation for `LLVM_ABI` and related annotations is found in the
LLVM repo
[here](https://github.com/llvm/llvm-project/blob/main/llvm/docs/InterfaceExportAnnotations.rst).
## Overview
The bulk of these changes were generated automatically using the
[Interface Definition Scanner (IDS)](https://github.com/compnerd/ids)
tool, followed formatting with `git clang-format`.
The following manual adjustments were also applied after running IDS:
- Explicitly instantiate `CallLowering::setArgFlags` template method
instances in `CodeGen/GlobalISel/CallLowering.h` and annotate them with
`LLVM_ABI`. These methods are already explicitly instantiated in
`lib/CodeGen/GlobalISel/CallLowering.cpp` but were not `extern` declared
in the header.
- Annotate several explicit template instantiations with
`LLVM_EXPORT_TEMPLATE`.
- Include `llvm/CodeGen/Passes.h` from
`llvm/lib/CodeGen/GlobalMergeFunctions.cpp` to pick up the declaration
of `llvm::createGlobalMergeFuncPass` with the `LLVM_ABI` annotation
(instead of adding `LLVM_ABI` to the function definition in this file)
## Validation
Local builds and tests to validate cross-platform compatibility. This
included llvm, clang, and lldb on the following configurations:
- Windows with MSVC
- Windows with Clang
- Linux with GCC
- Linux with Clang
- Darwin with Clang
Names are used for debugging purpose and have no impact on codegen. For
a non-trivial project, reading them consumes a lot of memory and slows
down the compilation significantly. This patch adds a field in the
indexed CGData to remember the total size of Names, and creates a
command-line option to skip reading Names by advancing the pointer when
deserializing the indexed CGData.
createCast in MergeFunctions did not consider ArrayTypes, which results
in the creation of a bitcast between ArrayTypes in the thunk function,
leading to an assertion failure in the provided test case.
The version of createCast in GlobalMergeFunctions does handle
ArrayTypes, so this common code has been factored out into the
IRBuilder.
For global function merging, the target of the arc-attached call must be
a constant and cannot be parameterized.
This change adds a check to bypass this case in `canParameterizeCallOperand()`.
This update follows up on change #112671 and is mostly a NFC, with the following exceptions:
- Introduced `-global-merging-skip-no-params` to bypass merging when no parameters are required.
- Parameter count is now calculated based on the unique hash count.
- Added `-global-merging-inst-overhead` to adjust the instruction overhead, reflecting the machine instruction size.
- Costs and benefits are now computed using the double data type. Since the finalization process occurs offline, this should not significantly impact build time.
- Moved a sorting operation outside of the loop.
This is a patch for
https://discourse.llvm.org/t/rfc-global-function-merging/82608.
This update follows up on change #112671 and is mostly a NFC, with the following exceptions:
- Introduced `-global-merging-skip-no-params` to bypass merging when no parameters are required.
- Parameter count is now calculated based on the unique hash count.
- Added `-global-merging-inst-overhead` to adjust the instruction overhead, reflecting the machine instruction size.
- Costs and benefits are now computed using the double data type. Since the finalization process occurs offline, this should not significantly impact build time.
- Moved a sorting operation outside of the loop.
This is a patch for
https://discourse.llvm.org/t/rfc-global-function-merging/82608.
This is a follow-up PR to refactor the initial global merge function
pass implemented in #112671.
It first collects stable functions relevant to the current module and
iterates over those only, instead of iterating through all stable
functions in the stable function map.
This is a patch for
https://discourse.llvm.org/t/rfc-global-function-merging/82608.
This is a follow-up PR to refactor the initial global merge function
pass implemented in #112671.
It first collects stable functions relevant to the current module and
iterates over those only, instead of iterating through all stable
functions in the stable function map.
This is a patch for
https://discourse.llvm.org/t/rfc-global-function-merging/82608.
This implements a global function merging pass. Unlike traditional
function merging passes that use IR comparators, this pass employs a
structurally stable hash to identify similar functions while ignoring
certain constant operands. These ignored constants are tracked and
encoded into a stable function summary. When merging, instead of
explicitly folding similar functions and their call sites, we form a
merging instance by supplying different parameters via thunks. The
actual size reduction occurs when identically created merging instances
are folded by the linker.
Currently, this pass is wired to a pre-codegen pass, enabled by the
`-enable-global-merge-func` flag.
In a local merging mode, the analysis and merging steps occur
sequentially within a module:
- `analyze`: Collects stable function hashes and tracks locations of
ignored constant operands.
- `finalize`: Identifies merge candidates with matching hashes and
computes the set of parameters that point to different constants.
- `merge`: Uses the stable function map to optimistically create a
merged function.
We can enable a global merging mode similar to the global function
outliner
(https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753/),
which will perform the above steps separately.
- `-codegen-data-generate`: During the first round of code generation,
we analyze local merging instances and publish their summaries.
- Offline using `llvm-cgdata` or at link-time, we can finalize all these
merging summaries that are combined to determine parameters.
- `-codegen-data-use`: During the second round of code generation, we
optimistically create merging instances within each module, and finally,
the linker folds identically created merging instances.
Depends on #112664
This is a patch for
https://discourse.llvm.org/t/rfc-global-function-merging/82608.
