The commands to run the compilation when printed with `-###` contain
various irrelevant lines for the perf-training. Most of them are
filtered out already but when configured with
`CLANG_CONFIG_FILE_SYSTEM_DIR` a new line like the following is
added and needs to be filtered out:
`Configuration file: /etc/clang/x86_64-redhat-linux-gnu-clang.cfg`
Simpler detection of dynamic library operands as the readelf one seems
to be unreliable (works on my setup, not on buildbots).
This is a follow-up to #127020
This change removes the need to call the clang-bolt target in order to
apply bolt optimizations to clang. Now running `ninja clang` will build
a clang with bolt optimizations, and `ninja check-clang` and `ninja
install-clang` will test and install bolt optimized clang too.
The clang-bolt target has been kept for compatibilty reasons, but it is
now just an alias to the clang target.
Also, this new design for applying the bolt optimizations to clang will
be easier to generalize and use to optimize other binaries/libraries in
the project.
---------
Co-authored-by: Amir Ayupov <fads93@gmail.com>
Co-authored-by: Petr Hosek <phosek@google.com>
The first path argument was always being ignored, and since most calls
to this command only passed one path, it wasn't actually doing anything
in most cases.
This bug was introduced by dd0356d741aefa25ece973d6cc4b55dcb73b84b4.
The new CLANG_PGO_TRAINING_DATA_SOURCE_DIR allows users to specify a
CMake project to use for generating the profile data. For example, to
use the llvm-test-suite to generate profile data you would do:
$ cmake -G Ninja -B build -S llvm -C <path to
source>/clang/cmake/caches/PGO.cmake \
-DBOOTSTRAP_CLANG_PGO_TRAINING_DATA_SOURCE_DIR=<path to llvm-test-suite>
\
-DBOOTSTRAP_CLANG_PGO_TRAINING_DEPS=runtimes
Note that the CLANG_PERF_TRAINING_DEPS has been renamed to
CLANG_PGO_TRAINING_DEPS.
---------
Co-authored-by: Petr Hosek <phosek@google.com>
This reverts commit 745883bba69007f1d2c5135f3d5b0f1efcfc82cd.
This is failing to configure on many of our bots:
https://lab.llvm.org/buildbot/#/builders/245/builds/19468
This did not get caught right away because generally bots only
clean the build every so often.
Split up and refactor CLANG_BOLT_INSTRUMENT into support for
BOLT instrumentation, perf no-LBR and perf with LBR profiling.
Differential Revision: https://reviews.llvm.org/D143617
This is an ongoing series of commits that are reformatting our
Python code.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Reviewed By: MatzeB
Differential Revision: https://reviews.llvm.org/D150761
`findFilesWithExtension` helper checks for `endswith(extension)` instead of
exactly matching the file extension. This causes it to match unrelated files,
for example, `.profdata` files while matching `.fdata` files:
http://157.230.108.44:8011/#/builders/56/builds/247
```
Merging data from /worker/worker/bolt-x86_64-ubuntu-clang-bolt-gcc/build/tools/clang/prof.fdata.1124569.fdata...
Merging data from /worker/worker/bolt-x86_64-ubuntu-clang-bolt-gcc/build/tools/clang/test/Frontend/Output/optimization-remark-with-hotness-new-pm.c.tmp.profdata...
```
Reviewed By: phosek
Differential Revision: https://reviews.llvm.org/D141342
`findFilesWithExtension` helper checks for `endswith(extension)` instead of
exactly matching the file extension. This causes it to match unrelated files,
for example, `.profdata` files while matching `.fdata` files:
http://157.230.108.44:8011/#/builders/56/builds/247
```
Merging data from /worker/worker/bolt-x86_64-ubuntu-clang-bolt-gcc/build/tools/clang/prof.fdata.1124569.fdata...
Merging data from /worker/worker/bolt-x86_64-ubuntu-clang-bolt-gcc/build/tools/clang/test/Frontend/Output/optimization-remark-with-hotness-new-pm.c.tmp.profdata...
```
Reviewed By: phosek
Differential Revision: https://reviews.llvm.org/D141342
This patch adds `CLANG_BOLT_INSTRUMENT` option that applies BOLT instrumentation
to Clang, performs a bootstrap build with the resulting Clang, merges resulting
fdata files into a single profile file, and uses it to perform BOLT optimization
on the original Clang binary.
The projects and targets used for bootstrap/profile collection are configurable via
`CLANG_BOLT_INSTRUMENT_PROJECTS` and `CLANG_BOLT_INSTRUMENT_TARGETS`.
The defaults are "llvm" and "count" respectively, which results in a profile with
~5.3B dynamically executed instructions.
The intended use of the functionality is through BOLT CMake cache file, similar
to PGO 2-stage build:
```
cmake <llvm-project>/llvm -C <llvm-project>/clang/cmake/caches/BOLT.cmake
ninja clang++-bolt # pulls clang-bolt
```
Stats with a recent checkout (clang-16), pre-built BOLT and Clang, 72vCPU/224G
| CMake configure with host Clang + BOLT.cmake | 1m6.592s
| Instrumenting Clang with BOLT | 2m50.508s
| CMake configure `llvm` with instrumented Clang | 5m46.364s (~5x slowdown)
| CMake build `not` with instrumented Clang |0m6.456s
| Merging fdata files | 0m9.439s
| Optimizing Clang with BOLT | 0m39.201s
Building Clang:
```cmake ../llvm-project/llvm -DCMAKE_C_COMPILER=... -DCMAKE_CXX_COMPILER=...
-DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS=clang
-DLLVM_TARGETS_TO_BUILD=Native -GNinja```
| | Release | BOLT-optimized
| cmake | 0m24.016s | 0m22.333s
| ninja clang | 5m55.692s | 4m35.122s
I know it's not rigorous, but shows a ballpark figure.
Reviewed By: phosek
Differential Revision: https://reviews.llvm.org/D132975
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Current clang fail to bootstrap in PGO mode when only python3 is available,
because perf-helper.py is not compatible with python3.
Commited on behalf of Romain Geissler.
Differential Revision: https://reviews.llvm.org/D54071
llvm-svn: 350955
In Python3, dict.items, dict.keys, dict.values, zip, map and filter no longer return lists, they create generator instead.
The portability patch consists in forcing an extra `list` call if the result is actually used as a list.
`map` are replaced by list comprehension and `filter` by filtered list comprehension.
Differential Revision: https://reviews.llvm.org/D55197
llvm-svn: 349501
This reverts commit r277487.
Removing the probe predicate was a red herring. It results in more symbols being placed in the final order file, but they are symbols from outside the clang image.
llvm-svn: 277492
Having the dtrace predicate setup to only show probes in clang filters out static initializers executed by dyld, which we do want included in the order files.
llvm-svn: 277487
Dtrace probemod needs to be based on the first argument of the command, not the first argument of the args. This error was introduced a while back when I added support for skipping the driver and invoking cc1 directly.
llvm-svn: 277401
After r272599, -DLLVM_BUILD_INSTRUMENTED passes a default argument to
-fprofile-instr-generate. This confuses the perf-helper script because
the runtime emits a note stating that the default is overridden by the
LLVM_PROFILE_FILE environment variable.
Change the perf-helper script s.t it does not treat these notes as
failures.
This isn't a strictly NFC change, but I don't see a simple way to add a
test for it.
llvm-svn: 272695
When LIT parallelizes the profraw file generation we need to generate unique temp filenames then clean them up after the driver executes.
llvm-svn: 264021
This patch adds a new set of substitutions to the lit run lines for order files and PGO generation which run the clang driver to get the cc1 command, then execute the cc1 command directly. This allows the scripts to bypass profiling the clang driver over and over again.
The approach in this patch was discussed via IRC with Sean Silvas.
Special thanks to Daniel Dunbar whose out-of-tree code I liberally plagiarized.
llvm-svn: 263997
Summary:
This patch extends the lit-based perf-training tooling supplied for PGO data generation to also generate linker order files using dtrace.
This patch should work on any system that has dtrace. If CMake can find the dtrace tool it will generate a target 'generate-order-file' which will run the per-training tests wrapped by dtrace to capture function entries. There are several algorithms implemented for sorting the order files which can be experimented with for best performance. The dtrace wrapper also supports bot oneshot and pid probes.
The perf-helper.py changes to support order file construction are ported from internal changes by ddunbar; he gets all the credit for the hard work here, I just copy and pasted.
Note: I've tested these patches on FreeBSD and OS X 10.10.
Reviewers: ddunbar, bogner, silvas
Subscribers: llvm-commits, emaste
Differential Revision: http://reviews.llvm.org/D16134
llvm-svn: 257934
Summary:
This patch adds support for using LIT to drive generating PGO profile data for clang.
This first pass implementation should work on Linux and Unix based platforms. If you build clang using CMake with LLVM_BUILD_INSTRUMENTED=On the CMake build generates a generate-profdata target that will use the just-built clang to build any test files (see hello_world.cpp as an example). Each test compile will generate profraw files for each clang process. After all tests have run CMake will merge the profraw files using llvm-profdata.
Future opportunities for extension:
* Support for Build->Profile->Build bootstrapping
* Support for linker order file generation using a similar mechanism and the same training data
* Support for Windows
Reviewers: dexonsmith, friss, bogner, cmatthews, vsk, silvas
Subscribers: cfe-commits
Differential Revision: http://reviews.llvm.org/D15462
llvm-svn: 255740
