We do not gain any additional security from having PIE on the test
binaries, and this does have an impact on test times. It is about 10%
from my measurements. Not as much as previous reports
(https://reviews.llvm.org/D140880), but still significant. This is a
similar gain to finish enabling the lit internal shell, which will take
significantly more effort. I also do not think this is a meaningful
enough difference from the "default" configuration to keep it the
default, especially given the option is mostly set by distros.
This patch makes all of the ninja commands in the monolithic-* scripts
write to log files in the current working directory. The plan is to use
this to feed the ninja log into generate_test_report_github.py so we can
surface compilation errors.
Related to #152246.
Reviewers: Keenuts, lnihlen, cmtice, dschuff, gburgessiv
Reviewed By: Keenuts, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/152331
We currently log every single test that we run in premerge. This leads
to gigantic logs (200k+ lines on Linux) that can be difficult to parse
through. Having an indicator of progress is nice, especially for the
LLVM tests, but is not strictly necessary and not often used (I
imagine). Having a progress indicator from lit that works in CI cases is
on my TODO list.
For the rare cases where someone does need to see the list of tests that
run, the JUnit XML emitted by lit is available in the artifacts.
This patch refactors big chunks of the common functionality shared
between monolithic-linux.sh and monolithic-windows.sh to a separate
script, utils.sh, that is then sourced in both of the files. This makes
it a bit easier to maintain the scripts.
Platform differences should not be a large deal for the setup here as we
are using bash as the shell on both Linux and Windows.
Reviewers:
lnihlen, gburgessiv, Keenuts, DavidSpickett, dschuff, cmtice, Endilll
Reviewed By: DavidSpickett, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/152199
This patch moves the platform specific test report titles to the
generate_test_report_github.py script. This means that the functions in
the monolithic-* scripts are exactly the same now and can be moved into
a separate script that can be shared between the two scripts.
Reviewers: DavidSpickett, cmtice, lnihlen, dschuff, Keenuts, gburgessiv
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/152198
We run a couple Github only steps in the monolithic-* scripts like
writing to $GITHUB_STEP_SUMMARY and denoting log groups. These can throw
errors when running the scripts locally or as part of other CI systems
(like buildbot). This patch makes it so that we only run these commands
when in the relevant environment. We also add a $POSTCOMMIT_CI check for
groups so that the annotated builder in buildbot works properly with
these scripts.
Reviewers: cmtice, ldionne, Endilll, gburgessiv, Keenuts, dschuff, lnihlen
Pull Request: https://github.com/llvm/llvm-project/pull/152197
This change modifies CI scripts to add a pseudo-project for CIR and
detect when CIR-specific files are modified. It also enables building
clang with CIR enabled whenever both the clang and mlir projects are
being built.
Building and testing CIR is only enabled on Linux at this time, as CIR
doesn't properly support Windows or MacOS yet.
This is in preparation for migrating to Google Cloud Storage (GCS) based
caching soon which is only supported by sccache.
Reviewers: Keenuts, gburgessiv, dschuff, lnihlen, cmtice
Reviewed By: cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/149195
This patch drops support for clearing the cache with the CLEAR_CACHE
environment variable. This is an artifact of the old infrastructure as
we now do not persist the cache across builds, instead redownloading the
cache directory everytime. This makes the scripts slightly simpler as we
are no longer supporting unneeded functionality.
Reviewers: Endilll, cmtice, dschuff, Keenuts, lnihlen, gburgessiv
Reviewed By: Keenuts, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/149193
This patch removes the generate_test_report_buildkite script entrypoint
as we no longer need it now that buildkite has been sunsetted. Also
remove the calls in the monolithic-* scripts given they are adding
complexity for no value.
Also remove the generate-buildkite-pipeline-premerge script as it is
no longer needed.
Reviewers: tstellar, DavidSpickett, lnihlen, cmtice
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/143480
This reverts commit 6f62979a5a5bcf70d65f23e0991a274e6df5955b.
The reapplies commit 80ea5f46df3e365a0a2112889bb91732167b6214.
That commit was reverted because it was causing compiler-rt test
failures due to tysan not having its dependencies set up properly within
CMake. That situation has since been rectified in
3cef099ceddccefca8e11268624397cde9e04af6.
Reviewers: lnihlen, rnk, gburgessiv, cmtice
Reviewed By: rnk, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/144033
This patch migrates the premerge pipeline to use LLVM_ENABLE_RUNTIMES to
build libc and compiler-rt.
Reviewers: DavidSpickett, tstellar, cmtice, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142696
This patch adjusts the compute_projects script to explicitly determine
what runtimes should be built and what runtimes should be tested. This
is mainly to support enabling runtimes for LLDB testing but not test
them unless we are building clang.
Reviewers: Endilll, tstellar, DavidSpickett, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142695
This patch switches us to using LLVM_ENABLE_RUNTIMES rather than using
separate runtimes builds for some reductions in CMake configuration time
and some simplification of the monolithic-linux.sh script.
Reviewers: DavidSpickett, cmtice, lnihlen, Endilll, tstellar
Reviewed By: Endilll, DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142694
Buildkite has a bunch of custom emoji that include Linux and Windows
logos -
https://github.com/buildkite/emojis?tab=readme-ov-file#smileys--emotion.
GitHub doesn't have these so let's just use a penguin and a generic
window. 🐧🪟
These are standard emoji so Buildkite does support them too, but there's
no motivation to change it there.
Plus, I think the metrics collection might be tied to Buildkite pipeline
name so keeping it the same saves hassle there.
Currently we will fail if there are no JUnit XML files produced from
llvm-lit invocations. This can happen if the build fails and no test
suites end up getting run or if we test a project that does not use
llvm-lit, libe libc.
This fixes#142038.
This enables a script to come through later and download all the test
files for further offline analysis. This is intended to enable
developing a tool that can spot flaky tests.
The CI scripts install some python dependencies primarily for testing
this patch moves these over to a single requirements file that also hash
pins everything using pip-compile to conform to best security and
reproducibility practices.
Some improvements to monolithic-linux CI:
1) Add correct configuration and dependencies for LLDB testing which
is actually relevant for clang changes.
2) Skip clang installation and separate configuration for runtimes.
They will be built with the just built clang either way.
This avoids building the runtimes twice when LLDB is also tested.
3) Make sure any generated clang reproducers end up as artifacts.
4) Set up llvm-symbolizer environment variable so that its preferred
over
any symbolizer just built, as it can be much slower when built for
debugging.
5) Add all projects as dependencies of `.ci`, to make sure everything is
tested when it changes.
This patch uses an env variable instead of the --break-system-packages
flag. This enables the heterogenous configuration between the old and
new premerge systems as the old premerge container does not recognize
the --break-system-packages flag. An env variable will work on new
premerge and have no impact on old premerge.
This reverts commit 23fb048ce35f672d8db3f466a2522354bbce66e5.
This broke the new premerge system as it appears the pip installations within
the CI image do not support this option. Buildkite was unaffected.
This patch fixes the monolithic linux build in Ubuntu 24.04. Newer
versions of debian/ubuntu pass a warning when installing packages at the
system level using pip as it interferes with system package manager
installed python packages. We do not use any system package manager
installed python packages, so we just ignore the warning (that is an
error without passing the flag) by passing the --break-system-packages
flag.
This patch adds rich test failure information to the Github output,
using the same library that is used for the buildkite pipeline.
Eventually I think we want to add more information like reproduction
information using the containers, but that is very divergent between
Github and Buildkite, so we probably want to wait until we've switched
over before doing that.
Reviewers: Keenuts, tstellar, lnihlen, DavidSpickett
Reviewed By: DavidSpickett, Keenuts
Pull Request: https://github.com/llvm/llvm-project/pull/133197
This patch refactors the generate_test_report script, namely turning it
into a proper library, and pulling the script/unittests out into
separate files, as is standard with most python scripts. The main
purpose of this is to enable reusing the library for the new Github
premerge.
Reviewers: tstellar, DavidSpickett, Keenuts, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/133196
Between C++26 and Clang modules build of runtimes, which are used as an
additional testing for Clang, the only difference are
`LIBCXX_TEST_PARAMS` and `LIBCXXABI_TEST_PARAMS`. Both of them are
transformed into actual lit configuration lines, and put into
`SERIALIZED_LIT_PARAMS`, which end up in `libcxx/test/cmake-bridge.cfg`
via `configure_file` command. Notably, it seems that they are not used
in any other way.
I checked that if those variables are changed, subsequent runs of CMake
configuration step regenerate `cmake-bridge.cfg` with the new values.
Which means that we don't need to do clean builds for every runtimes
configuration we want to test.
I hope that together with #131913, this will be enough to alleviate
Linux CI pains we're having, and we wouldn't have to make a tough choice
between C++26 and Clang modules builds for pre-merge CI.
This patch cleans up the runtimes build in premerge due to queuing
delays, dropping the C++03 testing, but keeping the C++20 and Modules
configurations as they are deemed important by clang contributors.
This patch also makes it easier in the future when we need to rework the
runtimes build to anticipate the deprecation of building most of the
runtimes with LLVM_ENABLE_PROJECTS.
This reverts commit 95d28fe503cc3d2bc0bb980442d3defaf199ea5a.
I did not fully realize the implications of this change when reviewing.
With how it is set up currently, it causes clang and all of the runtimes
to be built and tested everytime a change to MLIR is made. This is a
large regression in build/test time, which seems to have been causing
large queueing delays.
Reverting for now. Once we rework the runtimes build for premerge (which
I hope to do soon, ideally in the next week), I will make sure flang-rt
gets added in.
Flang's runtime can now be built using LLVM's LLVM_ENABLE_RUNTIMES
mechanism, with the intent to remove the old mechanism in #124126.
Update the pre-merge builders to use the new mechanism.
In the current form, #124126 actually will add
LLVM_ENABLE_RUNTIMES=flang-rt implicitly, so no change is strictly
needed. I still think it is a good idea to do it explicitly and in
advance.
On Windows, flang-rt also requires compiler-rt, but which is not
building on Windows anyway.
In this build:
https://buildkite.com/llvm-project/github-pull-requests/builds/126961
The builds actually failed, probably because prerequisite of a test
suite failed to build.
However they still ran other tests and all those passed. This meant that
the test reports were green even though the build was red. On some level
this is technically correct, but it is very misleading in practice.
So I've also passed the build script's return code, as it was when we
entered the on exit handler, to the generator, so that when this happens
again, the report will draw the viewer's attention to the overall
failure. There will be a link in the report to the build's log file, so
the next step to investigate is clear.
It would be nice to say "tests failed and there was some other build
error", but we cannot tell what the non-zero return code was caused by.
Could be either.
The script handles the following situations now:
| Have Result Files? | Tests reported failed? | Return code | Report |
|--------------------|------------------------|-------------|-----------------------------------------------------------------------------|
| Yes | No | 0 | Success style report. |
| Yes | Yes | 0 | Shouldn't happen, but if it did, failure style report
showing the failures. |
| Yes | No | 1 | Failure style report, showing no failures but noting
that the build failed. |
| Yes | Yes | 1 | Failure style report, showing the test failures. |
| No | ? | 0 | No test report, success shown in the normal build
display. |
| No | ? | 1 | No test report, failure shown in the normal build
display. |
This patch modifies the monolithic shell scrips to only run if the
buildkite-agent application is present. This allows for running the
scripts to completion outside of buildkite (eg inside of a GHA
pipeline).
This reverts commit 8a1ca6cad9cd0e972c322910cdfbbe9552c6c7ca.
I have fixed 2 things:
* The report is now sent by stdin so we do not hit the limit on the size
of command line arguments.
* The report is limited to 1MB in size and if we exceed that we fall back
to listing only the totals with a note telling you to check the full log.
This reverts commit e74a002433b4cf7f891ceedb61bd862867218a8b.
As it is failing on Linux with "OSError: [Errno 7] Argument list too long: 'buildkite-agent'".
The CI builds now send the results of every lit run to a unique file.
This means we can read them all to make a combined report for all
tests.
This report will be shown as an "annotation" in the build results:
https://buildkite.com/docs/agent/v3/cli-annotate#creating-an-annotation
Here is an example:
https://buildkite.com/llvm-project/github-pull-requests/builds/112660
(make sure it is showing "All" instead of "Failures")
This is an alternative to using the existing Buildkite plugin:
https://github.com/buildkite-plugins/junit-annotate-buildkite-plugin
As the plugin is:
* Specific to Buildkite, and we may move away from Buildkite.
* Requires docker, unless we were to fork it ourselves.
* Does not let you customise the report format unless again,
we make our own fork.
Annotations use GitHub's flavour of Markdown so the main code in the
script generates that text. There is an extra "style" argument generated
to make the formatting nicer in Buildkite.
"context" is the name of the annotation that will be created. By using
different context names for Linux and Windows results we get 2 separate
annotations.
The script also handles calling the buildkite-agent. This makes passing
extra arguments to the agent easier, rather than piping the output of
this script into the agent.
In the future we can remove the agent part of it and simply use
the report content. Either printed to stdout or as a comment on
the GitHub PR.
In this patch I'm using a new lit option so that the pipeline writes
many results files, one for each time lit is run:
```
--use-unique-output-file-name
When enabled, lit will add a unique element to the output file name, before the extension. For example "results.xml" will become "results.<something>.xml". The
"<something>" is not ordered in any way and is chosen so that existing files are not overwritten. [Default: Off]
```
(I added this to lit recently)
Alternatives were considered:
* mkfifo - does not work on bash for Windows.
* tail -f - does not print full content on file truncation
* lit wrapper script - more complication than using an option to lit
itself
* ninja/mv file/ninja/mv file etc - lots of changes needed to make the
scripts build each target separately
And after feedback I decided that using an option to lit itself is the
cleanest way to go. It can be removed when we no longer need it.
If I run the Linux build after this change:
```
$ bash ./.ci/monolithic-linux.sh "clang;lldb;lld" "check-lldb-shell check-lld" "libcxx;libcxxabi" "check-libcxx check-libcxxabi"
```
I get multiple test result files. In my case some tests fail so runtimes
aren't checked, but all projects are so there is 1 file for lldb and one
for lld:
```
$ ls build/*.xml
build/test-results.klc82utf.xml build/test-results.majylh73.xml
```
This change just collects the XML files as artifacts. Once I know that's
working, I can set up test reporting to make a summary of them.
This patch removes LLDB from a list of projects that are excluded from
building and testing on pre-merge CI on Linux.
Windows environment needs to be prepared in order to test LLDB
(https://github.com/llvm/llvm-project/pull/94208#issuecomment-2146256857),
but we don't have enough maintenance resources to do that at the moment.
Because LLDB has been in the list of projects that need to be tested on
Clang changes, this PR make this happen on Linux. This seems to be the
consensus in the discussion of this PR.
Following the discussion in
https://github.com/llvm/llvm-project/pull/93233#issuecomment-2127920882,
this patch merges `clang-ci` pipeline into main `GitHub Pull Requests`
pipeline. `clang-ci` enables additional test coverage for Clang by
compiling it, and then using it to compile and test libc++, libc++abi,
and libunwind in C++03, C++26, and Clang Modules modes.
Additional work we skip and total time savings we should see:
1. Checking out the repo to generate the clang-ci pipeline (2 minutes)
2. Building Clang (3.5 minutes)
3. Uploading the artifacts once, then downloading them 3 times and
unpacking 3 times (0.5 minutes)
Note that because previously-split jobs for each mode are now under a
single Linux job, it now takes around 8 minutes more see the Linux CI
results despite total time savings.
The primary goal of this patch is to reduce the load of CI by removing
duplicated work. I consider this goal achieved. I could keep the job
parallelism we had (3 libc++ jobs depending on a main Linux job), but I
don't consider it worth the effort and opportunity cost, because
parallelism is not helping once the pool of builders is fully
subscribed.
This fixes all shellcheck warnings we have in `monolithic-linux.sh` and
`monolithic-windows.sh`.
All of them have to do with
[SC2086](https://www.shellcheck.net/wiki/SC2086) - Double quote to
prevent globbing and word splitting.
#84828 added `-k 0` to pre-merge CI so that if one job fails the others
would continue building. This pull request fixes the location of `-k 0`
in the ninja command line.
Resolves#84842 and #83371
The -k option allows to continue the build after failures as much as
possible. This is useful here because when we run
> ninja check-llvm check-clang
we would like the clang tests to run even if there is a failure in a
llvm tests.
The downside is that a build failure in one file that would prevent from
running any test does not prevent from building more targets, wasting
build resources potentially.
Fixes#83371
