We do not gain any additional security from having PIE on the test
binaries, and this does have an impact on test times. It is about 10%
from my measurements. Not as much as previous reports
(https://reviews.llvm.org/D140880), but still significant. This is a
similar gain to finish enabling the lit internal shell, which will take
significantly more effort. I also do not think this is a meaningful
enough difference from the "default" configuration to keep it the
default, especially given the option is mostly set by distros.
This attempts https://github.com/llvm/llvm-project/issues/132795 again.
Last time we tried this we didn't have enough infra capacity, so had to
revert. According to recent communication from the Infrastructure Area
Team, we should now have enough capacity to re-enable the LLDB tests.
We were passing the wrong directory to `generate_test_report_github.py`,
so no logs actually got passed there. This fixes that and ensures that
the logs are uploaded as artifacts to enable future debugging.
This patch finishes up the plumbing so that generate_test_report will dump build
failures into the Github checks summary.
Reviewers: Keenuts, lnihlen, dschuff, gburgessiv, cmtice, DavidSpickett
Reviewed By: cmtice, DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/152622
This patch makes it so that generate_report will add information about
failed build actions to the summary report. This makes it significantly
easier to find compilation failures, especially given we run ninja with
-k 0.
This patch only does the integration into generate_report (along with
testing). Actual utilization in the script is split into a separate
patch to try and keep things clean.
Reviewers: dschuff, cmtice, DavidSpickett, Keenuts, lnihlen, gburgessiv
Reviewed By: cmtice, DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/152621
This patch adds in support for taking the CLI output from ninja and
parsing it for failures. This is intended to be used in the cases where
all tests pass (or none have run), but the build fails to easily surface
where exactly the build failed.
The actual integration will happen in a future patch.
Reviewers: gburgessiv, dschuff, lnihlen, DavidSpickett, Keenuts, cmtice
Reviewed By: DavidSpickett, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/152620
This patch makes all of the ninja commands in the monolithic-* scripts
write to log files in the current working directory. The plan is to use
this to feed the ninja log into generate_test_report_github.py so we can
surface compilation errors.
Related to #152246.
Reviewers: Keenuts, lnihlen, cmtice, dschuff, gburgessiv
Reviewed By: Keenuts, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/152331
We currently log every single test that we run in premerge. This leads
to gigantic logs (200k+ lines on Linux) that can be difficult to parse
through. Having an indicator of progress is nice, especially for the
LLVM tests, but is not strictly necessary and not often used (I
imagine). Having a progress indicator from lit that works in CI cases is
on my TODO list.
For the rare cases where someone does need to see the list of tests that
run, the JUnit XML emitted by lit is available in the artifacts.
This patch refactors big chunks of the common functionality shared
between monolithic-linux.sh and monolithic-windows.sh to a separate
script, utils.sh, that is then sourced in both of the files. This makes
it a bit easier to maintain the scripts.
Platform differences should not be a large deal for the setup here as we
are using bash as the shell on both Linux and Windows.
Reviewers:
lnihlen, gburgessiv, Keenuts, DavidSpickett, dschuff, cmtice, Endilll
Reviewed By: DavidSpickett, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/152199
This patch adds libcxx/libcxxabi/libunwind to the excludes list in
compute_projects.py for both Windows and MacOS. Neither of these
platforms have ever built the runtimes as the scripts do not have
support for it. Explicitly disable them so that compute_projects_test.py
is more consistent.
This patch moves the platform specific test report titles to the
generate_test_report_github.py script. This means that the functions in
the monolithic-* scripts are exactly the same now and can be moved into
a separate script that can be shared between the two scripts.
Reviewers: DavidSpickett, cmtice, lnihlen, dschuff, Keenuts, gburgessiv
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/152198
We run a couple Github only steps in the monolithic-* scripts like
writing to $GITHUB_STEP_SUMMARY and denoting log groups. These can throw
errors when running the scripts locally or as part of other CI systems
(like buildbot). This patch makes it so that we only run these commands
when in the relevant environment. We also add a $POSTCOMMIT_CI check for
groups so that the annotated builder in buildbot works properly with
these scripts.
Reviewers: cmtice, ldionne, Endilll, gburgessiv, Keenuts, dschuff, lnihlen
Pull Request: https://github.com/llvm/llvm-project/pull/152197
Partially to give me something to copy and paste into other scripts that
need to run all of the tests on Windows and partially to increase test
coverage on Windows. This does expose an issue with the script on
Windows, so that needs to be fixed. Given it does not impact anything it
should be fine for now though.
This patch adds some initial unittests for the metrics collection
script. This initial patch focuses on getting the files setup and adding
unittests for uploading metrics. A subsequent patch will add tests for
the workflow collection which is significantly more complicated.
Reviewers: Keenuts, gburgessiv, dschuff, cmtice, lnihlen
Reviewed By: cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/150360
This patch ensures that we run all the tests when someone touches
third-party. Some of the projects (like SipHash and Benchmark) do not
get used by every project and we could get away with testing less.
However, it takes quite a bit of effort to tease out what actually does
depend on which library, it is liable to becoming out of date, and it
adds complexity. Just testing all the projects is simple. Given the
commit frequency to third-party is pretty low (a couple commits per
month), we should have no capacity problems here.
Fixes#149154
Reviewers: dschuff, gburgessiv, Keenuts, lnihlen, dpaoliello, cmtice
Reviewed By: dpaoliello
Pull Request: https://github.com/llvm/llvm-project/pull/150251
This patch makes it so that we actually run all the tests when we touch
the workflow file in .github/workflows/premerge.yaml. Before this we
would not run any tests and it was annoying to have to manually touch a
file to get something to trigger.
Reviewers: Keenuts, lnihlen, cmtice, dschuff, gburgessiv
Reviewed By: gburgessiv
Pull Request: https://github.com/llvm/llvm-project/pull/150250
This patch introduces the concept of meta projects into the
compute_projects script. Meta projects are projects like CIR and
GoogleTest where they do not have their own top level project. This
patch adds a little bit of extra code in exchange for making meta
projects a first level concept that can be configured almost entirely by
changing around the mappings at the top of the file. This patch also
refactors the project skipping functionality to use meta projects.
This (arguably) makes the CIR support a little bit cleaner and is
necessary for some future improvements like running all the tests on
Github workflow changes and running tests when third-party changes.
Reviewers: Endilll, andykaylor, lnihlen, gburgessiv, dschuff, Keenuts, cmtice
Reviewed By: andykaylor
Pull Request: https://github.com/llvm/llvm-project/pull/150249
The metrics script did not have a license header or a docstring. The
compute_projects_test.py file had a placeholder module level docstring
that I edited into something more reasonable.
This change modifies CI scripts to add a pseudo-project for CIR and
detect when CIR-specific files are modified. It also enables building
clang with CIR enabled whenever both the clang and mlir projects are
being built.
Building and testing CIR is only enabled on Linux at this time, as CIR
doesn't properly support Windows or MacOS yet.
This is in preparation for migrating to Google Cloud Storage (GCS) based
caching soon which is only supported by sccache.
Reviewers: Keenuts, gburgessiv, dschuff, lnihlen, cmtice
Reviewed By: cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/149195
This patch drops support for clearing the cache with the CLEAR_CACHE
environment variable. This is an artifact of the old infrastructure as
we now do not persist the cache across builds, instead redownloading the
cache directory everytime. This makes the scripts slightly simpler as we
are no longer supporting unneeded functionality.
Reviewers: Endilll, cmtice, dschuff, Keenuts, lnihlen, gburgessiv
Reviewed By: Keenuts, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/149193
This patch makes it so that generate_test_report_github.py generates a
test report even when we don't get any test results. This otherwise
created a pretty confusing user experience on the Github side if the
build failed before any tests ran or in cases like running check-libc
where none of the tests are run through lit.
Reviewers: lnihlen, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/147871
This is another Buildkite relic that we can get rid of now to simplify
things a bit.
Reviewers: cmtice, DavidSpickett, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/147791
This patch removes the buildkite_info object from
generate_test_report_lib as we no longer support buildkite so it is dead
code now. Eventually we should set up equivalent functionality on the
Github side for downloading the logs from a convenient link, but for now
clean up the code.
Reviewers: cmtice, DavidSpickett, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/147784
This patch removes setting the MAX_PARLLEL_COMPILE_JOBS and
MAX_PARALLEL_LINK_JOBS env variables in the windows runs. These were
originally used to control the parallelism on the old infrastructure and
we set them on the new infrastructure explicitly so that we could
maintain both at the same time. Now it does not make sense to keep them
explicitly set that we do not need to explicitly control the parallelism
given the amount of RAM we have on the machines. This also adds a
maintnenace cost as evidenced by the fact that these have been incorrect
(64 instead of 32) for quite a while.
This patch removes the generate_test_report_buildkite script entrypoint
as we no longer need it now that buildkite has been sunsetted. Also
remove the calls in the monolithic-* scripts given they are adding
complexity for no value.
Also remove the generate-buildkite-pipeline-premerge script as it is
no longer needed.
Reviewers: tstellar, DavidSpickett, lnihlen, cmtice
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/143480
This patch resolves a fixme in the compute_projects script to actually
test all the projects we can when touching something in the .ci
directory. This ensures we test things like compiler-rt before landing
changes.
Reviewers: gburgessiv, lnihlen, cmtice
Reviewed By: cmtice, gburgessiv
Pull Request: https://github.com/llvm/llvm-project/pull/144034
I have a habit of using early exits given it is in the LLVM coding
standards, but most of the early exits used within this script were
trivial and actually adding complexity. These are all instances where we
only perform one operation after the early exit, so removing the early
exit means less lines of code and arguably more readable code.
Reviewers: DavidSpickett, tstellar, cmtice, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/143478
This reverts commit 6f62979a5a5bcf70d65f23e0991a274e6df5955b.
The reapplies commit 80ea5f46df3e365a0a2112889bb91732167b6214.
That commit was reverted because it was causing compiler-rt test
failures due to tysan not having its dependencies set up properly within
CMake. That situation has since been rectified in
3cef099ceddccefca8e11268624397cde9e04af6.
Reviewers: lnihlen, rnk, gburgessiv, cmtice
Reviewed By: rnk, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/144033
This patch migrates the premerge pipeline to use LLVM_ENABLE_RUNTIMES to
build libc and compiler-rt.
Reviewers: DavidSpickett, tstellar, cmtice, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142696
This patch adjusts the compute_projects script to explicitly determine
what runtimes should be built and what runtimes should be tested. This
is mainly to support enabling runtimes for LLDB testing but not test
them unless we are building clang.
Reviewers: Endilll, tstellar, DavidSpickett, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142695
This patch switches us to using LLVM_ENABLE_RUNTIMES rather than using
separate runtimes builds for some reductions in CMake configuration time
and some simplification of the monolithic-linux.sh script.
Reviewers: DavidSpickett, cmtice, lnihlen, Endilll, tstellar
Reviewed By: Endilll, DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142694
Buildkite has a bunch of custom emoji that include Linux and Windows
logos -
https://github.com/buildkite/emojis?tab=readme-ov-file#smileys--emotion.
GitHub doesn't have these so let's just use a penguin and a generic
window. 🐧🪟
These are standard emoji so Buildkite does support them too, but there's
no motivation to change it there.
Plus, I think the metrics collection might be tied to Buildkite pipeline
name so keeping it the same saves hassle there.
Currently we will fail if there are no JUnit XML files produced from
llvm-lit invocations. This can happen if the build fails and no test
suites end up getting run or if we test a project that does not use
llvm-lit, libe libc.
This fixes#142038.
Previously we were using continue-on-error within the workflows to
prevent sending out notifications for workflow failures. We worked
around this in the metrics container to see what was actually
failing/passing by looking at the individual steps. Now that we have
gotten rid of the continue-on-error flag, we just have to look at the
job status.
This enables a script to come through later and download all the test
files for further offline analysis. This is intended to enable
developing a tool that can spot flaky tests.
This patch removes the "test only please ignore" tagline from the
premerge job names. Now that we are looking to sunset the old
infrastructure pretty soon and the new infrastructure is reporting
errors, we want people to actually pay attention to the failures and
report anything erroneous.
It seems like these tests were never run during the development of
30747cfe41f5d4f0b0083750ba9c20cfcccec117, which proceeded to break them.
This patch updates the tests to correspond to the changes introduced in
that patch.
The premerge system will fail somewhat often due to issues unrelated to
the patch being tested. This patch adds a link within the long form
outputs to the issue tracker prompting users to open an issue if they
see flakes/soemthing broken at HEAD/anything else wrong.
Looks like some of these were not properly formatted at some point. This
patch reformats these files so that future diffs are cleaner when
running the formatter over the whole file.
This patch renames the new premerge job as suggested in
https://discourse.llvm.org/t/github-ci-notifications-and-main-branch/85868/10.
This uses more industry standard terms (like CI vs premerge checks which
might be somewhat of a LLVM CI idiom?) and makes it more generic if we
end up doing postcommit testing through Github.
The CI scripts install some python dependencies primarily for testing
this patch moves these over to a single requirements file that also hash
pins everything using pip-compile to conform to best security and
reproducibility practices.
