This patch removes the generate_test_report_buildkite script entrypoint
as we no longer need it now that buildkite has been sunsetted. Also
remove the calls in the monolithic-* scripts given they are adding
complexity for no value.
Also remove the generate-buildkite-pipeline-premerge script as it is
no longer needed.
Reviewers: tstellar, DavidSpickett, lnihlen, cmtice
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/143480
This patch resolves a fixme in the compute_projects script to actually
test all the projects we can when touching something in the .ci
directory. This ensures we test things like compiler-rt before landing
changes.
Reviewers: gburgessiv, lnihlen, cmtice
Reviewed By: cmtice, gburgessiv
Pull Request: https://github.com/llvm/llvm-project/pull/144034
I have a habit of using early exits given it is in the LLVM coding
standards, but most of the early exits used within this script were
trivial and actually adding complexity. These are all instances where we
only perform one operation after the early exit, so removing the early
exit means less lines of code and arguably more readable code.
Reviewers: DavidSpickett, tstellar, cmtice, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/143478
This reverts commit 6f62979a5a5bcf70d65f23e0991a274e6df5955b.
The reapplies commit 80ea5f46df3e365a0a2112889bb91732167b6214.
That commit was reverted because it was causing compiler-rt test
failures due to tysan not having its dependencies set up properly within
CMake. That situation has since been rectified in
3cef099ceddccefca8e11268624397cde9e04af6.
Reviewers: lnihlen, rnk, gburgessiv, cmtice
Reviewed By: rnk, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/144033
This patch migrates the premerge pipeline to use LLVM_ENABLE_RUNTIMES to
build libc and compiler-rt.
Reviewers: DavidSpickett, tstellar, cmtice, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142696
This patch adjusts the compute_projects script to explicitly determine
what runtimes should be built and what runtimes should be tested. This
is mainly to support enabling runtimes for LLDB testing but not test
them unless we are building clang.
Reviewers: Endilll, tstellar, DavidSpickett, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142695
This patch switches us to using LLVM_ENABLE_RUNTIMES rather than using
separate runtimes builds for some reductions in CMake configuration time
and some simplification of the monolithic-linux.sh script.
Reviewers: DavidSpickett, cmtice, lnihlen, Endilll, tstellar
Reviewed By: Endilll, DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/142694
Buildkite has a bunch of custom emoji that include Linux and Windows
logos -
https://github.com/buildkite/emojis?tab=readme-ov-file#smileys--emotion.
GitHub doesn't have these so let's just use a penguin and a generic
window. 🐧🪟
These are standard emoji so Buildkite does support them too, but there's
no motivation to change it there.
Plus, I think the metrics collection might be tied to Buildkite pipeline
name so keeping it the same saves hassle there.
Currently we will fail if there are no JUnit XML files produced from
llvm-lit invocations. This can happen if the build fails and no test
suites end up getting run or if we test a project that does not use
llvm-lit, libe libc.
This fixes#142038.
Previously we were using continue-on-error within the workflows to
prevent sending out notifications for workflow failures. We worked
around this in the metrics container to see what was actually
failing/passing by looking at the individual steps. Now that we have
gotten rid of the continue-on-error flag, we just have to look at the
job status.
This enables a script to come through later and download all the test
files for further offline analysis. This is intended to enable
developing a tool that can spot flaky tests.
This patch removes the "test only please ignore" tagline from the
premerge job names. Now that we are looking to sunset the old
infrastructure pretty soon and the new infrastructure is reporting
errors, we want people to actually pay attention to the failures and
report anything erroneous.
It seems like these tests were never run during the development of
30747cfe41f5d4f0b0083750ba9c20cfcccec117, which proceeded to break them.
This patch updates the tests to correspond to the changes introduced in
that patch.
The premerge system will fail somewhat often due to issues unrelated to
the patch being tested. This patch adds a link within the long form
outputs to the issue tracker prompting users to open an issue if they
see flakes/soemthing broken at HEAD/anything else wrong.
Looks like some of these were not properly formatted at some point. This
patch reformats these files so that future diffs are cleaner when
running the formatter over the whole file.
This patch renames the new premerge job as suggested in
https://discourse.llvm.org/t/github-ci-notifications-and-main-branch/85868/10.
This uses more industry standard terms (like CI vs premerge checks which
might be somewhat of a LLVM CI idiom?) and makes it more generic if we
end up doing postcommit testing through Github.
The CI scripts install some python dependencies primarily for testing
this patch moves these over to a single requirements file that also hash
pins everything using pip-compile to conform to best security and
reproducibility practices.
Some improvements to monolithic-linux CI:
1) Add correct configuration and dependencies for LLDB testing which
is actually relevant for clang changes.
2) Skip clang installation and separate configuration for runtimes.
They will be built with the just built clang either way.
This avoids building the runtimes twice when LLDB is also tested.
3) Make sure any generated clang reproducers end up as artifacts.
4) Set up llvm-symbolizer environment variable so that its preferred
over
any symbolizer just built, as it can be much slower when built for
debugging.
5) Add all projects as dependencies of `.ci`, to make sure everything is
tested when it changes.
Before this commit, we only pushed a queue/running count when the value
was not zero. This makes building Grafana alerting a bit harder.
Changing this to always upload a value for watched workflows.
This patch uses an env variable instead of the --break-system-packages
flag. This enables the heterogenous configuration between the old and
new premerge systems as the old premerge container does not recognize
the --break-system-packages flag. An env variable will work on new
premerge and have no impact on old premerge.
This reverts commit 23fb048ce35f672d8db3f466a2522354bbce66e5.
This broke the new premerge system as it appears the pip installations within
the CI image do not support this option. Buildkite was unaffected.
These changes are mostly pushed by the gnsyncbot directly to main and
thus don't go through a PR, but we still test on main to see if main is
broken. Given these touch llvm/, they end up burning a decent amount of
testing time for no real benefit, so I think it makes sense to exclude
them from premerge testing explicitly.
This patch fixes the monolithic linux build in Ubuntu 24.04. Newer
versions of debian/ubuntu pass a warning when installing packages at the
system level using pip as it interferes with system package manager
installed python packages. We do not use any system package manager
installed python packages, so we just ignore the warning (that is an
error without passing the flag) by passing the --break-system-packages
flag.
This patch adds rich test failure information to the Github output,
using the same library that is used for the buildkite pipeline.
Eventually I think we want to add more information like reproduction
information using the containers, but that is very divergent between
Github and Buildkite, so we probably want to wait until we've switched
over before doing that.
Reviewers: Keenuts, tstellar, lnihlen, DavidSpickett
Reviewed By: DavidSpickett, Keenuts
Pull Request: https://github.com/llvm/llvm-project/pull/133197
Currently when someone touches a docs directory in a subproject, it is
treated as if the source code of that project got touched, so the
project is built, it is tested, and the same for all of its enumerated
dependents. This is wasteful, particularly for patches just touching
docs in places like LLVM where we might spend an hour of node time to do
nothing useful given changes in the docs shouldn't cause test failures
and there is already another workflow that tests the documentation build
completes successfully.
Reviewers: Keenuts, tstellar, lnihlen
Reviewed By: tstellar
Pull Request: https://github.com/llvm/llvm-project/pull/133185
This patch migrates the CI over to the new compute_projects.py script
for calculating what projects need to be tested based on a change to
LLVM.
Reviewers: lnihlen, ldionne, tstellar, Endilll, joker-eph, Keenuts
Reviewed By: Keenuts, tstellar
Pull Request: https://github.com/llvm/llvm-project/pull/132642
This patch refactors the generate_test_report script, namely turning it
into a proper library, and pulling the script/unittests out into
separate files, as is standard with most python scripts. The main
purpose of this is to enable reusing the library for the new Github
premerge.
Reviewers: tstellar, DavidSpickett, Keenuts, lnihlen
Reviewed By: DavidSpickett
Pull Request: https://github.com/llvm/llvm-project/pull/133196
Before this patch, making a change to a runtime directory (like libcxx)
would cause the project to be added to the LLVM_ENABLE_PROJECTS CMake
flag which is illegal as they can only be built as part of
LLVM_ENABLE_RUNTIMES. This patch fixes that behavior. Test added.
This patch adds a python script, compute_projects, and associated unit
tests for computing the projects and runtimes that need to be tested in
premerge. Rewriting in Python opens up a couple new
improvements/opportunities:
1. I personally find python to be much easier to work with than shell
scripts for tasks like this. Particularly it becomes a lot easier to
work with paths with proper array support.
2. Unit testing becomes easier which makes it a lot easier to reason
about behavior changes, especially in review.
3. Most of the configuration is now setup in some dictionaries, which
makes changes much easier to apply for most of the common changes.
This preserves the behavior of the existing premerge scripts as much as
possible.
Reviewers: ldionne, lnihlen, Endilll, tstellar, Keenuts
Reviewed By: Keenuts
Pull Request: https://github.com/llvm/llvm-project/pull/132634
Between C++26 and Clang modules build of runtimes, which are used as an
additional testing for Clang, the only difference are
`LIBCXX_TEST_PARAMS` and `LIBCXXABI_TEST_PARAMS`. Both of them are
transformed into actual lit configuration lines, and put into
`SERIALIZED_LIT_PARAMS`, which end up in `libcxx/test/cmake-bridge.cfg`
via `configure_file` command. Notably, it seems that they are not used
in any other way.
I checked that if those variables are changed, subsequent runs of CMake
configuration step regenerate `cmake-bridge.cfg` with the new values.
Which means that we don't need to do clean builds for every runtimes
configuration we want to test.
I hope that together with #131913, this will be enough to alleviate
Linux CI pains we're having, and we wouldn't have to make a tough choice
between C++26 and Clang modules builds for pre-merge CI.
This patch cleans up the runtimes build in premerge due to queuing
delays, dropping the C++03 testing, but keeping the C++20 and Modules
configurations as they are deemed important by clang contributors.
This patch also makes it easier in the future when we need to rework the
runtimes build to anticipate the deprecation of building most of the
runtimes with LLVM_ENABLE_PROJECTS.
This reverts commit 95d28fe503cc3d2bc0bb980442d3defaf199ea5a.
I did not fully realize the implications of this change when reviewing.
With how it is set up currently, it causes clang and all of the runtimes
to be built and tested everytime a change to MLIR is made. This is a
large regression in build/test time, which seems to have been causing
large queueing delays.
Reverting for now. Once we rework the runtimes build for premerge (which
I hope to do soon, ideally in the next week), I will make sure flang-rt
gets added in.
This patch bumps the maximum number of metrics to look through when
collecting metrics data. We are currently running into issues where we
are losing data due to the most recent 1000 workflows not containing the
workflows that we actually need to query. Just double it for now.
I plan on monitoring this reasonably closely to ensure we do not run
into issues, mainly API rate limits.
The current container focuses on Github metrics. Before deprecating
BuildKite, we want to make sure the new infra quality is better, or at
least the same.
Being able to compare buildkite metrics with github metrics on grafana
will allow us to easily present the comparison.
BuildKite API allows filtering, but doesn't allow changing the result
ordering. Meaning we are left with builds ordered by IDs. This means a
completed job can appear before a running job in the list. 2 solutions
from there:
- keep the cursor on the oldest running workflow
- keep a list of running workflows to compare.
Because there is no guarantees in workflow ordering, waiting for the
oldest build to complete before reporting any newer build could mean
delaying the more recent build completion reporting by a few hours. And
because grafana cannot ingest metrics older than 2 hours, this is not an
option.
Thus we leave with the second solution: remember what jobs were running
during the last iteration, and record them as soon as they are
completed. Buildkite has at most ~100 pending jobs, so keeping all those
IDs should be OK.
Flang's runtime can now be built using LLVM's LLVM_ENABLE_RUNTIMES
mechanism, with the intent to remove the old mechanism in #124126.
Update the pre-merge builders to use the new mechanism.
In the current form, #124126 actually will add
LLVM_ENABLE_RUNTIMES=flang-rt implicitly, so no change is strictly
needed. I still think it is a good idea to do it explicitly and in
advance.
On Windows, flang-rt also requires compiler-rt, but which is not
building on Windows anyway.
Yesterday, the monitoring reported a job queued for 23h59. After some
checks, it appears no such job existed: the age of the workflows on
completion was at most 5 hours during the last 48 hours.
After some digging, I found out GitHub could return a job with a start
date slightly before the creation date, or completion date before start
date.
This would cause python to compute a negative timedelta, which would
then be reported in grafana as a full 24h delta due to the conversions.
Adding code to ignore negative delta, but logging them.
Before this patch, the job/workflow name impacted the metric name,
meaning a change in the workflow definition could break monitoring. This
patch adds a map to get a stable name on metrics from a workflow name.
In addition, it reworks a bit how we track the last processed workflow:
the github queries are broken if filtering is applied, meaning we have a
list of workflow, ordered by 'created_at', which mixes completed &
running workflows.
We have no guarantees over the order of completion, meaning we cannot
stop at the first completed job we found (even per-workflow).
This PR processed the last 1000 workflows, but allows an early stop if
the created_at time is older than 8 hours. This means we could miss
long-running workflows (>8 hours), and if the number of workflows
started before another one completes becomes high (>1000), we'll miss
it.
To detect this kind of behavior, a new metric is added "oldest workflow
processed", which should at least indicate if the depth is too small.
An alternative without arbitrary cut would be to initially parse all
workflows, and then record the last non-completed one we find and always
start from the last (moving the lower bound as they complete). But LLVM
has forever-queued workflows runs (>1 years), hence this would cause us
to iterate over a very large number of jobs.
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
