…tting" (#108104)"
This recommits 0f56ba13bff7ab72bfafcf7c5cf9e5b8bd16d895 (reverted by
6007ad79afeffb1288781b4a7241290386293aff). In the original patch
llvm/utils/lit/tests/escape-color.py failed on Windows because it diffed
llvm-lit output with a file containing '\n' newlines rather than '\r\n'.
This issue is avoided by calling 'diff --strip-trailing-cr'.
Original description below:
Test output that carried color across newlines previously resulted in
the formatting around the output also being colored. Detect the current
ANSI color and reset it when printing formatting, and then reapply it.
As an added bonus an unterminated color code is also detected,
preventing it from leaking out into the rest of the terminal.
Fixes#106633
Test output that carried color across newlines previously resulted in
the formatting around the output also being colored. Detect the current
ANSI color and reset it when printing formatting, and then reapply it.
As an added bonus an unterminated color code is also detected,
preventing it from leaking out into the rest of the terminal.
Fixes#106633
This reverts commit 5af4ba2684b9b59de3bf8135f62e05ab68cfc489.
The previous patch was missing the closing parenthesis `)` in the
`CHECK` statement in the `llvm/utils/lit/tests/shtest-glob.py` file:
`# CHECK: FAIL: shtest-glob :: glob-mkdir.txt ({{[^)]*}}`
This issue broke some build bots. This patch corrects the `CHECK`
statement by adding the closing parenthesis:
`# CHECK: FAIL: shtest-glob :: glob-mkdir.txt ({{[^)]*}})`
This patch introduces a precommit test to verify the current behavior of
glob expansion in lit's internal shell. The motivation for this test
stems from an issue encountered during the BOLT test suite when running
with the lit internal shell using the command:
`LIT_USE_INTERNAL_SHELL=1 ninja check-bolt`
During execution, the following error was observed:
```
File "/usr/local/google/home/harinidonthula/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 416, in executeBuiltinEcho
stdout.write(encode(maybeUnescape(args[-1])))
TypeError: string argument expected, got 'GlobItem'
```
The `executeBuiltinEcho` function in the lit testing framework expects a
string to be passed to `stdout.write`, but it received a `GlobItem`
object instead. This precommit test is designed to check the current
behavior where the glob pattern isn't correctly expanded, leading to
this `TypeError`.
While this patch doesn't fix the issue, it helps in understanding and
verifying the current behavior. The feedback I received from this
[PR](https://github.com/llvm/llvm-project/pull/105925) suggests using
`cmd.args = expand_glob_expressions(cmd.args, shenv.cwd)` to match the
behavior of `executeBuiltinMkdir` and `executeBuiltinRm`, but it is
recognized that the internal shell should ideally expand globs before
calling any built-in command.
**Request for Feedback:**
I'm looking for feedback on how to improve this precommit test,
specifically regarding the handling and expansion of glob patterns for
commands like mkdir and rm within the internal shell. Currently, the
args are expanded at the beginning of these functions, which should
ensure proper glob expansion. However, I'd appreciate guidance on
whether I should write additional tests to verify that mkdir and rm are
handling glob expansions correctly.
If such tests are recommended, I would also appreciate advice on the
best approach to implement them, considering the existing framework and
the way glob expansion is expected to function in the internal shell.
Should these tests confirm that the current implementation passes, or
are there specific edge cases I should be aware of?
**Next Steps:**
In my follow-up PR, I plan to address the UNRESOLVED error by expanding
the entire command, ensuring correct and consistent behavior across all
commands. The current test checks for an unresolved issue with the glob
expansion, specifically looking for a `TypeError` due to an unexpanded
`GlobItem`. This will be updated to reflect the correct behavior once
the issue is resolved.
This change is relevant for [[RFC] Enabling the Lit Internal Shell by
Default](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179/3)
This patch addresses an issue with lit's internal shell when env is
without any arguments, it fails with exit code 127 because `env`
requires a subcommand. This patch addresses the issue by encoding the
command to properly return environment variables even when no arguments
are provided.
The error occurred when running the command
` LIT_USE_INTERNAL_SHELL=1 ninja check-llvm`.
fixes: #102383
This is part of the test cleanups proposed in the RFC: [[RFC] Enabling
the Lit Internal Shell by
Default](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179)
from PEP8
(https://peps.python.org/pep-0008/#programming-recommendations):
> Comparisons to singletons like None should always be done with is or
is not, never the equality operators.
Co-authored-by: Eisuke Kawashima <e-kwsm@users.noreply.github.com>
This patch fixes the incorrect usage of lit's built-in `export` command.
There is a typo in raising the error itself where the error being raised
had the wrong number of parameters passed in.
Fixes https://github.com/llvm/llvm-project/issues/102386.
On macOS the dynamic loader prunes dyld specific environment variables
such as `DYLD_INSERT_LIBRARIES`, `DYLD_LIBRARY_PATH`, etc. If these are
set in the lit config it's safe to assume that the user actually wanted
their subprocesses to run with these variables, versus the python
interpreter that gets executed with them before they are pruned. This
change exports all known variables in the shell script instead of
relying on them being passed through.
Substitutions can be added in a couple different ways; they can be added
via the calling python scripts by adding entries to the
config.substitutions dictionary, or via DEFINE lines in the scripts
themselves.
The substitution strings passed to Python's re classes are interpreted
so that backslashes expand to escape sequences, and literal backslashes
need to be escaped.
On Unix, the script defined substitutions don't (usually, so far)
contain backslashes - but on Windows, they often do, due to paths
containing backslashes. This lead to a Windows specific escaping of
backslashes before doing Python re substitutions - since
7c9eab8fef0ed79a5911d21eb97b6b0fa9d39f82. There's nothing inherently
Windows specific about this though - any intended literal backslashes in
the substitution strings need to be escaped; this is how the Python re
API works.
The DEFINE lines were added later, and in order to cope with
backslashes, escaping of backslashes was added in the SubstDirective
class in TestRunner, applying to DEFINE lines in the tests only.
The fact that the escaping right before passing to the Python re API was
done conditionally on Windows led to two inconsistencies:
- DEFINE lines in the tests that contain backslashes got double
backslashes on Windows. (This was visible as a FIXME in
llvm/utils/lit/tests/Inputs/shtest-define/value-escaped.txt.)
- Script provided substitutions containing backslashes did not work on
Unix, but they did work on Windows.
By removing the escaping from SubstDirective and escaping it
unconditionally in the processLine function, before feeding the
substitutions to Python's re classes, we should have consistent
behaviour across platforms, and get rid of the FIXME in the lit test.
This fixes issues with substitutions containing backslashes on Unix
platforms, as encountered in PR #86649.
When a test case inside of a gtest suite fails, we report a failure
which causes the entire `ninja check-lldb` invocation to fail, even if
the outer test case is marked as XFAIL - each test case result is
reported as its own lit test run. This PR updates lit so it checks
whether each test case's parent test suite is XFAIL before setting the
status to FAIL.
This is especially problematic because the failing tests can't manually
be marked as XFAIL, due to
https://github.com/llvm/llvm-project/issues/102264.
Fixes https://github.com/llvm/llvm-project/issues/102265
### Repro instructions
1. Modify any gtest test case to generate a failure.
2. Mark the outer lit test with XFAIL using either `--xfail-tests` flag
or `LIT_XFAIL` env var.
3. Run the tests
4. Observe the lit test is XFAIL as expected, but the failed child test
cases show up as separate failures.
Co-authored-by: kendal <kendal@thebrowser.company>
When using the lit internal shell with the command:
```
LIT_USE_INTERNAL_SHELL=1 ninja check-compiler-rt
```
The follow error is encountered:
```
File "TestRunner.py", line 770, in _executeShCmd
inproc_builtin = inproc_builtins.get(args[0], None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'GlobItem'
```
This error is in a compiler-rt file:
```
TestCases/Linux/long-object-path.cpp
```
This error occurs because `args[0]` is of type `GlobItem`, which is not
hashable, leading to a `TypeError` when it is passed in
`inproc_builtins.get()`. To resolve this issue, I have updated the
implementation to ensure that `args[0]` is hashable before it is used in
`inproc_builtins`.
fixes: #102389
[link to
RFC](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179)
This patch separates the lit tests that check for the functionality of
lit's built-in cat command into its own test file and folder. This is a
prerequisite for https://github.com/llvm/llvm-project/pull/101530.
When working on very busy systems, check-offload frequently fails many
tests with this diagnostic:
```
clang: error: cannot determine amdgcn architecture: /tmp/llvm/build/bin/amdgpu-arch: Child timed out: ; consider passing it via '-march'
```
This patch accepts the environment variable
`CLANG_TOOLCHAIN_PROGRAM_TIMEOUT` to set the timeout. It also increases
the timeout from 10 to 60 seconds.
FileCheck has special handline for the `-EMPTY` suffix, that should
match empty lines. Overloading the suffix can be a source of confusion
when reading tests. Additionally, the current implementation seems to
match the following expressions, which appears to be a bug in FileCheck.
LLVM lit assumes control of the test parallelism when running a test
suite. This style of testing doesn't play nicely with build systems like
Buck or Bazel since it prefers finer grained actions on a per-test
level. In order for external build systems to control the test
parallelism, add an option to disable `.lit_test_times.txt` under the
`--skip-test-time-recording` flag, thus allowing other build systems
to determine the parallelism and avoid race conditions when writing
to that file. I went for `--skip-test-time-recording` instead of `--time-tests` in
order to preserve the original functionality of writing to `.lit_test_times.txt`
as the default behavior and only opt-in for those who do _not_ want
`.lit_test_times.txt` file.
When using -v in lit's internal implementation of cat, there is a
TypeError when the file data is passed into convertToCaretAndMNotation()
as a str, because bytearray() requires an encoded string. This patch
encodes the str before passing it through bytearray().
This reverts commit 9f6dd1f761a5121e9a69e5d1f67c2438c065b499.
Reverting to investigate buildbot failures e.g.:
"new failure on builder ppc64le-mlir-rhel-clang running on ppc64le-mlir-rhel-test"
LD_LIBRARY_PATH will become invalid when LIBPATH is also set on AIX.
See below example on AIX:
```
$ldd a.out
a.out needs:
/usr/lib/libc.a(shr.o)
Cannot find libtest.a
/unix
/usr/lib/libcrypt.a(shr.o)
$./a.out
Could not load program ./a.out:
Dependent module libtest.a could not be loaded.
Could not load module libtest.a.
System error: No such file or directory
$export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/tmp
$./a.out ; echo $?
10
$export LIBPATH=./
$./a.out ; echo $? >>>>>> Now LD_LIBRARY_PATH is not used by system loader
Could not load program ./a.out:
Dependent module libtest.a could not be loaded.
Could not load module libtest.a.
System error: No such file or directory
```
This breaks many AIX LIT cases on our downstream buildbots which sets
LIBPATH.
---------
Co-authored-by: Anh Tuyen Tran <34661776+anhtuyenibm@users.noreply.github.com>
Co-authored-by: David Tenty <daltenty.dev@gmail.com>
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.
* Ensure that every target is in a folder
* Use a folder hierarchy with each LLVM subproject as a top-level folder
* Use consistent folder names between subprojects
* When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
To support auto-conversion on z/OS text files need to be opened as text files. These changes will fix a number of LIT failures due to text files not being converted to the internal code page.
update a number of tools so they open the text files as text files
add support in the cat.py to open a text file as a text file (Windows will continue to treat all files as binary so new lines are handled correctly)
add env var definitions to enable auto-conversion in the lit config file.
This can happen when you do not choose a specific target triple, and do
not enable the host architecture when building (if you do enable it, it
would become the default target). Such as only enabling RISC-V, when
building on an AArch64 machine.
Originally reported
https://discourse.llvm.org/t/llvm-test-error-could-not-turn-into-itanium-abi-triple/76013.
When attempting to run a single test via lit you get: `Could not turn ''
into Itanium ABI triple`
Setting a default triple with `LLVM_DEFAULT_TARGET_TRIPLE` works around
the issue.
This change copies the existing host triple check to target triple, and
adds a note to highlight the potential issue. As `check-clang` on my
AArch64 machine failed 32% of tests in this configuration.
Which is to be expected and is ok if you only want to run specific
tests, but for anyone unintentionally building this way the note is a
clue to the cause.
There are a few test that check access permissions, so they need to be
disabled when running the tests as root.
The most common use case for running tests as root is inside of a
container. GitHub Actions, for example, only supports running the root
user inside of containers, so this change is necessary in order to run
the tests inside of a container running in the GitHub Actions
environment.
My #73541 added lines to
`llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt` so what
was previously on line 7 is now on line 12.
Before:
28412d1800/llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt (L7-L8)
After:
6fb7c2d713/llvm/utils/lit/tests/Inputs/testrunner-custom-parsers/test.txt (L12-L13)
This didn't show up in the PR checks, but caused a buildbot failure
after merging, https://lab.llvm.org/buildbot/#/builders/139/builds/54597
:
```
# | ======================================================================
# | FAIL: test_commands (__main__.TestIntegratedTestKeywordParser)
# | ----------------------------------------------------------------------
# | Traceback (most recent call last):
# | File "/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/utils/lit/tests/unit/TestRunner.py", line 135, in test_commands
# | self.assertEqual(value[1].command.strip(), "%dbg(MY_RUN: at line 7) foo bar")
# | AssertionError: '%dbg(MY_RUN: at line 12) foo bar' != '%dbg(MY_RUN: at line 7) foo bar'
# | - %dbg(MY_RUN: at line 12) foo bar
# | ? ^^
# | + %dbg(MY_RUN: at line 7) foo bar
# | ? ^
```
Apologies for the build break 🙀
The const.cpp testcase fails when running in MSVC mode, while it does
succeed in MinGW mode.
In MSVC mode, there are more constructor invocations than expected, as
the printout looks like this:
A(1), this = 0000025597930000
A(1), this = 0000025597930000
f: this = 0000025597930000, val = 1
A(1), this = 0000025597930000
f: this = 0000025597930000, val = 1
~A, this = 0000025597930000, val = 1
~A, this = 0000025597930000, val = 1
~A, this = 0000025597930000, val = 1
While the expected printout looks like this:
A(1), this = 000002C903E10000
f: this = 000002C903E10000, val = 1
f: this = 000002C903E10000, val = 1
~A, this = 000002C903E10000, val = 1
Reapplying #70991 with the XFAIL changed to check the host triple, not
the target triple. On an MSVC based build of Clang, but with the default
target triple set to PS4/PS5, we will still see the failure. And a Linux
based build of Clang that targets PS4/PS5 won't see the issue.
This will allow for configuring tests according to AIX version.
Reviewed By: daltenty, #libc, Mordante
Differential Revision: https://reviews.llvm.org/D149660
This patch adds "nice-to-have" feature in lit.
it prints the total number of discovered tests at the beginning. It is
covenient to see the total number of tests and avoid scrolling up to the
beginning of log.
Further, this patch also prints %ge of tests.
This patch fixes tests pointed by previous attempt of landing this
patch.
Reviewed By: RoboTux, jdenny-ornl
Co-authored-by: Madhur A <madhura@nvidia.com>
Without this patch, the functions `executeScriptInternal` and thus
`runOnce` in `llvm/utils/lit/lit/TestRunner.py` return either a tuple like
`(out, err, exitCode, timeoutInfo)` or a `lit.Test.Result` object. They
return the latter only when there's a lit internal shell parse error in
a RUN line. In my opinion, a more straight-forward way to handle
exceptional cases like that is to use python exceptions.
For that purpose, this patch introduces `ScriptFatal`. Thus, this patch
changes `executeScriptInternal` to always either return the tuple or
raise the `ScriptFatal` exception. It updates `runOnce` and
`libcxx/utils/libcxx/test/format.py` to catch the exception rather than
check for the special return type.
This patch also changes `runOnce` to convert the exception to a
`Test.UNRESOLVED` result instead of `TEST.FAIL`. The former is the
proper result for such a malformed test, for which a rerun (given an
`ALLOW_RETRIES:`) serves no purpose. There are at least two benefits
from this change. First, `_runShTest` no longer has to specially and
cryptically handle this case to avoid unnecessary reruns. Second, an
`XFAIL:` directive no longer hides such a failure [as we saw
previously](https://reviews.llvm.org/D154987#4501125).
To facilitate the `_runShTest` change, this patch inserts the internal
shell parse error diagnostic into the format of the test's normal debug
output rather than suppressing the latter entirely. That change is also
important for [D154987](https://reviews.llvm.org/D154987), which
proposes to reuse `ScriptFatal` for python compile errors in PYTHON
lines or in `config.prologue`. In that case, the diagnostic might follow
debugging output from the test's previous RUN or PYTHON lines, so
suppressing the normal debug output would lose information.
This patch adds "nice-to-have" feature in lit.
it prints the total number of discovered tests at the beginning. It is
covenient to see the total number of tests and avoid scrolling up to the
beginning of log.
Further, this patch also prints %ge of tests.
Reviewed By: RoboTux, jdenny-ornl
Co-authored-by: Madhur A <madhura@nvidia.com>
I could probably break this commit into more pieces.
---
This patch adds libc++ support for Android L (Android 5.0+) and up,
tested using the Android team's current compiler, a recent version of
the AOSP sysroot, and the x86[-64] Android Emulator.
CMake and Lit Configuration:
Add runtimes/cmake/android/Arch-${ARCH}.cmake files that configure CMake
to cross-compile to Android without using CMake's built-in NDK support
(which only works with an actual packaged NDK).
Add libcxx/cmake/caches/AndroidNDK.cmake that builds and tests libc++
(and libc++abi) for Android. This file configures libc++ to match what
the NDK distributes, e.g.:
- libc++_shared.so (includes libc++abi objects, there is no
libc++abi.so). libunwind is linked statically but not exported.
- libc++_static.a (does not include libc++abi) and libc++abi.a
- `std::__ndk1` namespace
- All the libraries are built with `__ANDROID_API__=21`, even when they
are linked to something targeting a higher API level.
(However, when the Android LLVM team builds these components, they do
not use these CMake cache files. Instead they use Python scripts to
configure the builds. See
https://android.googlesource.com/toolchain/llvm_android/.)
Add llvm-libc++[abi].android-ndk.cfg.in files that test the Android
NDK's libc++_shared.so. These files can target old or new Android
devices. The Android LLVM team uses these test files to test libc++ for
both arm/arm64 and x86/x86_64 architectures.
The Android testing mode works by setting %{executor} to adb_run.py,
which uses `adb push` and `adb shell` to run tests remotely. adb_run.py
always runs tests as the "shell" user even on an old emulator where "adb
unroot" doesn't work. The script has workarounds for old Android
devices. The script uses a Unix domain socket on the host
(--job-limit-socket) to restrict concurrent adb invocations. Compiling
the tests is a major part of libc++ testing run-time, so it's desirable
to exploit all the host cores without overburdening the test devices,
which can have far fewer cores.
BuildKite CI:
Add a builder to run-buildbot, `android-ndk-*`, that uses Android Clang
and an Android sysroot to build libc++, then starts an Android emulator
container to run tests.
Run the emulator and an adb server in a separate Docker container
(libcxx-ci-android-emulator), and create a separate Docker image for
each emulator OS system image. Set ADB_SERVER_SOCKET to connect to the
container's adb server. Running the only adb server inside the container
makes cleanup more reliable between test runs, e.g. the adb client
doesn't create a `~/.android` directory and the adb server can be
restarted along with the emulator using docker stop/run. (N.B. The
emulator insists on connecting to an adb server and will start one
itself if it can't connect to one.)
The suffix to the android-ndk-* job is a label that concisely specifies
an Android SDK emulator image. e.g.:
- "system-images;android-21;default;x86" ==> 21-def-x86
- "system-images;android-33;google_apis;x86_64" ==> 33-goog-x86_64
Fixes: https://github.com/llvm/llvm-project/issues/69270
Differential Revision: https://reviews.llvm.org/D139147
This patch addresses the missed review comment from PR #67063. It
renames LIT flag "--disable-gtest-sharding" to "--no-gtest-sharding"
and corrects the code style issue.
[unittest] Add lit option to allow disabling sharding in unittest
By default, googletest based unit tests uses sharding to speed up the
testing. However, when these unit tests are run through wrapper program
on a remote platform with large round trip time, the sharding will increase
the time cost dramatically. This patch adds a LLVM LIT option
"--disable-gtest-sharding" to allow sharding on gtest based unittest to
be disabled.
In PR #65242 (landed as 9e739fdb85ac672f3e25e971d96e71823e07ebda), I
claimed that RUN lines cannot contain newlines. Actually, they can after
substitution expansion. More generally, a lit config file can define
substitutions or preamble commands containing newlines. While both of
those cases seem unlikely in practice,
[D154987](https://reviews.llvm.org/D154987) proposes PYTHON directives
where it seems very likely.
Regardless of the use case, without this patch, such newlines break
expansion of `%dbg(RUN: at line N)`, and the fix is simple.
