This reverts commit 9f1a9d24813c0c1b89d24063825a7fb8ca64a3a1.
This caused quite a few buildbot failures (see failures on the PR). It
does not look like a trivial fix to me.
### Problem
When running `lit` with `-j1` in a sandboxed environment (e.g., [Cursor
Sandbox Mode](https://cursor.com/blog/enterprise#sandbox-mode)), `lit`
crashes with `PermissionError: [Errno 1] Operation not permitted`
because it still attempts to create a `multiprocessing.Pool`, which
requires POSIX semaphores (`/dev/shm/sem.*`) that are blocked by the
sandbox.
**Reproducer:** Run any lit test suite with `-j1` in a [restricted
sandbox](https://cursor.com/blog/enterprise#sandbox-mode):
`LIT_OPTS="-j1 -v" ninja check-lld`
**Error trace:**
https://gist.github.com/alx32/0dc6abb45c40cf5753669e4c6cce929d
### Solution
When `workers == 1`, skip `multiprocessing.Pool` entirely and run tests
sequentially in the main process. This avoids creating POSIX semaphores
that fail in sandboxed environments.
To properly support `--max-time` in single-process mode (where we can't
terminate workers mid-test), we temporarily set `maxIndividualTestTime`
to the remaining time before the deadline. If a test times out due to
hitting the deadline, it's marked as SKIPPED to match multiprocessing
behavior.
### Testing
Added `single-process.py` test that uses strace to verify:
- With `-j1`: No `/dev/shm/sem.*` POSIX semaphores are created
- With `-j2`: Semaphores ARE created (validates the test won't XPASS)
### Historical Context
Lit previously had a "single process mode" that was:
- **Added** in Sept 2017 (42b6dcbcef4f) for debugging
- **Enhanced** in Feb 2019 (96adb78b120b) to auto-enable for `-j1`
- **Removed** in Oct 2019 (f3c329986cf4) as it "did not carry its
weight"
The original motivation was performance/debugging. The new motivation is
**compatibility with sandboxed execution environments** where
multiprocessing primitives are unavailable.
[Assisted-by](https://t.ly/Dkjjk): Cursor IDE + claude-opus-4.5-high +
gpt-5.2-xhigh
This test invokes lit in a directory and then tries to overwrite
fail.txt within that directory. With the project sources mounted as
read-only, fail.txt ends up being marked readonly, which causes cp to
fail without -f. Use cp -f to ensure we overwrite the existing fail.txt.
We left an error message for users of %T through the LLVM 22 branch
point to hopefully make it easier for anyone migrating versions rather
than having the checks silently fail with a %T ending up in the executed
command. Now that 22 has branched, we can remove this check as we all
downstream users should have seen the error by this point.
This is the behavior of the main not binary that was not preserved in
the internal shell. Make it so that the builtin not command does
actually fail if we end up with a signal rather than just a non-zero
exit code.
Reviewers: petrhosek, ilovepi, jdenny-ornl, arichardson
Pull Request: https://github.com/llvm/llvm-project/pull/174298
FreeBSD does not support using ulimit to grow up max file number per
process. This characteristic is inherited by Darwin and thus we pass
this test on FreeBSD as well.
This reverts commit 3847648e84d2ff5194f605a8a9a5c0a5e5174939.
Relands https://github.com/llvm/llvm-project/pull/158043 which got
auto-merged on a revision which wasn't approved.
The only addition to the approved version was that we adjust how we set
the time for failed tests. We used to just assign it the negative value
of the elapsed time. But if the test failed with `0` seconds (which some
of the new tests do), we would mark it `-0`. But the check for whether
something failed checks for `time < 0`. That messed with the new
`--filter-failed` option of this PR. This was only an issue on Windows
CI, but presumably can happen on any platform. Happy to do this in a
separate PR.
---- Original PR
This patch adds a new --filter-failed option to llvm-lit, which when
set, will only run the tests that have previously failed.
Reverts llvm/llvm-project#158043
This was approved for earlier revisions but the tests were failing on
Windows. I pushed a speculative fix and that fixed the CI, which caused
auto-merge to merge the PR. But I'd like to have approval for the latest
revision. So reverting for now and resubmitting a new PR
To be able to test lit without having a configuration of LLVM, we need
to support invocations that are not going through the lit.site.cfg and
thus don't have a llvm_config set-up.
These helpers, which handle the difference between Python 2.x and Python
3.x, are no longer required.
Co-authored-by: Alexander Richardson <mail@alexrichardson.me>
Python multiprocessing is limited to 60 workers at most:
6bc65c30ff/Lib/concurrent/futures/process.py (L669-L672)
The limit being per thread pool, we can work around it by using multiple
pools on windows when we want to actually use more workers.
This patch makes it possible to detect in LLDB shell and API tests if
`-fbounds-safety` is supported by the compiler used for testing. The
motivation behind this is to allow upstreaming
https://github.com/swiftlang/llvm-project/pull/11835 but with the tests
disabled in upstream because the full implementation of -fbounds-safety
isn't available in Clang yet.
For shell tests when -fbounds-safety is available the
`clang-bounds-safety` feature is available which means tests can be
annotated with `# REQUIRES: clang-bounds-safety`.
API tests that need -fbounds-safety support in the compiler can use the
new `@skipUnlessBoundsSafety` decorator.
rdar://165225507
I'm not aware of any way for `%run` wrapper scripts like
`iosssim_run.py`
([ref](d2c7c60642/compiler-rt/test/sanitizer_common/ios_commands/iossim_run.py (L4)))
to know what testcase they are currently running. This can be useful if
these wrappers need to create a (potentially remote) temporary directory
for each test case.
This adds the `LIT_CURRENT_TESTCASE` environment variable to both the
internal shell and the external shell, containing the full name of the
current test being run.
Solaris doesn't define RLIMIT_NPROC, so this is expected to fail there.
This fixes a test failure in llvm/utils/lit/tests/verbosity.py on
Solaris due to this unexpected warning being included in the lit output.
With this change DiffUpdater can update expected files even if both
files are created by split-files, if one of them ends with ".expected".
This is useful when a file is created and then modified during the test.
The rate at which lit's progress bar progresses is not deterministic.
Don't try to match the contents of the progress bar, only the fact that
it's there.
Lit has a number of options controlling the output, but they don't
compose very well. This breaks the existing options down into smaller,
orthogonal options, and makes the existing options aliases of the new
ones.
This introduces the following options:
--test-output {off,failed,all}
--print-result-after {off,failed,all}
--diagnostic-level {error,warning,note}
--terse-summary
--no-terse-summary
--progress-bar (mirroring --no-progress-bar)
--test-output and --print-result-after are not entirely orthogonal, as
'--test-output X' requires that --print-result-after is set to at least
X, and implicitly does so if it isn't already. Conversely,
'--print-result-after Y' requires that --test-output is at most Y, and
implicitly lowers if it is higher. This means that the following
invocations have different end results, as they are applied in order:
'--test-output all --print-result-after off'
'--print-result-after off --test-output all'
The following existing options are now aliases as follows:
-q, --quiet
'--diagnostic-level error --test-output off --terse-summary'
-s, --succinct
'--progress-bar --print-result-after failed'
-v, --verbose
'--test-output failed'
-a, --show-all
'--test-output all'
These where all completely separate options and would override each
other in ad-hoc ways, with no regard to the order they were given.
This fixes https://github.com/llvm/llvm-project/issues/106643 This is
based on the RFC
https://discourse.llvm.org/t/rfc-new-command-line-options-for-controlling-llvm-lit-output/
with the addition of --terse-summary, which was a behaviour of -q that
was not captured by the original RFC. This also diverges from the RFC in
that --debug is NOT folded into --diagnostic-level, because it can be
useful to debug any configuration, including those specifying
--diagnostic-level.
Example combination that is possible now but wasn't before:
'--diagnostic-level error --test-output all --progress-bar' Another use
case is aliases, where you can alias e.g:
alias lit=llvm-lit --quiet
but still override the specified default options.
`lit.util.mkdir` and `lit.util.mkdir_p` were written during the Python
2.x era.
Since modern `pathlib` functions have similar functionality, we can
simply use those instead.
If you encounter a path length issue after this change, the registry
value `LongPathsEnabled` must be set as described in
https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
. Note that the Python runtime is already marked as a `longPathAware`
executable.
**Background:**
On Cygwin, a file named `file_name.exe` can be accessed without the
suffix, simply as `file_name`, as shown below:
```
$ echo > file_name.exe
$ file file_name.exe
file_name.exe: very short file (no magic)
$ file file_name
file_name: very short file (no magic)
```
In this situation, while running `mkdir file_name` works as intended,
checking for the existence of the target before calling `mkdir`
incorrectly reports that it already exists and thus skips the directory
creation.
```
$ test -e file_name && echo exists
exists
$ mkdir file_name && echo ok
ok
$ file file_name
file_name: directory
```
Therefore, the existence pre-check should be skipped on Cygwin. Instead
of add a workaround, refactored them.
LLVM now requires Python >= 3.8, and ConfigParser was renamed to
configparser in 3.0:
https://docs.python.org/3/whatsnew/3.0.html#library-changes
A few places imported it under the Python2 name even for Python3, I have
swapped those to the Python3 name.
This was reported by https://pypi.org/project/vermin/ as the file having
incompatible versions. Since once import is 2.x and one is 3.x.
160058fc19a9bcb70feb442a755229838b4dbc7a broke the Solaris bots because
they do not support RLIMIT_FSIZE despite it being in POSIX 2004. Disable
it there for now as the loss of test coverage should not be significant.
This enables the use of readfile substitutions for populating
environment variables. This is necessary in some compiler-rt tests.
Reviewers: pawosm-arm
Reviewed By: pawosm-arm
Pull Request: https://github.com/llvm/llvm-project/pull/165140
This fails on MacOS because setting it to unlimited there just sets the
limit to the max value which causes differences that show up in the
check lines.
This was supposed to be in 6ccd1e8626f331f2ec2b172c3e7e8fffee66ac95 but
got left out because I forgot to save the file inside of VSCode. This
was causing test failures on MacOS due to the previously mentioned
failures setting ulimit that caused the patch to be reverted in the
first place.
https://lab.llvm.org/buildbot/#/builders/190/builds/29990
This reverts commit 57722ddce172f569f04a50b76ccb2fc524adf8f5.
This caused some MacOS test failures due to resource there having issues
with RLIMIT_STACK. The underlying syscall fails with EINVAL despite
the values being correct. For now, move this to the non Darwin test.
There was a bug in llvm-lit related to setting PATH using env in the
internal shell.
The new PATH wasn't used when looking up the command to be executed. So
when doing things like this in a test case
RUN: mkdir %t
RUN: env PATH=%t program ...
the internal shell would search for "program" using the orignal PATH and
not the PATH set by env when preceeding the command.
It seems like this was a simple mistake in commit 57782eff31e9d454,
since the logic to pick a PATH from the cmd_shenv instead of shenv
actually was added in that patch, but the resulting path wasn't used.
These are the other options used in compiler-rt that we also need to
support.
Reviewers: arichardson, petrhosek, ilovepi
Reviewed By: ilovepi, arichardson
Pull Request: https://github.com/llvm/llvm-project/pull/165122
The recently added ulimit_reset.txt section in shtest-ulimit.py was
failing on some builders if the default file descriptor limit started
with 50. This patch fixes that by explicitly checking that the file
descriptor limit is equal to the default value.
When constructing a container in a default argument in Python, we
actually end up with a reference to the same default container for every
invocation of the function. Given this happened with the ulimit variable
in ShellEnvironment, we ended up persisting limits across tests. This
would cause some LLVM tests to fail, particularly jitlink and dsymutil
tests, if they ended up running after the one clang test that uses
ulimit -v to set a maximum amount of memory that can be allocated. This
patch fixes that behavior by constructing the dict inside the function
to ensure we get a new instance and adds test coverage.
This pull request makes a minor fix to the
`llvm/utils/lit/lit/llvm/config.py` file.
The change corrects a missing comma in the list of environment
variables, ensuring that `"UBSAN_SYMBOLIZER_PATH"` and `"ASAN_OPTIONS"`
are treated as separate entries instead of a single concatenated string.
* Fixed a missing comma between `"UBSAN_SYMBOLIZER_PATH"` and
`"ASAN_OPTIONS"` in the environment variable list in
`llvm/utils/lit/lit/llvm/config.py`, preventing potential configuration
issues.
This patch removes support for %T from llvm-lit. For now we mark the
test unresolved and add an error message noting the substitution is
deprecated. This is exactly the same as the error handling for other
substitution failures. We intend to remove support for the nice error
message once 22 branches as users should have moved over by the they are
upgrading to v23.
Reviewers: petrhosek, jh7370, ilovepi, pogo59, cmtice
Reviewed By: cmtice, jh7370, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/160028
This patch removes %T from tests in preparation for removing the
substitution completely. Splitting this off from the actual removal to
make it easier to revert the removal of %T in case I missed something.
Reviewers: RKSimon, jh7370, cmtice, petrhosek, ilovepi, pogo59
Reviewed By: cmtice, RKSimon, jh7370
Pull Request: https://github.com/llvm/llvm-project/pull/160027
This patch allows for two retry attempts for the readfile tests. This is
intended as a stop-gap until I have time to do proper investigation into
why exactly the tests are failing.
Python 3.13 considers "/file/does/not/exist" to not be an absolute path
on Windows, so the test runner does os.path.join(cwd, filePath), which
can end up with an output path such as "D:/file/does/not/exist".
Accept a potential prefix before the missing path here.
This fixes running the lit tests on Windows with Python 3.13.
This PR fixes a bunch of failures on AIX that occurred due to the switch
to lit internal shell by default. The failures deal with the following:
1. unset not being supported by lit internal shell
2. A bug with echo -n on AIX when there are multiple pipes in the RUN
command
3. ulimit test case not supported on AIX due to the -v option
4. platform specific error message for missing file
Before this change, rm would assume that a symlink to a directory was
actually a directory and require the recursive flag to be passed,
differing from other shells. Given the change in lit is about the same
length as the test change would be (minus tests), I think it makes sense
to just support this in the internal shell.
Reviewers: cmtice, petrhosek, ilovepi
Reviewed By: petrhosek, cmtice, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/158464
cat with no files passed to it is supposed to read from STDIN according
to POSIX. The builtin cat lacking this behavior led to the clang test in
dev-fd-fs.c to fail because it expected this behavior. This is a simple
modification and I do not think it is possible to rewrite the test
without this easily while preserving the semantics around named pipes.
Reviewers: petrhosek, arichardson, ilovepi, cmtice, jh7370
Reviewed By: jh7370, arichardson, ilovepi, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/158447
This patch adds support for the new lit %{readfile:<filename>}
substitution to the external shell. The implementation currently just
appends some test commands to ensure the file exists and uses a subshell
with cat. This is intended to enable running tests using the
substitution in the external shell before we fully switch over to the
internal shell.
This code is designed to be temporary with us deleting it once
everything has migrated over to the internal shell and we are able to
remove the external shell code paths.
Reviewers: petrhosek, cmtice, pogo59, ilovepi, arichardson
Reviewed By: cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/159431
