Noticed that the current
[docs](https://llvm.org/docs/TestingGuide.html#unit-and-regression-tests)
recommend using LIT_ARGS (config option in cmake script), rather than
LIT_OPTS (environment variable interpreted by lit directly). While
LIT_ARGS doesn't do anything at all when set with make, at least not in
a way outlined in the docs.
As someone on Discord was understandably confused because the build
directory does contain folder structures that look remarkably like the
source directory.
I used this page to explain it but realised that this must be from when
llvm was a separate repository. So `<user home>/llvm` probably was a
common path.
Now it's in llvm-project. So make that obvious in the instructions.
Historically, we've not automatically enforced how git tracks line
endings, but there are many, many commits that "undo" unintended CRLFs
getting into history.
`git log --pretty=oneline --grep=CRLF` shows nearly 100 commits
involving reverts of CRLF making its way into the index and then
history. As far as I can tell, there are none the other way round except
for specific cases like `.bat` files or tests for parsers that need to
accept such sequences.
Of note, one of the earliest of those listed in that output is:
```
commit 9795860250734e5c2a879546c534e35d9edd5944
Author: NAKAMURA Takumi <geek4civic@gmail.com>
Date: Thu Feb 3 11:41:27 2011 +0000
cmake/*: Add svn:eol-style=native and fix CRLF.
llvm-svn: 124793
```
...which introduced such a defacto policy for subversion.
With old versions of git, it's been a bit of a crap-shoot whether
enforcing storing line endings in the history will upset checkouts on
machines where such line endings are the norm. Indeed many users have
enforced that git checks out the working copy according to a global or
per-user config via core crlf, or core autocrlf.
For ~8 years now[1], however, git has supported the ability to "do as
the Romans do" on checkout, but internally store subsets of text files
with line-endings specified via a system of patterns in the
`.gitattributes` file. Since we now have this ability, and we've been
specifying attributes for various binary files, I think it makes sense
to rid us of all that work converting things "back", and just let git
handle the local checkout. Thus the new toplevel policy here is
* text=auto
In simple terms this means "unless otherwise specified, convert all
files considered "text" files to LF in the project history, but check
them out as expected on the local machine. What is "expected on the
local machine" is dependent on configuration and default.
For those files in the repository that *do* need CRLF endings, I've
adopted a policy of `eol=crlf` which means that git will store them in
history with LF, but regardless of user config, they'll be checked out
in tree with CRLF.
Finally, existing files have been "corrected" in history via `git add
--renormalize .`
End users should *not* need to adjust their local git config or
workflow.
[1]: git 2.10 was released with fixed support for fine-grained
line-ending tracking that respects user-config *and* repo policy. This
can be considered the point at which git will respect both the user's
local working tree preference *and* the history as specified by the
maintainers. See
https://github.com/git/git/blob/master/Documentation/RelNotes/2.10.0.txt#L248
for the release note.
Substitutions can be added in a couple different ways; they can be added
via the calling python scripts by adding entries to the
config.substitutions dictionary, or via DEFINE lines in the scripts
themselves.
The substitution strings passed to Python's re classes are interpreted
so that backslashes expand to escape sequences, and literal backslashes
need to be escaped.
On Unix, the script defined substitutions don't (usually, so far)
contain backslashes - but on Windows, they often do, due to paths
containing backslashes. This lead to a Windows specific escaping of
backslashes before doing Python re substitutions - since
7c9eab8fef0ed79a5911d21eb97b6b0fa9d39f82. There's nothing inherently
Windows specific about this though - any intended literal backslashes in
the substitution strings need to be escaped; this is how the Python re
API works.
The DEFINE lines were added later, and in order to cope with
backslashes, escaping of backslashes was added in the SubstDirective
class in TestRunner, applying to DEFINE lines in the tests only.
The fact that the escaping right before passing to the Python re API was
done conditionally on Windows led to two inconsistencies:
- DEFINE lines in the tests that contain backslashes got double
backslashes on Windows. (This was visible as a FIXME in
llvm/utils/lit/tests/Inputs/shtest-define/value-escaped.txt.)
- Script provided substitutions containing backslashes did not work on
Unix, but they did work on Windows.
By removing the escaping from SubstDirective and escaping it
unconditionally in the processLine function, before feeding the
substitutions to Python's re classes, we should have consistent
behaviour across platforms, and get rid of the FIXME in the lit test.
This fixes issues with substitutions containing backslashes on Unix
platforms, as encountered in PR #86649.
This reverts commit b6824c9d459da059e247a60c1ebd1aeb580dacc2.
This relands commit 0a6c74e21cc6750c843310ab35b47763cddaaf32.
The original commit was reverted due to buildbot failures. These bots
should be updated now, so hopefully this will stick.
Generally, IR and assembly test files benefit from being cleaned to
remove unnecessary details. However, for tests requiring elaborate
IR or assembly files where cleanup is less practical (e.g., large amount
of debug information output from Clang), the current practice is to
include the C/C++ source file and the generation instructions as
comments.
This is inconvenient when regeneration is needed. This patch adds
`llvm/utils/update_test_body.py` to allow easier regeneration.
`ld.lld --debug-names` tests (#86508) utilize this script for
Clang-generated assembly tests.
Note: `-o pipefail` is standard (since
https://www.austingroupbugs.net/view.php?id=789) but not supported by
dash.
Link:
https://discourse.llvm.org/t/utility-to-generate-elaborated-assembly-ir-tests/78408
This reverts commit f2931182fc877e813974a5f53721f859bfb5b130.
This was again causing buildbot failures. #83962 has been updated with the new
failures, notifying the buildbot maintainers that they need to update their
bots.
This reverts commit 2dfa30d0ca6fb6991640a18e53401d82f567f8ff.
This relands commit 0a6c74e21cc6750c843310ab35b47763cddaaf32.
This patch originally caused a host of buildbot failures due to several
buildbots not having the necessary python version. That was tracked in
issue #83962, and all bots that failed in the first round have now been
updated. This is an attempt to reland the patch to see if it sticks or
if there are a number of long-running bots where this patch will cause
failures.
This reverts commit 0a6c74e21cc6750c843310ab35b47763cddaaf32.
This created a lot of post-commit failures due to buildbots running
older versions of Python.
As per the RFC
https://discourse.llvm.org/t/rfc-upgrading-llvms-minimum-required-python-version/67571,
raise the minimum Python version to ensure that the Python syntax
doesn't become overly obsolete, to enable new Python feature usage,
and to improve the maintainability of CI.
One of the primary use cases for this higher Python version is to enable
python type annotations that are more aligned with current Python
best practices. This is not only important for our own internal Python
for testing, but for the Python bindings that are exposed to users.
Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths short and avoid hitting MAX_PATH limitations.
Changes lit logic to:
Use `os.path.abspath` on Windows, where `MAX_PATH` is a concern that we
can work around using substitute drives, which `os.path.realpath` would
resolve.
Use `os.path.realpath` on Unix, where the current directory always has
symlinks resolved, so it is impossible to preserve symlinks in the
presence of relative paths, and so we must make sure that all code paths
use real paths.
Also updates clang's `FileManager::getCanonicalName` and `ExtractAPI`
code to avoid resolving substitute drives (i.e. resolving to a path
under a different root).
How tested: built with `-DLLVM_ENABLE_PROJECTS=clang` and built `check-all` on both Windows
Differential Revision: https://reviews.llvm.org/D154130
Reviewed By: @benlangmuir
Patch by Tristan Labelle <tristan@thebrowser.company>!
There are a lot of conventions for writing tests that don't seem
to be documented anywhere right now, so this takes a stab at
writing down some "best practices".
Differential Revision: https://reviews.llvm.org/D142441
This patch fixes two incorrect descriptions in TestingGuide.rst.
1. test/lit.site.cfg --> test/lit.site.cfg.py
After https://reviews.llvm.org/D37838 , the `test/lit.site.cfg` had been added a .py extension.
So it should be `test/lit.site.cfg.py`.
2. $(LLVM_OBJ_ROOT)/$(BuildMode)/bin --> $(LLVM_OBJ_ROOT)/bin
The current build system doesn't create a $(BuildMode) directory any more.
So it should be removed.
Reviewed By: mehdi_amini, MaskRay
Differential Revision: https://reviews.llvm.org/D140780
These directives define per-test lit substitutions. The concept was
discussed at
<https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10>.
For example, the following directives can be inserted into a test file
to define `%{cflags}` and `%{fcflags}` substitutions with empty
initial values, which serve as the parameters of another newly defined
`%{check}` substitution:
```
// DEFINE: %{cflags} =
// DEFINE: %{fcflags} =
// DEFINE: %{check} = %clang_cc1 %{cflags} -emit-llvm -o - %s | \
// DEFINE: FileCheck %{fcflags} %s
```
The following directives then redefine the parameters before each use
of `%{check}`:
```
// REDEFINE: %{cflags} = -foo
// REDEFINE: %{fcflags} = -check-prefix=FOO
// RUN: %{check}
// REDEFINE: %{cflags} = -bar
// REDEFINE: %{fcflags} = -check-prefix=BAR
// RUN: %{check}
```
Of course, `%{check}` would typically be more elaborate, increasing
the benefit of the reuse.
One issue is that the strings `DEFINE:` and `REDEFINE:` already appear
in 5 tests. This patch adjusts those tests not to use those strings.
Our prediction is that, in the vast majority of cases, if a test
author mistakenly uses one of those strings for another purpose, the
text appearing after the string will not happen to have the syntax
required for these directives. Thus, the test author will discover
the mistake immediately when lit reports the syntax error.
This patch also expands the documentation on existing lit substitution
behavior.
Reviewed By: jhenderson, MaskRay, awarzynski
Differential Revision: https://reviews.llvm.org/D132513
This syntax allows to modify RUN lines based on features
available. For example:
RUN: ... | FileCheck %s --check-prefix=%if windows %{CHECK-W%} %else %{CHECK-NON-W%}
CHECK-W: ...
CHECK-NON-W: ...
The whole command can be put under %if ... %else:
RUN: %if tool_available %{ %tool %} %else %{ true %}
or:
RUN: %if tool_available %{ %tool %}
If tool_available feature is missing, we'll have an empty command in
this RUN line. LIT used to emit an error for empty commands, but now
it treats such commands as nop in all cases.
Multi-line expressions are also supported:
RUN: %if tool_available %{ \
RUN: %tool \
RUN: %} %else %{ \
RUN: true \
RUN: %}
Background and motivation:
D121727 [NVPTX] Integrate ptxas to LIT tests
https://reviews.llvm.org/D121727
Differential Revision: https://reviews.llvm.org/D122569
This adds 2 new lit helpers `%{fs-src-root}` and `%{fs-sep}`, these
allow writing tests that correctly handle slashes on Windows. In the
case of tests like clang/test/CodeGen/debug-prefix-map.c, these are
unable to correctly test behavior on both platforms, unless they fork
and add OS requirements, because the relevant logic hits host specific
codepaths like checking if paths are absolute.
Differential Revision: https://reviews.llvm.org/D111457
`opt -analyze` is legacy PM-specific. Show better ways of doing the same
thing, generally with some sort of `-passes=print<foo>`.
Reviewed By: asbirlea
Differential Revision: https://reviews.llvm.org/D119486
This patch document llvm/utils/update_* python scripts that are used to generate
assertions in many of the LLVM regression test cases.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D112936
This reverts commit 3229c971512404c512e041c3e88f22dbec2b650b.
With a2acac6 in place this should provide enough info to work out
any repeat of the failure in cross_ovver_uniform_dist.test.
Currently the UNSUPPORTED and XFAIL clauses support specifying
substrings of the target triple; but REQUIRES does not, which can trip
people up or lead to hacking config files to insert substitute feature
names. Consistency across all three lit clauses seems preferable.
Differential Revision: https://reviews.llvm.org/D107162
This patch augments Lit with the ability to parse regular expressions
in boolean expressions. This includes REQUIRES:, XFAIL:, UNSUPPORTED:,
and all other special Lit markup that evaluates to a boolean expression.
Regular expressions can be specified by enclosing them in {{...}},
similarly to how FileCheck handles such regular expressions. The regular
expression can either be on its own, or it can be part of an identifier.
For example, a match expression like {{.+}}-apple-darwin{{.+}} would match
the following variables:
x86_64-apple-darwin20.0
arm64-apple-darwin20.0
arm64-apple-darwin22.0
etc...
In the long term, this could be used to remove the need to handle the
target triple specially when parsing boolean expressions.
Differential Revision: https://reviews.llvm.org/D104572
The documentation link of Google Test on GitHub have been moved to the
top-level docs directory.
Thus, the original link is invalid now.
Reviewed By: Pavel Labath
Differential Revision: https://reviews.llvm.org/D99559
On z/OS, other error messages are not matched correctly in lit tests.
```
EDC5121I Invalid argument.
EDC5111I Permission denied.
```
This patch adds a lit substitution to fix it.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D95808
On z/OS, the following error message is not matched correctly in lit tests.
```
EDC5129I No such file or directory.
```
This patch uses a lit config substitution to check for platform specific error messages.
Reviewed By: muiez, jhenderson
Differential Revision: https://reviews.llvm.org/D95246
See https://lists.llvm.org/pipermail/llvm-dev/2020-July/143373.html
"[llvm-dev] Multiple documents in one test file" for some discussions.
This patch has explored several alternatives. The current semantics are similar to
what @dblaikie proposed.
`split-file filename output` splits the input file into multiple parts separated by
regex `^(.|//)--- filename` and write each part to the file `output/filename`
(`filename` can include path separators).
Use case A (organizing input of different formats (e.g. linker
script+assembly) in one file).
```
# RUN: split-file %s %t
# RUN: llvm-mc %t/asm -o %t.o
# RUN: ld.lld -T %t/lds %t.o -o %t
This is sometimes better than the %S/Inputs/ approach because the user
can see the auxiliary files immediately and don't have to open another file.
# asm
...
# lds
...
```
Use case B (for utilities which don't have built-in input splitting
feature):
```
// RUN: split-file %s %t
// RUN: llc < %t/1.ll | FileCheck %s --check-prefix=CASE1
// RUN: llc < %t/2.ll | FileCheck %s --check-prefix=CASE2
Combing tests prudently can improve readability.
For example, when testing parsing errors if the recovery mechanism isn't possible,
grouping the tests in one file can more readily see test coverage/strategy.
//--- 1.ll
...
//--- 2.ll
...
```
Since this is a new utility, there is no git history concerns for
UpperCase variable names. I use lowerCase variable names like mlir/lld.
Reviewed By: jhenderson, lattner
Differential Revision: https://reviews.llvm.org/D83834
This reverts commit d054c7ee2e9f4f98af7f22a5b00a941eb919bd59.
There are discussions about the utility name, its functionality and user interface.
Revert before we reach consensus.
See https://lists.llvm.org/pipermail/llvm-dev/2020-July/143373.html
"[llvm-dev] Multiple documents in one test file" for some discussions.
`extract part filename` splits the input file into multiple parts separated by
regex `^(.|//)--- ` and extract the specified part to stdout or the
output file (if specified).
Use case A (organizing input of different formats (e.g. linker
script+assembly) in one file).
```
// RUN: extract lds %s -o %t.lds
// RUN: extract asm %s -o %t.s
// RUN: llvm-mc %t.s -o %t.o
// RUN: ld.lld -T %t.lds %t.o -o %t
This is sometimes better than the %S/Inputs/ approach because the user
can see the auxiliary files immediately and don't have to open another file.
```
Use case B (for utilities which don't have built-in input splitting
feature):
```
// RUN: extract case1 %s | llc | FileCheck %s --check-prefix=CASE1
// RUN: extract case2 %s | llc | FileCheck %s --check-prefix=CASE2
Combing tests prudently can improve readability.
This is sometimes better than having multiple test files.
```
Since this is a new utility, there is no git history concerns for
UpperCase variable names. I use lowerCase variable names like mlir/lld.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D83834
This came up in a recent review, someone was wondering were was
this all documented and I couldn't find a reference to provide.
Differential Revision: https://reviews.llvm.org/D83816
