This is helping some windows users, here is the doc:
**LLVM_LIT_TOOLS_DIR**:PATH
The path to GnuWin32 tools for tests. Valid on Windows host. Defaults to
the empty string, in which case lit will look for tools needed for tests
(e.g. ``grep``, ``sort``, etc.) in your ``%PATH%``. If GnuWin32 is not
in your
``%PATH%``, then you can set this variable to the GnuWin32 directory so
that
lit can find tools needed for tests in that directory.
Retry landing https://github.com/llvm/llvm-project/pull/153373
## Major changes from previous attempt
- remove the test in CAPI because no existing tests in CAPI deal with
sanitizer exemptions
- update `mlir/docs/Dialects/GPU.md` to reflect the new behavior: load
GPU binary in global ctors, instead of loading them at call site.
- skip the test on Aarch64 since we have an issue with initialization there
---------
Co-authored-by: Mehdi Amini <joker.eph@gmail.com>
This PR implements "automatic" location inference in the bindings. The
way it works is it walks the frame stack collecting source locations
(Python captures these in the frame itself). It is inspired by JAX's
[implementation](523ddcfbca/jax/_src/interpreters/mlir.py (L462))
but moves the frame stack traversal into the bindings for better
performance.
The system supports registering "included" and "excluded" filenames;
frames originating from functions in included filenames **will not** be
filtered and frames originating from functions in excluded filenames
**will** be filtered (in that order). This allows excluding all the
generated `*_ops_gen.py` files.
The system is also "toggleable" and off by default to save people who
have their own systems (such as JAX) from the added cost.
Note, the system stores the entire stacktrace (subject to
`locTracebackFramesLimit`) in the `Location` using specifically a
`CallSiteLoc`. This can be useful for profiling tools (flamegraphs
etc.).
Shoutout to the folks at JAX for coming up with a good system.
---------
Co-authored-by: Jacques Pienaar <jpienaar@google.com>
This relands PRs #143108 and #144538.
The original PR was reverted due to a mistake that made all the mlir
tests run only if SPIRV target was enabled. This is now resolved since
enabling spirv-tools does not required SPIRV target any longer.
spirv-tools are not required by default to run SPIRV mlir tests, but
they can be optionally enabled in some SPIRV mlir test to verify that
the produced SPIRV assembly pass validation.
The other reverted PR #144685 is not longer needed and not part of this
relanding.
Original commit message:
> At the MLIR level unsigned integer and signless integers are different
types. Indeed when looking up the two types in type definition cache
they do not match.
> Hence when translating a SPIR-V module which contains both usign and
signless integers will contain the same type declaration twice
(something like OpTypeInt 32 0) which is not permitted in SPIR-V and
such generated modules fail validation.
> This patch solves the problem by mapping unisgned integer types to
singless integer types before looking up in the type definition cache.
---------
Signed-off-by: Davide Grohmann <davide.grohmann@arm.com>
This PR adds a feature that attaches a listener to all RewritePatterns that
logs information about the modified operations.
When the MLIR test suite is run, these debug outputs can
be filtered and combined into an index linking operations to the
patterns that insert, modify, or replace them. This index is intended to
be used to create a website that allows one to look up patterns from an
operation name.
The debug logs emitted can be viewed with --debug-only=generate-pattern-catalog,
and the lit config is modified to do this when the env var MLIR_GENERATE_PATTERN_CATALOG is set.
Example usage:
```
mkdir build && cd build
cmake -G Ninja ../llvm \
-DLLVM_ENABLE_PROJECTS="mlir" \
-DLLVM_TARGETS_TO_BUILD="host" \
-DCMAKE_BUILD_TYPE=DEBUG
ninja -j 24 check-mlir
MLIR_GENERATE_PATTERN_CATALOG=1 bin/llvm-lit -j 24 -v -a tools/mlir/test | grep 'pattern-logging-listener' | sed 's/^# | [pattern-logging-listener] //g' | sort | uniq > pattern_catalog.txt
```
Sample pattern catalog output (that fits in a gist):
https://gist.github.com/j2kun/02d1ab8d31c10d71027724984c89905a
---------
Co-authored-by: Jeremy Kun <j2kun@users.noreply.github.com>
Co-authored-by: Mehdi Amini <joker.eph@gmail.com>
At the MLIR level unsigned integer and signless integers are different
types. Indeed when looking up the two types in type definition cache
they do not match.
Hence when translating a SPIR-V module which contains both usign and
signless integers will contain the same type declaration twice
(something like OpTypeInt 32 0) which is not permitted in SPIR-V and
such generated modules fail validation.
This patch solves the problem by mapping unisgned integer types to
singless integer types before looking up in the type definition cache.
---------
Signed-off-by: Davide Grohmann <davide.grohmann@arm.com>
With the current design some of the values are sank into a selection
region, despite them being also used outside that region. This is
because the current deserializer logic sinks the entire basic block
containing a conditional branch forming a header of a selection
construct, without accounting for some values being used outside. This
manifests as (for example):
```
<unknown>:0: error: 'spirv.Variable' op failed control flow structurization: it has uses outside of the enclosing selection/loop construct
<unknown>:0: note: see current operation: %0 = "spirv.Variable"()<{storage_class = #spirv.storage_class<Function>}> : () -> !spirv.ptr<vector<4xf32>, Function>
```
The proposed solution to this problem is to split the conditional basic
block into two, one block containing just the conditional branch, and
other the rest of instructions. By doing this, the logic that structures
selection regions, only sinks the comparison, keeping the rest of
instructions outside the selection region.
A SPIR-V test is required, as the problem can happen only during
deserialization and cannot be tested with `--test-spirv-roundtrip`. An
MLIR test exhibiting the problematic behaviour would be an incorrect
MLIR in the first place.
This solution is proposed as an alternative to an unfinished PR #123371,
that is unlikely to be merged in the foreseeable future, as the author
"stepped away from this for a time being". There is also a Discourse
thread:
https://discourse.llvm.org/t/spir-v-uses-outside-the-selection-region/84494
that tried to solicit some feedback on the topic.
Running MLIR python tests unders asan currently fails with
```
executed command: 'LD_PRELOAD=$(/usr/bin/clang++-17' '-print-file-name=libclang_rt.asan-x86_64.so)' /scratch/slx-llvm/.venv-3.10/bin/python3.10 /scratch/slx-llvm/mlir/test/python/ir/context_lifecycle.py
| 'LD_PRELOAD=$(/usr/bin/clang++-17': command not found
```
because lit doesn't quite understand the syntax.
To fix, we resolve the path to `libclang_rt.asan-x86_64.so` within the
lit configuration. This has the additional benefit that we don't have to
call `clang++` for every test.
Co-authored-by: Philipp-Jan Honysz <Philipp.Honysz@amd.com>
With the removal of mlir-vulkan-runner (as part of #73457) in
e7e3c45bc70904e24e2b3221ac8521e67eb84668, mlir-cpu-runner is now the
only runner for all CPU and GPU targets, and the "cpu" name has been
misleading for some time already. This commit renames it to mlir-runner.
This follows up on 733be4ed7dcf976719f424c0cb81b77a14f91f5a, which made
mlir-vulkan-runner and its associated passes redundant, and completes
the main goal of #73457. The mlir-vulkan-runner tests become part of the
integration test suite, and the Vulkan runner runtime components become
part of ExecutionEngine, just as was done when removing other
target-specific runners.
Related to #121288
This PR fixes the miscopied `config.lldb_build_directory` variable in
`lit.cfg.py` inside MLIR's test suit. `config.mlir_obj_root` is used as
a replacement for the copied python executable's directory.
**PS**: Since this is a common work-around on macOS, should we promote
it as a utility across projects?
Co-authored-by: Luohao Wang <Luohaothu@users.noreply.github.com>
Co-authored-by: Kai Sasaki <lewuathe@gmail.com>
There are two things that make running MLIR tests with ASan on a mac
tedious:
1. The `DYLD_INSERT_LIBRARIES` environment variable needs to be set to
point to `libclang_rt.asan_osx_dynamic.dylib`
2. Mac is wrapping Python, which means that the `DYLD_INSERT_LIBRARIES`
environment variable is not being respected in the Python tests. The
solution is to find and use a non-wrapped Python binary.
With the above two changes, ASan works out of the box on mac's by
setting the `-DLLVM_USE_SANITIZER=Address` cmake flag.
I have stolen most of the code in this PR from other LLVM projects. It
may be a good idea to reconcile it somewhere.
This commit builds on and completes the work done in
9f6c632ecda08bfff76b798c46d5d7cfde57b5e9 to eliminate the need for a
separate mlir-spirv-cpu-runner binary. Since the MLIR processing is
already done outside this runner, the only real difference between it
and the mlir-cpu-runner is the final linking step between the nested
LLVM IR modules. By moving this step into mlir-cpu-runner behind a new
command-line flag (`--link-nested-modules`), this commit is able to
completely remove the runner component of the mlir-spirv-cpu-runner.
The runtime libraries and the tests are moved and renamed to fit into
the Execution Engine and Integration tests, following the model of the
similar migration done for the CUDA Runner in D97463.
For both mlir and polly, the lit internal shell is the default shell for
running lit tests. However, if the user wanted to switch back to the
external shell by setting `LIT_USE_INTERNAL_SHELL=0`, the `not` used in
the body of the `if` conditional changes `use_lit_shell` to be True
instead of the intended False. Removing `not` allows for this lit config
to work as intended.
Fixes https://github.com/llvm/llvm-project/issues/106459.
This exposes most of the `RewriterBase` methods to the C API.
This allows to manipulate both the `IRRewriter` and the
`PatternRewriter`. The
`IRRewriter` can be created from the C API, while the `PatternRewriter`
cannot.
The missing operations are the ones taking `Block::iterator` and
`Region::iterator` as
parameters, as they are not exposed by the C API yet AFAIK.
The Python bindings for these methods and classes are not implemented.
The introduction of the clause-based approach to defining OpenMP
operations can make it more difficult to detect and address certain
programming errors derived from this change. Specifically, it's possible
for an operation to inadvertently override otherwise
automatically-populated properties and result in unexpected and
difficult to debug errors or incomplete operation definitions.
This patch introduces a TableGen backend that doesn't produce any
output, but rather only checks for these potential oversights in the
definition of OpenMP dialect operations and flags them as warnings or
errors. This provides descriptive and early feedback before any code is
attempted to be generated for these problematic definitions.
The "Emulated" sub-directories under "ArmSVE" and
"ArmSME" have been removed. Associated tests
have been moved up a directory and now include
the "REQUIRES" constraint for the arm-emulator.
LLVM_HAS_NVPTX_TARGET is automatically set depending on whether NVPTX
was enabled when building LLVM. Use this instead of manually defining
MLIR_ENABLE_CUDA_CONVERSIONS (whose name is a bit misleading btw).
Being able to add custom dialects is one of the big missing pieces of
the C API. This change should make it achievable via IRDL. Hopefully
this should open custom dialect definition to non-C++ users of MLIR.
Transform dialect interpreter is designed to be usable outside of the
pass pipeline, as the main program transformation driver, e.g., for
languages with explicit schedules. Provide an example of such usage with
a couple of tests.
This adds a new `mlir_arm_runner_utils` library that contains utils
specific to Arm/AArch64. This is for use in MLIR integration tests.
This initial patch adds `setArmVLBits()` and `setArmSVLBits()`. This
allows changing vector length or streaming vector length at runtime (or
setting it to a known minimum, i.e. 128-bits).
GPU Dialect lowering to SYCL runtime is driven by spirv.target_env
attached to gpu.module. As a result of this, spirv.target_env remains as
an input to LLVMIR Translation.
A SPIRVToLLVMIRTranslation without any actual translation is added to
avoid an unregistered error in mlir-cpu-runner.
SelectObjectAttr.cpp is updated to
1) Pass binary size argument to getModuleLoadFn
2) Pass parameter count to getKernelLaunchFn
This change does not impact CUDA and ROCM usage since both
mlir_cuda_runtime and mlir_rocm_runtime are already updated to accept
and ignore the extra arguments.
Previously, we were inserting za.enable/disable intrinsics for functions
with the "arm_za" attribute (at the MLIR level), rather than using the
backend attributes. This was done to avoid a dependency on the SME ABI
functions from compiler-rt (which have only recently been implemented).
Doing things this way did have correctness issues, for example, calling
a streaming-mode function from another streaming-mode function (both
with ZA enabled) would lead to ZA being disabled after returning to the
caller (where it should still be enabled). Fixing issues like this would
require re-doing the ABI work already done in the backend within MLIR.
Instead, this patch switches to use the "arm_new_za" (backend) attribute
for enabling ZA for an MLIR function. For the integration tests, this
requires some way of linking the SME ABI functions. This is done via the
`%arm_sme_abi_shlib` lit substitution. By default, this expands to a
stub implementation of the SME ABI functions, but this can be overridden
by providing the `ARM_SME_ABI_ROUTINES_SHLIB` CMake cache variable
(pointing it at an alternative implementation). For now, the ArmSME
integration tests pass with just stubs, as we don't make use of nested
ZA-enabled calls.
A future patch may add an option to compiler-rt to build the SME
builtins into a standalone shared library to allow easily
building/testing with the actual implementation.
This reverts commit b96f6cf62902ca96ed5aa62d4e158292280284e1
since it has broken some Windows build bots
(see https://reviews.llvm.org/D157204).
Will recommit a fixed version later.
**For an explanation of these patches see D154153.**
Commit message:
This pass converts GPU modules into GPU binaries, serializing all targets present
in a GPU module by invoking the `serializeToObject` target attribute method.
Depends on D154147
Reviewed By: mehdi_amini
Differential Revision: https://reviews.llvm.org/D154149
The transform dialect has been around for a while and is sufficiently
stable at this point. Add the first three chapters of the tutorial
describing its usage and extension.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D151491
This is an ongoing series of commits that are reformatting our
Python code.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Differential Revision: https://reviews.llvm.org/D150782
Running:
MLIR_OPT_CHECK_IR_ROUNDTRIP=1 ninja check-mlir
will now exercises all of our test with a round-trip to bytecode and a comparison for equality.
Reviewed By: rriddle, ftynse, jpienaar
Differential Revision: https://reviews.llvm.org/D90088
This reverts commit 5561e174117ff395d65b6978d04b62c1a1275138
The logic was moved from cmake into lit fixing the issue that lead to the revert and potentially others with multi-config cmake generators
Differential Revision: https://reviews.llvm.org/D143925
This patch contains the changes required to make the vast majority of integration and runner tests run on Windows.
Historically speaking, the JIT support for Windows has been lacking behind, but recent versions of ORC JIT have now caught up and works for basically all examples in repo.
Sadly due to these tests previously not working on Windows, basically all of them are making unix-like assumptions about things like filenames, paths, shell syntax etc.
This patch fixes all these issues in one big swoop and enables Windows support for the vast majority of integration tests.
More specifically, following changes had to be done:
* The various JIT runners used paths to the runtime libraries that assumed a Unix toolchain layout and filenames. I abstracted the specific path and filename of these runtime libraries away by making the paths to the runtime libraries be passed from cmake into lit. This now also allows a much more convenient syntax: `--shared-libs=%mlir_c_runner_utils` instead of `--shared-libs=%mlir_lib_dir/lib/libmlir_c_runner_utils%shlibext`
* Some tests using python set environment variables using the `ENV=VALUE cmd` format. This works on Unix, but on Windows it has to prefixed using `env ENV=VALUE cmd`
* Some tests used C functions that are simply not available or exported on Windows (`fabsf`, `aligned_alloc`). These tests have either been adjusted or explicitly marked as `UNSUPPORTED`
Some tests remain disabled on Windows as before:
* In SparseTensor some tests have non-trivial logic for finding the runtime libraries which seems to be required for the use of emulators. I do not have the time to port these so I simply kept them disabled
* Some tests requiring special hardware which I simply cannot test remain disabled on Windows. These include usage of AVX512 or AMX
The tests for `mlir-vulkan-runner` and `mlir-spirv-runner` all work now as well and so do the vast majority of `mlir-cpu-runner`.
Differential Revision: https://reviews.llvm.org/D143925
Add a new OperationType handle type to the Transform dialect. This
transform type is parameterized by the name of the payload operation it
can point to. It is intended as a constraint on transformations that are
only applicable to a specific kind of payload operations. If a
transformation is applicable to a small set of operation classes, it can
be wrapped into a transform op by using a disjunctive constraint, such
as `Type<Or<[Transform_ConcreteOperation<"foo">.predicate,
Transform_ConcreteOperation<"bar">.predicate]>>` for its operand without
modifying this type. Broader sets of accepted operations should be
modeled as specific types.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D135586
The tools are called e.g. `toyc-ch1`, not `toy-ch1`.
Add missing toyc-ch6/7.
It turns out that the other substitutions are not needed more by specific circumstances rather than by design:
The lit test exec root is set to build/mlir/test, which is where all the test tools are placed by CMake and we wouldn't need to substitute them at all.
We shouldn't rely on this assumption though, because it will make things harder for standalone tests and other build systems.
Reviewed By: mehdi_amini
Differential Revision: https://reviews.llvm.org/D133842
