A CMake change included in CMake 4.0 makes `AIX` into a variable
(similar to `APPLE`, etc.)
ff03db6657
However, `${CMAKE_SYSTEM_NAME}` unfortunately also expands exactly to
`AIX` and `if` auto-expands variable names in CMake. That means you get
a double expansion if you write:
`if (${CMAKE_SYSTEM_NAME} MATCHES "AIX")`
which becomes:
`if (AIX MATCHES "AIX")`
which is as if you wrote:
`if (ON MATCHES "AIX")`
You can prevent this by quoting the expansion of "${CMAKE_SYSTEM_NAME}",
due to policy
[CMP0054](https://cmake.org/cmake/help/latest/policy/CMP0054.html#policy:CMP0054)
which is on by default in 4.0+. Most of the LLVM CMake already does
this, but this PR fixes the remaining cases where we do not.
When `LLVM_INSTALL_TOOLCHAIN_ONLY=ON`, the MLIR shared library
(`libMLIR*`) is not installed even though it is built with the
`INSTALL_WITH_TOOLCHAIN` argument to the `add_mlir_library` cmake
function. This patch ensures that `libMLIR*` is installed when
`LLVM_INSTALL_TOOLCHAIN_ONLY=ON`.
Patch verified
[here](https://github.com/llvm/llvm-project/issues/151247#issuecomment-3156387055).
fixes#151247
Depending on the order of CMake includes the
`tablegen_compile_commands.yml` was previously missing entries due to
being deleted after a `tablegen` commands.
This PR fixes this by 1) making sure `AddMLIR` includes `TableGen` such
that new compile commands are added to a fresh YML file and 2) using an
include guard to ensure the file is cleared exactly once
This PR introduces a new tool, mlir-irdl-to-cpp, that converts IRDL to
C++ definitions.
The C++ definitions allow use of the IRDL-defined dialect in MLIR C++
infrastructure, enabling the use of conversion patterns with IRDL
dialects for example. This PR also adds CMake utilities to easily
integrate the IRDL dialects into MLIR projects.
Note that most IRDL features are not supported. In general, we are only
able to define simple types and operations.
- The only type constraint supported is irdl.any.
- Variadic operands and results are not supported.
- Verifiers for the IRDL constraints are not generated.
- Attributes are not supported.
---------
Co-authored-by: Théo Degioanni <theo.degioanni.llvm.deluge062@simplelogin.fr>
Co-authored-by: Fehr Mathieu <mathieu.fehr@gmail.com>
This PR introduces a new tool, mlir-irdl-to-cpp, that converts IRDL to
C++ definitions.
The C++ definitions allow use of the IRDL-defined dialect in MLIR C++
infrastructure, enabling the use of conversion patterns with IRDL
dialects for example. This PR also adds CMake utilities to easily
integrate the IRDL dialects into MLIR projects.
Note that most IRDL features are not supported. In general, we are only
able to define simple types and operations.
- The only type constraint supported is `irdl.any`.
- Variadic operands and results are not supported.
- Verifiers for the IRDL constraints are not generated.
- Attributes are not supported.
---------
Co-authored-by: Théo Degioanni <theo.degioanni.llvm.deluge062@simplelogin.fr>
Co-authored-by: Fehr Mathieu <mathieu.fehr@gmail.com>
This reverts commit fa65a228f4b46346e69e9b95805a8bcfa8483a60.
This relands commit ab405fb6e9ff9202ca722f632b945d4b84c653f5.
There was an issue where CMake versions <3.23.0 would not properly parse
dep files, causing the build to file. This patch fixes that by just
making CMake versions <3.23.0 use the fallback behavior.
CMP0116 was originally set to old to get rid of warnings. However, this
behavior is now set to new by default with the minimum CMake version
that LLVM requires so does not produce any warnings, and setting it
explicitly to old does produce a warning in newer CMake versions. Due to
these reasons, remove this check for now.
Splitting off from removing the CMP0114 check just in case something
breaks.
Partially fixes#83727.
Fixes#126162
I check locally that it works without warning for:
- neither options are defined
- both defined to the same value
And I checked that it warns if:
- only one is defined
- they defined to different values
MLIR_MAIN_SRC_DIR and MLIR_INCLUDE_DIR point to the source directory,
which is not installed. As such, the installed MLIRConfig.cmake also
should not reference it.
The comment indicates that these are needed for mlir_tablegen(), but I
don't see any related uses.
The motivation for this is the use in flang, where we end up inheriting
a meaningless MLIR_MAIN_SRC_DIR from a previous MLIR build, whose source
directory doesn't exist anymore, and that cannot be overridden with the
correct path, because it's not a cached variable.
Instead do what all the other projects do for LLVM_MAIN_SRC_DIR and
initialize MLIR_MAIN_SRC_DIR to CMAKE_CURRENT_SOURCE_DIR/../mlir.
For MLIR_INCLUDE_DIR there already is an exported MLIR_INCLUDE_DIRS,
which can be used instead.
Context:
https://github.com/llvm/llvm-project/pull/107103#discussion_r1925834532
This code is brittle, especially when called from a superproject that
adds the `nanobind-*` target in a different source directory:
```cmake
get_property(all_targets DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY BUILDSYSTEM_TARGETS)
```
The changes here do help with my downstream build, but I'm not sure if
using the `MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES` option introduced
in https://github.com/llvm/llvm-project/pull/117934 is the right fix
given that the option is currently scoped directly to one location with
a matching name:
7ad8a3da47/mlir/cmake/modules/MLIRDetectPythonEnv.cmake (L4-L5)
Some other solutions to consider:
1. Search through an explicit list of target names using `if (TARGET)`
2. Iterate over _all_ targets in the project, not just the targets in
the current directory, using code like
https://stackoverflow.com/a/62311397
3. Iterate over targets in the directory known to MLIR
(`llvm-project/mlir/python`)
4. Move this `target_compile_options` setup into
`mlir_configure_python_dev_packages` (I started on this, but that runs
into similar issues where the target is defined in a different
directory)
Use `mlir_target_link_libraries()` to link dependencies of libraries
that are not included in libMLIR, to ensure that they link to the dylib
when they are used in Flang. Otherwise, they implicitly pull in all
their static dependencies, effectively causing Flang binaries to
simultaneously link to the dylib and to static libraries, which is never
a good idea.
I have only covered the libraries that are used by Flang. If you wish, I
can extend this approach to all non-libMLIR libraries in MLIR, making
MLIR itself also link to the dylib consistently.
[v3 with more `-DBUILD_SHARED_LIBS=ON` fixes]
Use `mlir_target_link_libraries()` to link dependencies of libraries
that are not included in libMLIR, to ensure that they link to the dylib
when they are used in Flang. Otherwise, they implicitly pull in all
their static dependencies, effectively causing Flang binaries to
simultaneously link to the dylib and to static libraries, which is never
a good idea.
I have only covered the libraries that are used by Flang. If you wish, I
can extend this approach to all non-libMLIR libraries in MLIR, making
MLIR itself also link to the dylib consistently.
[v2 with fixed `-DBUILD_SHARED_LIBS=ON` build]
See https://cmake.org/cmake/help/latest/policy/CMP0175.html
> The `OUTPUT` form does not accept `PRE_BUILD`, `PRE_LINK`, or
`POST_BUILD` keywords.
When using CMake version 3.31+, this results in ~2000 lines of warning
spam in my downstream project:
```
CMake Warning (dev) at build-gcc/lib/cmake/mlir/AddMLIRPython.cmake:606 (add_custom_command):
The following keywords are not supported when using
add_custom_command(OUTPUT): PRE_BUILD.
Policy CMP0175 is not set: add_custom_command() rejects invalid arguments.
Run "cmake --help-policy CMP0175" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
Call Stack (most recent call first):
build-gcc/lib/cmake/mlir/AddMLIRPython.cmake:222 (add_mlir_python_sources_target)
build-gcc/lib/cmake/mlir/AddMLIRPython.cmake:256 (_process_target)
compiler/bindings/python/CMakeLists.txt:239 (add_mlir_python_modules)
This warning is for project developers. Use -Wno-dev to suppress it.
```
General docs:
https://cmake.org/cmake/help/latest/command/add_custom_command.html.
Note that `PRE_BUILD` only appears in the _second_ signature for the
function (which takes `TARGET`) not the first (which takes `OUTPUT`).
Use `mlir_target_link_libraries()` to link dependencies of libraries
that are not included in libMLIR, to ensure that they link to the dylib
when they are used in Flang. Otherwise, they implicitly pull in all
their static dependencies, effectively causing Flang binaries to
simultaneously link to the dylib and to static libraries, which is never
a good idea.
I have only covered the libraries that are used by Flang. If you wish, I
can extend this approach to all non-libMLIR libraries in MLIR, making
MLIR itself also link to the dylib consistently.
`NanobindAdaptors.h` uses `PyClassMethod_New` to build `pure_subclass`es
but nanobind doesn't declare this API as undefined in its linker flags.
So we need to declare it as such for downstream users that do not do
something like `-undefined dynamic_lookup`
This is a companion to #118583, although it can be landed independently
because since #117922 dialects do not have to use the same Python
binding framework as the Python core code.
This PR ports all of the in-tree dialect and pass extensions to
nanobind, with the exception of those that remain for testing pybind11
support.
This PR also:
* removes CollectDiagnosticsToStringScope from NanobindAdaptors.h. This
was overlooked in a previous PR and it is duplicated in Diagnostics.h.
---------
Co-authored-by: Jacques Pienaar <jpienaar@google.com>
Relands #118583, with a fix for Python 3.8 compatibility. It was not
possible to set the buffer protocol accessers via slots in Python 3.8.
Why? https://nanobind.readthedocs.io/en/latest/why.html says it better
than I can, but my primary motivation for this change is to improve MLIR
IR construction time from JAX.
For a complicated Google-internal LLM model in JAX, this change improves
the MLIR
lowering time by around 5s (out of around 30s), which is a significant
speedup for simply switching binding frameworks.
To a large extent, this is a mechanical change, for instance changing
`pybind11::` to `nanobind::`.
Notes:
* this PR needs Nanobind 2.4.0, because it needs a bug fix
(https://github.com/wjakob/nanobind/pull/806) that landed in that
release.
* this PR does not port the in-tree dialect extension modules. They can
be ported in a future PR.
* I removed the py::sibling() annotations from def_static and def_class
in `PybindAdapters.h`. These ask pybind11 to try to form an overload
with an existing method, but it's not possible to form mixed
pybind11/nanobind overloads this ways and the parent class is now
defined in nanobind. Better solutions may be possible here.
* nanobind does not contain an exact equivalent of pybind11's buffer
protocol support. It was not hard to add a nanobind implementation of a
similar API.
* nanobind is pickier about casting to std::vector<bool>, expecting that
the input is a sequence of bool types, not truthy values. In a couple of
places I added code to support truthy values during casting.
* nanobind distinguishes bytes (`nb::bytes`) from strings (e.g.,
`std::string`). This required nb::bytes overloads in a few places.
Why? https://nanobind.readthedocs.io/en/latest/why.html says it better
than I can, but my primary motivation for this change is to improve MLIR
IR construction time from JAX.
For a complicated Google-internal LLM model in JAX, this change improves
the MLIR
lowering time by around 5s (out of around 30s), which is a significant
speedup for simply switching binding frameworks.
To a large extent, this is a mechanical change, for instance changing
`pybind11::`
to `nanobind::`.
Notes:
* this PR needs Nanobind 2.4.0, because it needs a bug fix
(https://github.com/wjakob/nanobind/pull/806) that landed in that
release.
* this PR does not port the in-tree dialect extension modules. They can
be ported in a future PR.
* I removed the py::sibling() annotations from def_static and def_class
in `PybindAdapters.h`. These ask pybind11 to try to form an overload
with an existing method, but it's not possible to form mixed
pybind11/nanobind overloads this ways and the parent class is now
defined in nanobind. Better solutions may be possible here.
* nanobind does not contain an exact equivalent of pybind11's buffer
protocol support. It was not hard to add a nanobind implementation of a
similar API.
* nanobind is pickier about casting to std::vector<bool>, expecting that
the input is a sequence of bool types, not truthy values. In a couple of
places I added code to support truthy values during casting.
* nanobind distinguishes bytes (`nb::bytes`) from strings (e.g.,
`std::string`). This required nb::bytes overloads in a few places.
While MLIR currently supports building a libMLIR.so, it does not support
actually linking against it for its own tools. When building with LTO,
this means we have to relink the world for every tool, and the resulting
binaries are large.
This adds basic support for MLIR_LINK_MLIR_DYLIB, modelled after how
CLANG_LINK_CLANG_DYLIB is implemented: Libraries that are part of
libMLIR.so should be added via mlir_target_link_libraries instead of
target_link_libraries. This will replace them with libMLIR.so if
MLIR_LINK_MLIR_DYLIB is enabled.
This adds basic support, I think there are two more things that can be
done here:
* C API unit tests should link against libMLIR-C.so. Currently these
still link statically.
* Linking the test libs (not part of libMLIR.so) still pulls in
dependencies statically that should come from libMLIR.so.
This PR allows out-of-tree dialects to write Python dialect modules
using nanobind instead of pybind11.
It may make sense to migrate in-tree dialects and some of the ODS Python
infrastructure to nanobind, but that is a topic for a future change.
This PR makes the following changes:
* adds nanobind to the CMake and Bazel build systems. We also add
robin_map to the Bazel build, which is a dependency of nanobind.
* adds a PYTHON_BINDING_LIBRARY option to various CMake functions, such
as declare_mlir_python_extension, allowing users to select a Python
binding library.
* creates a fork of mlir/include/mlir/Bindings/Python/PybindAdaptors.h
named NanobindAdaptors.h. This plays the same role, using nanobind
instead of pybind11.
* splits CollectDiagnosticsToStringScope out of PybindAdaptors.h and
into a new header mlir/include/mlir/Bindings/Python/Diagnostics.h, since
it is code that is no way related to pybind11 or for that matter,
Python.
* changed the standalone Python extension example to have both pybind11
and nanobind variants.
* changed mlir/python/mlir/dialects/python_test.py to have both pybind11
and nanobind variants.
Notes:
* A slightly unfortunate thing that I needed to do in the CMake
integration was to use FindPython in addition to FindPython3, since
nanobind's CMake integration expects the Python_ names for variables.
Perhaps there's a better way to do this.
Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which
gates doing package discovery and configuration for Python dev packages
by MLIR (this was made opt-out to preserve compatibility with
find_package(MLIR) based uses which do not set the standard options).
The default Python setup that MLIR does has been a problem for
super-projects that include LLVM for a long time because it forces a
very specific package discovery mechanism that is not uniform in all
uses.
When reviewing #117922, I noted that this would effectively be a break
the world event for downstreams, forcing them to adapt their nanobind
dep to the exact way that MLIR does it. Adding the option to just
wholesale skip the built-in configuration heuristics at least gives us a
mechanism to tell downstreams to migrate to, giving them complete
control and not requiring packaging workarounds. This seemed a better
option than (once again) creating a situation where downstreams could
not integrate the dep change without doing tricky infra upgrades, and it
removes the burden from the author of that patch from needing to think
about how this affects super-projects that include MLIR (i.e. they can
just be told to do it themselves as needed vs being in a wedged state
and unable to upgrade).
This PR updates the minimal required version of pybind11 from 2.9.0 to
2.10.0. New new version is almost 2.5 years old, which is half a year
less than the previous version. This change is necessary to support the
changes introduced in #115307, which does not compile with pybind11
v.2.9.
Signed-off-by: Ingo Müller <ingomueller@google.com>
NumPy header files were required for building MLIR, however the NumPy
C-API is never used. In other words, NumPy is not a build time
dependency. `numpy`, the python package, is required at runtime for the
python bindings tests. In particular the file
`mlir/python/mlir/runtime/np_to_memref.py` and all tests which may use
it. This commit removes the build time dependency, but the runtime
dependency remains through the `requirements.txt` file.
Co-authored-by: Erick Ochoa <erick@ceci-nest-pas.me>
Make MLIR Python sources idempotent by moving the custom commands off
the custom target for each source.
Previously, a custom target was created for each MLIR Python source file
and a custom command added to symlink it to a destination path. However
this causes the build to not be idempotent because custom targets always
run.
Instead, this PR separates the symlink/copy to destination paths into
custom commands and makes the custom target depend on the output. That
prevents re-running them and restores idempotency.
Testing:
- Build with `-D MLIR_ENABLE_BINDINGS_PYTHON=ON`. Prior to this change,
the build is not idempotent (building twice re-runs the symlink/copy
command for Python source targets). After this change it is idempotent.
As discussed in #89743, when using the Visual Studio solution
generators, object library projects are displayed as a collection of
non-editable *.obj files. To look for the corresponding source files,
one has to browse (or search) to the library's obj.libname project. This
patch tries to avoid this as much as possible.
For Clang, there is already an exception for XCode. We handle MSVC_IDE
the same way.
For MLIR, this is more complicated. There are explicit references to the
obj.libname target that only work when there is an object library. This
patch cleans up the reasons for why an object library is needed:
1. The obj.libname is modified in the calling CMakeLists.txt. Note that
with use-only references, `add_library(<name> ALIAS <target>)` could
have been used.
2. An `libMLIR.so` (mlir-shlib) is also created. This works by adding
linking the object libraries' object files into the libMLIR.so (in
addition to the library's own .so/.a). XCode is handled using the
`-force_load` linker option instead. Windows is not supported. This
mechanism is different from LLVM's llvm-shlib that is created by linking
static libraries with `-Wl,--whole-archive` (and `-Wl,-all_load` on
MacOS).
3. The library might be added to an aggregate library. In-tree, the
seems to be only `libMLIR-C.so` and the standalone example. In XCode, it
uses the object library and `-force_load` mechanism as above. Again,
this is different from `libLLVM-C.so`.
4. Build an object library whenever it was before this patch, except
when generating a Visual Studio solution. This condition could be
removed, but I am trying to avoid build breakages of whatever
configurations others use.
This seems to never have worked with XCode because of the explicit
references to obj.libname (reason 1.). I don't have access to XCode, but
I tried to preserve the current working. IMHO there should be a common
mechanism to build aggregate libraries for all LLVM projects instead of
the 4 that we have now.
As far as I can see, this means for LLVM there are the following changes
on whether object libraries are created:
1. An object library is created even in XCode if FORCE_OBJECT_LIBRARY is
set. I do not know how XCode handles it, but I also know CMake will
abort otherwise.
2. An object library is created even for explicitly SHARED libraries for
building `libMLIR.so`. Again, mlir-shlib does not work otherwise.
`libMLIR.so` itself is created using SHARED so this patch is marking it
as EXCLUDE_FROM_LIBMLIR.
3. For the second condition, it is now sensitive to whether the
mlir-shlib is built at all (LLVM_BUILD_LLVM_DYLIB). However, an object
library is still built using the fourth condition unless using the MSVC
solution generator. That is, except with MSVC_IDE, when an object
library was built before, it will also be an object library now.
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.
* Ensure that every target is in a folder
* Use a folder hierarchy with each LLVM subproject as a top-level folder
* Use consistent folder names between subprojects
* When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
Adds an option to `mlir-tblgen -gen-op-defs` `op-shard-count=N` that
divides the
op class definitions and op list into N segments, e.g.
```
// mlir-tblgen -gen-op-defs -op-shard-count=2
void FooDialect::initialize() {
addOperations<
>();
addOperations<
>();
}
```
When split across multiple source files, this can help significantly
improve
dialect compile time for dialects with a large opset.
Adds two new CMake functions to query the host system:
* `check_hwcap`,
* `check_emulator`.
Together, these functions are used to check whether a given set of MLIR
integration tests require an emulator. If yes, then the corresponding
CMake var that defies the required emulator executable is also checked.
`check_hwcap` relies on ELF_HWCAP for discovering CPU features from
userspace on Linux systems. This is the recommended approach for Arm
CPUs running on Linux as outlined in this blog post:
* https://community.arm.com/arm-community-blogs/b/operating-systems-blog/posts/runtime-detection-of-cpu-features-on-an-armv8-a-cpu
Other operating systems (e.g. Android) and CPU architectures will
most likely require some other approach. Right now these new hooks are
only used for SVE and SME integration tests.
This relands #86489 with the following changes:
* Replaced:
`set(hwcap_test_file ${CMAKE_BINARY_DIR}/${CMAKE_FILES_DIRECTORY}/hwcap_check.c)`
with:
`set(hwcap_test_file ${CMAKE_BINARY_DIR}/temp/hwcap_check.c)`
The former would trigger an infinite loop when running `ninja`
(after the initial CMake configuration).
* Fixed commit msg. Previous one was taken from the initial GH PR
commit rather than the final re-worked solution (missed this when
merging via GH UI).
* A couple more NFCs/tweaks.
Integration tests for ArmSME require an emulator (there's no hardware
available). Make sure that CMake complains if `MLIR_RUN_ARM_SME_TESTS`
is set while `ARM_EMULATOR_EXECUTABLE` is empty.
I'm also adding a note in the docs for future reference.
