e4af5b102b
# Problem:
There are two build system bugs on MacOS in the case where one intends
to use multiple bindings packages simultaneously (same Python
interpreter session):
1. The nanobind modules are built with
[`-Wl,-flat_namespace`](8518d2c405/llvm/cmake/modules/HandleLLVMOptions.cmake (L268))
thereby leading to ambiguous symbols across multiple whatever dylibs;
2. Intra-library symbol resolution (within the C API aggregate dylib)
fails to resolve symbols correctly unless things are built with
`-DCMAKE_C_VISIBILITY_PRESET=hidden -DCMAKE_CXX_VISIBILITY_PRESET=hidden
-DCMAKE_VISIBILITY_INLINES_HIDDEN=ON`.
# Repro:
On a Mac (with this patch applied):
1. Build without `twolevel_namespace` and without hidden vis properties
and run `LIT_FILTER=test.toy ninja check-mlir` (assuming you have
`-DLLVM_BUILD_EXAMPLES=ON -DLLVM_INCLUDE_EXAMPLES=ON`) and you will see:
```
LLVM ERROR: can't create Attribute 'mlir::StringAttr' because storage
uniquer isn't initialized: the dialect was likely not loaded, or the
attribute wasn't added with addAttributes<...>() in the
Dialect::initialize() method.
```
2. Build with `twolevel_namespace` but not hidden vis and run the same
lit test and you will see:
```
LLVM ERROR: Attempting to attach an interface to an unregistered
operation builtin.unrealized_conversion_cast.
```
# Fix
We only do a partial fix here (adding `twolevel_namespace` to Python
bindings modules) because a full fix requires adding visibility
attributes to all object files. I added docs discussing this.
# Why is this not happening on Linux
Using `DYLD_PRINT_BINDINGS=1` I observe that for the checked-in/updated
test (without the fix) `libMLIRPythonCAPI` resolves many of its symbols
to `libStandalonePythonCAPI`:
```
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#22> -> 0x11348fa9c <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v>)
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#23> -> 0x11348f990 <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v>)
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#24> -> 0x11348eec0 <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v>)
```
Turns out this is "expected" behavior:
> It appears on macOS, when a static library is compiled without
-fvisibility=hidden, its C++ template instantiations could lead to
leftover weak symbols that are resolved and bound at runtime
https://joyeecheung.github.io/blog/2025/01/11/executable-loading-and-startup-performance-on-macos/
🤷
70 lines
3.0 KiB
TOML
70 lines
3.0 KiB
TOML
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
# See https://llvm.org/LICENSE.txt for license information.
|
|
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
# Copyright (c) 2025.
|
|
|
|
[project]
|
|
name = "standalone-python-bindings"
|
|
dynamic = ["version"]
|
|
requires-python = ">=3.8"
|
|
dependencies = [
|
|
"numpy>=1.19.5, <=2.1.2",
|
|
"PyYAML>=5.4.0, <=6.0.1",
|
|
"ml_dtypes>=0.1.0, <=0.6.0; python_version<'3.13'",
|
|
"ml_dtypes>=0.5.0, <=0.6.0; python_version>='3.13'",
|
|
]
|
|
|
|
[project.urls]
|
|
Homepage = "https://github.com/llvm/llvm-project"
|
|
Discussions = "https://discourse.llvm.org/"
|
|
"Issue Tracker" = "https://github.com/llvm/llvm-project/issues?q=is%3Aissue%20state%3Aopen%20label%3Amlir%3Apython%20"
|
|
"Source Code" = "https://github.com/llvm/llvm-project/tree/main/mlir/python"
|
|
|
|
[build-system]
|
|
requires = [
|
|
"scikit-build-core>=0.10.7",
|
|
"typing_extensions>=4.12.2",
|
|
"nanobind>=2.9, <3.0",
|
|
]
|
|
build-backend = "scikit_build_core.build"
|
|
|
|
[tool.scikit-build]
|
|
# This is the minimum version of scikit-build-core.
|
|
minimum-version = "0.10.7"
|
|
# This pyproject.toml must be adjacent to the root CMakeLists.txt (wherever project(...) is specified).
|
|
cmake.source-dir = "."
|
|
# This is for installing/distributing the python bindings target and only the python bindings target.
|
|
build.targets = ["StandalonePythonModules"]
|
|
install.components = ["StandalonePythonModules"]
|
|
|
|
[tool.scikit-build.cmake.define]
|
|
# Optional
|
|
CMAKE_C_COMPILER = { env = "CMAKE_C_COMPILER", default = "" }
|
|
CMAKE_CXX_COMPILER = { env = "CMAKE_CXX_COMPILER", default = "" }
|
|
CMAKE_C_COMPILER_LAUNCHER = { env = "CMAKE_C_COMPILER_LAUNCHER", default = "" }
|
|
CMAKE_CXX_COMPILER_LAUNCHER = { env = "CMAKE_CXX_COMPILER_LAUNCHER", default = "" }
|
|
CMAKE_GENERATOR = { env = "CMAKE_GENERATOR", default = "Ninja" }
|
|
LLVM_USE_LINKER = { env = "LLVM_USE_LINKER", default = "" }
|
|
# Optional but highly recommended (this makes the bindings compatible with other bindings packages
|
|
# by preventing symbol collisions).
|
|
CMAKE_VISIBILITY_INLINES_HIDDEN = "ON"
|
|
CMAKE_C_VISIBILITY_PRESET = "hidden"
|
|
CMAKE_CXX_VISIBILITY_PRESET = "hidden"
|
|
|
|
# Non-optional (alternatively you could use CMAKE_PREFIX_PATH here).
|
|
MLIR_DIR = { env = "MLIR_DIR", default = "" }
|
|
# Non-optional
|
|
CMAKE_BUILD_TYPE = { env = "CMAKE_BUILD_TYPE", default = "Release" }
|
|
MLIR_ENABLE_BINDINGS_PYTHON = "ON"
|
|
|
|
# Effectively non-optional (any downstream project should specify this).
|
|
MLIR_BINDINGS_PYTHON_NB_DOMAIN = { env = "MLIR_BINDINGS_PYTHON_NB_DOMAIN", default = "mlir_standalone" }
|
|
MLIR_PYTHON_PACKAGE_PREFIX = { env = "MLIR_PYTHON_PACKAGE_PREFIX", default = "mlir_standalone" }
|
|
|
|
# This specifies the directory in the install directory (i.e., /tmp/pip-wheel/platlib) where _mlir_libs, dialects, etc.
|
|
# are installed. Thus, this will be the package location (and the name of the package) that pip assumes is
|
|
# the root package.
|
|
MLIR_BINDINGS_PYTHON_INSTALL_PREFIX = { env = "MLIR_BINDINGS_PYTHON_INSTALL_PREFIX", default = "mlir_standalone" }
|
|
# Optional
|
|
MLIR_INCLUDE_TESTS = { env = "MLIR_INCLUDE_TESTS", default = "ON" }
|