…se of tensor pack
When the vector sizes are not passed as inputs to the vector transform
operation, the vector sizes are queried from the static result shape in
the case of tensor.pack op.
A `sparse_tensor.extract_space %tensor at %iterator` extracts a *sparse*
iteration space defined `%tensor`, the operation to traverse the
iteration space will be introduced in following PRs.
The existing lowering for tosa.max_pool2d only supports dynamic
dimensions when the dynamic dimension is the batch dimension. This
change updates the lowering to support arbitrary dynamic dimensions on
the inputs and outputs of the tosa.max_pool2d operation.
This change also fixes a bug in the implementation of implicit
broadcasting in the tosa-to-linalg pass, which was introducing uses of
constant ops that violated dominance requirements.
At the moment there is no support for vector.shuffle for scalable
vectors - various hooks/helpers related to `vector.shuffle` simply
ignore the scalable flags (e.g. ` ShuffleOp::inferReturnTypes`).
This is unlikely to change any time soon (vector shuffles are known to
be tricky for scalable vectors), hence this patch restricts
`vector.shuffle` to fixed width vectors.
This adds a simple rewrite/legalization to decompose constant splats
larger than a single ArmSME tile into multiple SME virtual tile sized
splats. E.g. a constant splat to `vector<[8]x[8]xi32>` would decompose
into four `vector<[4]x[4]xi32>` splats.
This commit generalizes and cleans up the `ValueBoundsConstraintSet`
API. The API used to provide function overloads for comparing/computing
bounds of:
- index-typed SSA value
- dimension of shaped value
- affine map + operands
This commit removes all overloads. There is now a single entry point for
each `compare` variant and each `computeBound` variant. These functions
now take a `Variable`, which is internally represented as an affine map
and map operands.
This commit also adds support for computing bounds for an affine map +
operands. There was previously no public API for that.
Add uplifting from `scf.while` to `scf.for`.
This uplifting expects a very specific ops pattern:
* `before` block consisting of single `arith.cmp` op
* `after` block containing `arith.addi`
We also have a set of patterns to cleanup `scf.while` loops to get them
close to the desired form, they will be added in separate PRs.
This is part of upstreaming `numba-mlir` scf uplifting pipeline: `cf ->
scf.while -> scf.for -> scf.parallel`
Original code:
https://github.com/numba/numba-mlir/blob/main/mlir/lib/Transforms/PromoteToParallel.cpp
There are execution modes need target width as their extra operands.
SignedZeroInfNanPreserve is one of them. This patch adds `target width`
as one of SPIR-V ABI attributes.
This commit is a code cleanup. It defines the test ops the are used for
the `ValueBoundsOpInterface` tests in TableGen, along with proper
verifiers.
---------
Co-authored-by: Benjamin Maxwell <benjamin.maxwell@arm.com>
RFC:
https://discourse.llvm.org/t/rfc-a-poly-dialect-for-polynomial-arithmetic/73891
This PR implements the minimal work needed to represent the polynomial
type such that it can be tested with `lit`.
In this PR:
- Dialect shell
- `Polynomial` data structure needed for folding
- Polynomial attributes (`PolynomialAttr` and `RingAttr` which store a polynomial)
- `polynomial.polynomial` type
- Basic lit tests
---------
Co-authored-by: Jeremy Kun <j2kun@users.noreply.github.com>
`attr-dict` currently prints any attribute (inherent or discardable)
that does not occur elsewhere in the assembly format. `prop-dict` on the
other hand, always prints and parses all properties (including inherent
attributes stored as properties) as part of the property dictionary,
regardless of whether the properties or attributes are used elsewhere.
(with the exception of default-valued attributes implemented recently in
https://github.com/llvm/llvm-project/pull/87970).
This PR changes the behavior of `prop-dict` to only print and parse
attributes and properties that do not occur elsewhere in the assembly
format. This is achieved by 1) adding used attributes and properties to
the elision list when printing and 2) using a custom version of
`setPropertiesFromAttr` called `setPropertiesFromParsedAttr` that is
sensitive to the assembly format and auto-generated by ODS.
The current and new behavior of `prop-dict` and `attr-dict` were also
documented.
Happens to also fix https://github.com/llvm/llvm-project/issues/88506
The current lowering has issues with large/subnormal values. This po
XLA's lowering and was verified using XLA's test suite and the
MLIR-based emitters.
This updates https://github.com/llvm/llvm-project/pull/88691 to also
update the correctness test for rsqrt(0). I checked C++ and Python, they
both agree the result should be (inf, nan). Updated the correctness test
to match this.
This patch defines a common interface to be shared by all OpenMP loop
wrapper operations. The main restrictions these operations must meet in
order to be considered a wrapper are:
- They contain a single region.
- Their region contains a single block.
- Their block only contains another loop wrapper or `omp.loop_nest` and
a terminator.
The new interface is attached to the `omp.parallel`, `omp.wsloop`,
`omp.simdloop`, `omp.distribute` and `omp.taskloop` operations. It is
not currently enforced that these operations meet the wrapper
restrictions, which would break existing OpenMP loop-generating code.
Rather, this will be introduced progressively in subsequent patches.
The current lowering has issues with large/subnormal values. This ports
XLA's lowering and was verified using XLA's test suite and the
MLIR-based emitters.
These resulted in link failures:
```
/usr/bin/ld:
tools/mlir/test/CAPI/CMakeFiles/mlir-capi-translation-test.dir/translation.c.o:
in function `main':
translation.c:(.text.main+0x58): undefined reference to
`LLVMContextCreate'
/usr/bin/ld: translation.c:(.text.main+0x9b): undefined reference to
`LLVMDumpModule'
/usr/bin/ld: translation.c:(.text.main+0xa3): undefined reference to
`LLVMDisposeModule'
/usr/bin/ld: translation.c:(.text.main+0xb3): undefined reference to
`LLVMContextDispose'
```
Found in mlir-hs. Not sure why this hasn't been flagged elsewhere.
Tensor copy insertion currently uses memory_space = 0 when creating a
tensor copy using alloc_tensor. This memory space should instead be the
default memory space provided in bufferization options.
This patch introduces an operation intended to hold loop information
associated to the `omp.distribute`, `omp.simdloop`, `omp.taskloop` and
`omp.wsloop` operations. This is a stopgap solution to unblock work on
transitioning these operations to becoming wrappers, as discussed in
[this
RFC](https://discourse.llvm.org/t/rfc-representing-combined-composite-constructs-in-the-openmp-dialect/76986).
Long-term, this operation will likely be replaced by
`omp.canonical_loop`, which is being designed to address missing support
for loop transformations, etc.
This diff causes the `tblgen`-erated printProperties() function to skip
printing a `DefaultValuedAttr` property when the value is equal to the
default.
Co-authored-by: Biao Wang <biaow@nvidia.com>
`verifySchedule` was not looking at the operands of nested operations,
which caused incorrect schedule to be allowed in some cases, potentially
leading to crash during expansion.
There is also minor fix in `cloneAndUpdateOperands` in the pipeline
expander that prevents double visit of the cloned op. This one has no
functional impact, so no test for it.
In order to support various external frameworks (JAX vs PyTorch) we need
a bit more flexibility in [dis]assembling external buffers to and from
sparse tensors in MLIR land. This PR adds a direct-out option that
avoids the rigid pre-allocated for copy-out semantics.
Note that over time, we expect the [dis]assemble operations to converge
into something that supports all sorts of external frameworks. Until
then, this option helps in experimenting with different options.
Added lowering support for IS_DEVICE_PTR and HAS_DEVICE_ADDR clauses for
OMP TARGET directive and added related tests for these changes.
IS_DEVICE_PTR and HAS_DEVICE_ADDR clauses apply to OMP TARGET directive
OpenMP spec states
The **is_device_ptr** clause indicates that its list items are device
pointers.
The **has_device_addr** clause indicates that its list items already
have device addresses and therefore they may be directly accessed from a
target device.
Whereas USE_DEVICE_PTR and USE_DEVICE_ADDR clauses apply to OMP TARGET
DATA directive and OpenMP spec for them states
Each list item in the **use_device_ptr** clause results in a new list
item that is a device pointer that refers to a device address
Each list item in a **use_device_addr** clause that is present in the
device data environment is treated as if it is implicitly mapped by a
map clause on the construct with a map-type of alloc
Fixed build error caused by Squash merge which needs rebase
This implementation has a number of issues and ultimately does not work
on gfx9.
* It does not reduce bank conflicts with wide memory accesses.
* It does not correctly account for when LDS bank conflicts occur on
amdgpu.
* The implementation is too fragile to be used on real-world code. For
example, the code bails out on any `memref.subview` in the root op, even
when the subview is not a user of any of the `memref.alloc` ops.
I do not see how these can be easily fixed, therefore I think it's
better to delete this code.
For example, 1e30^1.2 currently overflows.
Also forward fastmath flags.
This ports XLA's logic and was verified with its test suite. Note that
rsqrt and sqrt are still broken.
The current implementation fails for very small and very large values.
For example, (0, -inf) should return inf, but it returns -inf.
This ports the logic used in XLA. Tested with XLA's
exhaustive_binary_test_f32_f64.
I missed this before. For by-ref reductions, the private reduction
variable is a pointer to the pointer to the variable. So an extra load
is required to get the right value.
See the "red.private.value.n" loads in the reduction combiner region for
reference.
This ports openxla/xla#10503 by @pearu. In addition to the filecheck
test here, the accuracy was tested with XLA's complex_unary_op_test and
its MLIR emitters.
This is a fixed version of
https://github.com/llvm/llvm-project/pull/88260. The previous version
relied on implementation-specific behavior in the order of evaluation of
maxAbsOfRealPlusOneAndImagMinusOne's operands.
This commit adds a new public API to `ValueBoundsOpInterface` to compare
values/dims. Supported comparison operators are: LT, LE, EQ, GE, GT.
The new `ValueBoundsOpInterface::compare` API replaces and generalizes
`ValueBoundsOpInterface::areEqual`. Not only does it provide additional
comparison operators, it also works in cases where the difference
between the two values/dims is non-constant. The previous implementation
of `areEqual` used to compute a constant bound of `val1 - val2` (check
if it `== 0` or `!= 0`).
Note: This commit refactors, generalizes and adds a public API for
value/dim comparison. The comparison functionality itself was introduced
in #85895 and is already in use for analyzing `scf.if`.
In the long term, this improvement will allow for a more powerful
analysis of subset ops. A future commit will update
`areOverlappingSlices` to use the new comparison API.
(`areEquivalentSlices` is already using the new API.) This will improve
subset equivalence/disjointness checks with non-constant
offsets/sizes/strides.
Patterns in `LowerContractionToSMMLAPattern` are designed to handle
vector-to-matrix multiplication but not matrix-to-vector. This leads to
the following error when processing `rhs` with rank < 2:
```
iree-compile: /usr/local/google/home/kooljblack/code/iree-build/llvm-project/tools/mlir/include/mlir/IR/BuiltinTypeInterfaces.h.inc:268: int64_t mlir::detail::ShapedTypeTrait<mlir::VectorType>::getDimSize(unsigned int) const [ConcreteType = mlir::VectorType]: Assertion `idx < getRank() && "invalid index for shaped type"' failed.
```
Updates to explicitly check the rhs rank and fail cases that cannot
process.
This ports https://github.com/openxla/xla/pull/10503 by @pearu. The new
implementation matches mpmath's results for most inputs, see caveats in
the linked pull request. In addition to the filecheck test here, the
accuracy was tested with XLA's complex_unary_op_test and its MLIR
emitters.
Summary:
These entires are generic for offloading with the new driver now. Having
the `omp` prefix was a historical artifact and is confusing when used
for CUDA. This patch just renames them for now, future patches will
rework the binary format to make it more common.
Added lowering support for IS_DEVICE_PTR and HAS_DEVICE_ADDR clauses for
OMP TARGET directive and added related tests for these changes.
IS_DEVICE_PTR and HAS_DEVICE_ADDR clauses apply to OMP TARGET directive
OpenMP spec states
`The **is_device_ptr** clause indicates that its list items are device
pointers.`
`The **has_device_addr** clause indicates that its list items already
have device addresses and therefore they may be directly accessed from a
target device.`
Whereas USE_DEVICE_PTR and USE_DEVICE_ADDR clauses apply to OMP TARGET
DATA directive and OpenMP spec for them states
`Each list item in the **use_device_ptr** clause results in a new list
item that is a device pointer that refers to a device address`
`Each list item in a **use_device_addr** clause that is present in the
device data environment is treated as if it is implicitly mapped by a
map clause on the construct with a map-type of alloc`
