238 Commits

Author SHA1 Message Date
Longsheng Mou
0aa4a016d6
[mlir][sparse] Fix a crash if block not have terminator (#180741)
This PR fixes a crash in `verifyNumBlockArgs` if region not end with a
terminator. Fixes #180720.
2026-02-11 09:08:45 +08:00
Matthias Springer
f76433761a
[mlir][Interfaces] Split successor inputs from region successor (#175815)
This commit simplifies the design of the `RegionBranchOpInterface`. The
property of being a successor input is now independent of the region
branch point.

There is a new API for querying successor inputs:
`RegionBranchOpInterface::getSuccessorInputs(RegionSuccessor)`. Note
that this function does **not** take a `RegionBranchPoint` as parameter.

The `RegionSuccessor` API is now also simpler: it no longer stores
successor inputs. A region successor is simply `Region *`, wrapped
around a convenience API.

Note: This commit is mostly mechanical. Analyses / transformations that
build on top of the `RegionBranchOpInterface` (e.g.,
`visitNonControlFlowArguments` API) can likely be simplified in
follow-up commits.

Note for LLVM integration: Split
`RegionBranchOpInterface::getSuccessorRegion` implementations into two
functions: `getSuccessorRegion` and `getSuccessorInputs. (There are many
examples in this commit.)

RFC:
https://discourse.llvm.org/t/rfc-simplify-regionbranchopinterface-separate-successor-inputs-from-region-successor/89420/7
2026-01-16 10:16:53 +01:00
Matthias Springer
5f3b40ec7a
[mlir][Interfaces][NFC] Simplify and align RegionSuccessor design / API (#174945)
Simplify the design of `RegionSuccessor`. There is no need to store the
`Operation *` pointer when branching out of the region branch op (to the
parent). There is no API to even access the `Operation *` pointer.

Add a new helper function `RegionSuccessor::parent` to construct a
region successor that points to the parent. This aligns the
`RegionSuccessor` design and API with `RegionBranchPoint`:
* Both classes now have a `parent()` helper function.
`ClassName::parent()` can be used in documentation to precisely describe
the source/target of a region branch.
* Both classes now use `nullptr` internally to represent "parent".

This API change also protects against incorrect API usage: users can no
longer pass an incorrect parent op. If a region successor is not a
region of the region branch op, it *must* branch out of region branch op
itself ("parent"). However, the previous API allowed passing other
operations. There was one such API violation in a [test
case](https://github.com/llvm/llvm-project/pull/174945/files#diff-d5717e4a8d7344b2ff77762b8fa480bcfec0eeee97a86195c787d791a6217e13L71).

Also clean up the documentation to use the correct terminology (such as
"successor operands", "successor inputs") consistently.

Note: This PR effectively rolls back some changes from #161575. That PR
introduced `llvm::PointerUnion<Region *, Operation *>
successor{nullptr};`. It is unclear from the commit message why that
change was made.

Note for LLVM integration: You may have to slightly modify
`getSuccessorRegion` implementations: Replace
`RegionSuccessor(getOperation(), getOperation()->getResults())` with
`RegionSuccessor::parent(getResults())`.
2026-01-14 10:57:22 +01:00
Mehdi Amini
f719e11cc1 [MLIR] Apply clang-tidy fixes for misc-use-internal-linkage in SparseTensorDialect.cpp (NFC) 2025-12-17 05:30:01 -08:00
Mehdi Amini
41f65666f6
[MLIR] Revamp RegionBranchOpInterface (#165429)
This is still somehow a WIP, we have some issues with this interface
that are not trivial to solve. This patch tries to make the concepts of
RegionBranchPoint and RegionSuccessor more robust and aligned with their
definition:
- A `RegionBranchPoint` is either the parent (`RegionBranchOpInterface`)
op or a `RegionBranchTerminatorOpInterface` operation in a nested
region.
- A `RegionSuccessor` is either one of the nested region or the parent
`RegionBranchOpInterface`

Some new methods with reasonnable default implementation are added to
help resolving the flow of values across the RegionBranchOpInterface.

It is still not trivial in the current state to walk the def-use chain
backward with this interface. For example when you have the 3rd block
argument in the entry block of a for-loop, finding the matching operands
requires to know about the hidden loop iterator block argument and where
the iterargs start. The API is designed around forward-tracking of the
chain unfortunately.

Try to reland #161575 ; I suspect a buildbot incremental build issue.
2025-10-28 09:53:56 -07:00
Mehdi Amini
e3c547179f
Revert " [MLIR] Revamp RegionBranchOpInterface " (#165356)
Reverts llvm/llvm-project#161575

Broke Windows on ARM buildbot build, needs investigations.
2025-10-28 01:06:14 -07:00
Mehdi Amini
ab1fd21b54
[MLIR] Revamp RegionBranchOpInterface (#161575)
This is still somehow a WIP, we have some issues with this interface
that are not trivial to solve. This patch tries to make the concepts of
RegionBranchPoint and RegionSuccessor more robust and aligned with their
definition:
- A `RegionBranchPoint` is either the parent (`RegionBranchOpInterface`)
op or a `RegionBranchTerminatorOpInterface` operation in a nested
region.
- A `RegionSuccessor` is either one of the nested region or the parent
`RegionBranchOpInterface`

Some new methods with reasonnable default implementation are added to
help resolving the flow of values across the RegionBranchOpInterface.

It is still not trivial in the current state to walk the def-use chain
backward with this interface. For example when you have the 3rd block
argument in the entry block of a for-loop, finding the matching operands
requires to know about the hidden loop iterator block argument and where
the iterargs start. The API is designed around forward-tracking of the
chain unfortunately.
2025-10-28 07:47:26 +00:00
Maksim Levental
972ac59c9a
[mlir][NFC] update mlir/Dialect create APIs (21/n) (#149928)
See https://github.com/llvm/llvm-project/pull/147168 for more info.
2025-07-22 08:15:27 -04:00
Kazu Hirata
433a5a7c20
[mlir] Remove unused includes (NFC) (#148396) 2025-07-12 15:46:34 -07:00
Jakub Kuderski
6512ca7ddb
[mlir] Add isStatic* size check for ShapedTypes. NFCI. (#147085)
The motivation is to avoid having to negate `isDynamic*` checks, avoid
double negations, and allow for `ShapedType::isStaticDim` to be used in
ADT functions without having to wrap it in a lambda performing the
negation.

Also add the new functions to C and Python bindings.
2025-07-07 14:57:27 -04:00
Kazu Hirata
1cf1c21b84
[mlir] Strip away lambdas (NFC) (#143280)
We don't need lambdas here.
2025-06-08 01:34:17 -07:00
Kazu Hirata
03697cf8fc
[mlir] Use llvm::find_if (NFC) (#139912) 2025-05-14 12:48:48 -07:00
Hongren Zheng
3db668e0bb
[mlir][SparseTensor][NFC] Migrate to OpAsmAttrInterface for ASM alias generation (#130483)
After the introduction of `OpAsmAttrInterface`, it is favorable to
migrate code using `OpAsmDialectInterface` for ASM alias generation,
which lives in `Dialect.cpp`, to use `OpAsmAttrInterface`, which lives
in `Attrs.td`. In this way, attribute behavior is placed near its
tablegen definition and people won't need to go through other files to
know what other (unexpected) hooks comes into play.
2025-03-11 00:49:21 +08:00
Kazu Hirata
af6e1881e0
[mlir] Avoid repeated map lookups (NFC) (#113122) 2024-10-21 06:52:24 -07:00
Kazu Hirata
2077fb80ff
[mlir] Avoid repeated map lookups (NFC) (#113074) 2024-10-20 10:42:28 -07:00
Matthias Springer
77f8297c6f
[mlir][sparse] Improve sparse tensor type constraints (#112133)
Sparse tensors are always ranked tensors. Encodings cannot be attached
to unranked tensors. Change the type constraint to `RankedTensorOf`, so
that we generate `TypedValue<RankedTensorType>` instead of
`TypedValue<TensorType>`. This removes the need for type casting in some
cases.

Also improve the verifiers (missing `return` statements) and switch a
few other `AnyTensor` to `AnyRankedTensor`.

This commit is in preparation of a dialect conversion commit that
required fixes in the sparse dialect.
2024-10-13 21:12:38 +02:00
Peiming Liu
b48ef8d8d4
[mlir][sparse] unify block arguments order between iterate/coiterate operations. (#105567) 2024-08-23 14:49:00 -07:00
Peiming Liu
f607102a0d
[mlir][sparse] partially support lowering sparse coiteration loops to scf.while/for. (#105565) 2024-08-23 10:47:44 -07:00
Peiming Liu
c44202574f
[mlir][sparse] support sparsification to coiterate operations. (#102546) 2024-08-20 11:13:38 -07:00
Matthias Springer
7359a6b799
[mlir][ODS] Verify type constraints in Types and Attributes (#102326)
When a type/attribute is defined in TableGen, a type constraint can be
used for parameters, but the type constraint verification was missing.

Example:
```
def TestTypeVerification : Test_Type<"TestTypeVerification"> {
  let parameters = (ins AnyTypeOf<[I16, I32]>:$param);
  // ...
}
```

No verification code was generated to ensure that `$param` is I16 or
I32.

When type constraints a present, a new method will generated for types
and attributes: `verifyInvariantsImpl`. (The naming is similar to op
verifiers.) The user-provided verifier is called `verify` (no change).
There is now a new entry point to type/attribute verification:
`verifyInvariants`. This function calls both `verifyInvariantsImpl` and
`verify`. If neither of those two verifications are present, the
`verifyInvariants` function is not generated.

When a type/attribute is not defined in TableGen, but a verifier is
needed, users can implement the `verifyInvariants` function. (This
function was previously called `verify`.)

Note for LLVM integration: If you have an attribute/type that is not
defined in TableGen (i.e., just C++), you have to rename the
verification function from `verify` to `verifyInvariants`. (Most
attributes/types have no verification, in which case there is nothing to
do.)

Depends on #102657.
2024-08-09 22:04:40 +02:00
Peiming Liu
785a24f156
[mlir][sparse] introduce sparse_tensor.coiterate operation. (#101100)
This PR introduces `sparse_tensor.coiterate` operation, which represents
a loop that traverses multiple sparse iteration space.
2024-07-31 15:14:27 -07:00
Peiming Liu
12189f8005
[mlir][sparse] introduce sparse_tensor.extract_value operation. (#101219) 2024-07-30 16:26:18 -07:00
Peiming Liu
a02010b3e9
[mlir][sparse] support sparsifying sparse kernels to sparse-iterator-based loop (#95858) 2024-06-17 16:50:12 -07:00
Peiming Liu
a43d79af78
[mlir][sparse] add canonicalization patterns for IterateOp. (#95569) 2024-06-14 10:31:29 -07:00
Peiming Liu
e276cf0831
[mlir][sparse] introduce sparse_tensor.iterate operation (#88955)
A `sparse_tensor.iterate` iterates over a sparse iteration space
extracted from `sparse_tensor.extract_iteration_space` operation
introduced in https://github.com/llvm/llvm-project/pull/88554.
2024-06-10 10:20:24 -07:00
Peiming Liu
13af97a70e
[mlir][sparse] allow multiple COO segments in sparse encodings. (#91786)
**NOTE**: we still have implementation holes when handling multiple COO
segments in the encoding. But the format should be considered to be
legal.
2024-05-10 11:36:01 -07:00
Yinying Li
83f3b1cb48
[mlir][sparse] Add verification for explicit/implicit value (#90111)
1. Verify that the type of explicit/implicit values should be the same
as the tensor element type.
2. Verify that implicit value could only be zero.
3. Verify that explicit/implicit values should be numeric.
4. Fix the type change issue caused by SparseTensorType(enc).
2024-05-07 20:28:39 -04:00
Yinying Li
e71eacc5b1
[mlir][sparse] Support explicit/implicit value for complex type (#90771) 2024-05-02 12:28:34 -04:00
Yinying Li
a10d67f9fb
[mlir][sparse] Enable explicit and implicit value in sparse encoding (#88975)
1. Explicit value means the non-zero value in a sparse tensor. If
explicitVal is set, then all the non-zero values in the tensor have the
same explicit value. The default value Attribute() indicates that it is
not set.

2. Implicit value means the "zero" value in a sparse tensor. If
implicitVal is set, then the "zero" value in the tensor is equal to the
implicit value. For now, we only support `0` as the implicit value but
it could be extended in the future. The default value Attribute()
indicates that the implicit value is `0` (same type as the tensor
element type).

Example:

```
#CSR = #sparse_tensor.encoding<{
  map = (d0, d1) -> (d0 : dense, d1 : compressed),
  posWidth = 64,
  crdWidth = 64,
  explicitVal = 1 : i64,
  implicitVal = 0 : i64
}>
```

Note: this PR tests that implicitVal could be set to other values as
well. The following PR will add verifier and reject any value that's not
zero for implicitVal.
2024-04-24 16:20:25 -07:00
Christian Sigg
a5757c5b65
Switch member calls to isa/dyn_cast/cast/... to free function calls. (#89356)
This change cleans up call sites. Next step is to mark the member
functions deprecated.

See https://mlir.llvm.org/deprecation and
https://discourse.llvm.org/t/preferred-casting-style-going-forward.
2024-04-19 15:58:27 +02:00
Peiming Liu
481bd5d416
[mlir][sparse] introduce sparse_tensor.extract_iteration_space operation. (#88554)
A `sparse_tensor.extract_space %tensor at %iterator` extracts a *sparse*
iteration space defined `%tensor`, the operation to traverse the
iteration space will be introduced in following PRs.
2024-04-16 11:32:30 -07:00
Peiming Liu
b9556532c7
Revert "[mlir][sparse] introduce sparse_tensor.iterate operation" (#88953)
Reverts llvm/llvm-project#88807 (merged by mistake)
2024-04-16 11:31:33 -07:00
Peiming Liu
8debcf03c5
[mlir][sparse] introduce sparse_tensor.iterate operation (#88807)
A `sparse_tensor.iterate` iterates over a sparse iteration space
extracted from `sparse_tensor.extract_iteration_space` operation
introduced in https://github.com/llvm/llvm-project/pull/88554.

*DO NOT MERGE* before https://github.com/llvm/llvm-project/pull/88554
2024-04-16 11:31:09 -07:00
Peiming Liu
62fa12ad24
[mlir][sparse] support querying sparse buffer types from sparse tenso… (#88308)
…r encodings.
2024-04-11 10:26:03 -07:00
Peiming Liu
a54930e696
[mlir][sparse] allow YieldOp to yield multiple values. (#87261) 2024-04-01 10:30:36 -07:00
Justin Fargnoli
513cdb8222
[mlir] Declare promised interfaces for all dialects (#78368)
This PR adds promised interface declarations for all interfaces declared
in `InitAllDialects.h`.

Promised interfaces allow a dialect to declare that it will have an
implementation of a particular interface, crashing the program if one
isn't provided when the interface is used.
2024-03-15 20:23:20 -07:00
Peiming Liu
94e27c265a
[mlir][sparse] reuse tensor.insert operation to insert elements into … (#84987)
…a sparse tensor.
2024-03-12 16:59:17 -07:00
Yinying Li
ced1fac8a3
[mlir][sparse] Move n:m printing into toMLIRString (#84264) 2024-03-07 16:13:45 -05:00
Peiming Liu
52b69aa32f
[mlir][sparse] support sparsifying batch levels (#83898) 2024-03-04 14:39:06 -08:00
Aart Bik
e10dc60ad7
[mlir][sparse][nfc] fixed typo in "translate" (#83891) 2024-03-04 12:57:48 -08:00
Peiming Liu
6bc7c9df7f
[mlir][sparse] infer returned type for sparse_tensor.to_[buffer] ops (#83343)
The sparse structure buffers might not always be memrefs with rank == 1
with the presence of batch levels.
2024-02-28 16:10:20 -08:00
Peiming Liu
0d1f95760b
[mlir][sparse] support type conversion from batched sparse tensors to… (#83163)
… memrefs.
2024-02-27 12:05:28 -08:00
Peiming Liu
56d58295dd
[mlir][sparse] Introduce batch level format. (#83082) 2024-02-26 16:08:28 -08:00
Peiming Liu
5248a98724
[mlir][sparse] support SoA COO in codegen path. (#82439)
*NOTE*: the `SoA` property only makes a difference on codegen path, and
is ignored in libgen path at the moment (only SoA COO is supported).
2024-02-20 17:06:21 -08:00
Peiming Liu
f740366fa6
[mlir][sparse] support type conversion from SoA COO to memrefs. (#82398) 2024-02-20 11:19:13 -08:00
Peiming Liu
088c7ce429
[mlir][sparse] introduce SoA level property on singleton level. (#81942) 2024-02-15 16:41:10 -08:00
Peiming Liu
aaf916456a
Reapply "[mlir][sparse] remove LevelType enum, construct LevelType from LevelFormat and Properties" (#81923) (#81934) 2024-02-15 14:48:52 -08:00
Mehdi Amini
513448d28e
Revert "[mlir][sparse] remove LevelType enum, construct LevelType from LevelF…" (#81923)
Reverts llvm/llvm-project#81799 ; this broke the mlir gcc7 bot.
2024-02-15 13:26:44 -08:00
Peiming Liu
235ec0f791
[mlir][sparse] remove LevelType enum, construct LevelType from LevelF… (#81799)
…ormat and properties instead.
2024-02-15 12:31:03 -08:00
Yinying Li
2a6b521b36
[mlir][sparse] Add more tests and verification for n:m (#81186)
1. Add python test for n out of m
2. Add more methods for python binding
3. Add verification for n:m and invalid encoding tests
4. Add e2e test for n:m

Previous PRs for n:m #80501 #79935
2024-02-09 14:34:36 -05:00