67 Commits

Author SHA1 Message Date
bixia1
90aa436291 [mlir][sparse] Add layout to the memref for the indices buffers to prepare for the AOS storage optimization for COO regions.
Fix relevant FileCheck tests.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D140742
2023-01-04 07:36:11 -08:00
Peiming Liu
781eabeb40 [mlir][sparse] refactoring loop emitter into its own files.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D140701
2022-12-27 19:12:05 +00:00
Peiming Liu
a3672add76 [mlir][sparse] avoid unnecessary tmp COO buffer and convert when lowering ConcatentateOp.
When concat along dim 0, and all inputs/outputs are ordered with identity dimension ordering,
the concatenated coordinates will be yield in lexOrder, thus no need to sort.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D140228
2022-12-16 18:26:39 +00:00
Peiming Liu
b4e2b7f90c [mlir][sparse] avoid sorting when unnecessary when convert sparse tensors.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D139744
2022-12-10 00:24:42 +00:00
Aart Bik
65074179f2 [mlir][sparse] make fusion for SDDMM more robust
Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D139456
2022-12-06 14:32:19 -08:00
Kazu Hirata
1a36588ec6 [mlir] Use std::nullopt instead of None (NFC)
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated.  The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.

This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-12-03 18:50:27 -08:00
wren romano
2af2e4dbb7 [mlir][sparse] Breaking up openSparseTensor to better support non-permutations
This commit updates how the `SparseTensorConversion` pass handles `NewOp`.  It breaks up the underlying `openSparseTensor` function into two parts (`SparseTensorReader::create` and `SparseTensorReader::readSparseTensor`) so that the pass can inject code for constructing `lvlSizes` between those two parts.  Migrating the construction of `lvlSizes` out of the runtime and into the pass is a necessary first step toward fully supporting non-permutations.  (The alternative would be for the pass to generate a `FuncOp` for performing the construction and then passing that to the runtime; which doesn't seem to have any benefits over the design of this commit.)  And since the pass now generates the code to call these two functions, this change also removes the `Action::kFromFile` value from the enum used by `_mlir_ciface_newSparseTensor`.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D138363
2022-12-02 11:10:57 -08:00
bixia1
101a0c84f7 [mlir][sparse] Improve concatenate operator rewrite for annotated all dense dimensions results.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D138823
2022-11-29 12:25:52 -08:00
bixia1
aedf5d5831 [mlir][sparse] Improve concatenate operator rewriting for dense tensor results.
Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D138465
2022-11-28 07:56:01 -08:00
bixia1
974b4bf9fd [mlir][sparse] Add expand_symmetry attribute to the new operator.
The attribute tells the operator to handle symmetric structures for 2D tensors.
By default, the operator assumes the input tensor is not symmetric.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D138230
2022-11-23 16:32:15 -08:00
bixia1
6c01b5cdad [mlir][sparse] Fix a bug in concatenate operator rewriting.
When calculating the dynamic dimensions for the concatenate result, we
shouldn't accumulate the sizes for the non-concatenating dimensions.

Reviewed By: aartbik, Peiming

Differential Revision: https://reviews.llvm.org/D138436
2022-11-22 08:17:35 -08:00
Aliia Khasanova
399638f98c Merge kDynamicSize and kDynamicSentinel into one constant.
resolve conflicts

Differential Revision: https://reviews.llvm.org/D138282
2022-11-21 13:01:26 +00:00
bixia1
96b3bf4292 [mlir][sparse] Fix a problem in the new operator rewriter.
The getSparseTensorReaderNextX functions should return void.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D138226
2022-11-17 13:21:10 -08:00
Peiming Liu
8d615a23ef [mlir][sparse] fix crash on sparse_tensor.foreach operation on tensors with complex<T> elements.
Reviewed By: aartbik, bixia

Differential Revision: https://reviews.llvm.org/D138223
2022-11-17 19:36:15 +00:00
bixia1
c374ef2eb7 [mlir][sparse] Extend the operator new rewriter to handle isSymmetric flag.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D138214
2022-11-17 10:48:24 -08:00
bixia1
f81f0cb75a [mlir][sparse] Split SparseTensorRewrite into PreSparsificationRewrite and PostSparsificationRewrite.
Reviewed By: aartbik, wrengr

Differential Revision: https://reviews.llvm.org/D138153
2022-11-17 07:13:55 -08:00
Peiming Liu
91e7b9e525 [mlir][sparse] annotate loops that are generated by loop emitter.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D138155
2022-11-17 00:09:33 +00:00
Peiming Liu
61e5c14fa8 [mlir][sparse] fix memory leakage in concatenate rewriter.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D138074
2022-11-16 00:10:33 +00:00
Aart Bik
0e1708ff64 [mlir][sparse] cleanup small vector constant hints
Following advise from

https://llvm.org/docs/ProgrammersManual.html#llvm-adt-smallvector-h

This revision removes the size hints from SmallVector (unless we are
certain of the resulting number of elements). Also, this replaces
SmallVector references with SmallVectorImpl references.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D138063
2022-11-15 15:09:06 -08:00
bixia1
555e7835f4 [mlir][sparse] Fix rewriting for convert op and concatenate op.
Fix a problem in convert op rewriting where it used the original index for
ToIndicesOp.

Extend the concatenate op rewriting to handle dense destination and dynamic
shape destination.

Make the concatenate op integration test run on the codegen path.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D138057
2022-11-15 14:45:21 -08:00
Peiming Liu
c3821e1684 [mlir][sparse] fix bugs in concatenate rewriter.
Reviewed By: aartbik, bixia

Differential Revision: https://reviews.llvm.org/D138053
2022-11-15 19:49:59 +00:00
Peiming Liu
8ffdcc594e [mlir][sparse] fix memory leak sparse2sparse reshape
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137994
2022-11-15 00:19:51 +00:00
bixia1
fa46de16db [mlir][sparse][NFC] Add comments to tests that are run for with and without runtime libraries.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137869
2022-11-14 08:21:33 -08:00
Peiming Liu
725e0849b7 [mlir][sparse] fix incorrect coordinates ordering computed by the foreach operation.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137877
2022-11-12 04:08:50 +00:00
bixia1
57416d872a [mlir][sparse] Fix a bug in rewriting dense2dense convert op.
Permutation wasn't handled correctly. Add a test for the rewriting.

Extend an integration test to run with enable_runtime_library=false to
also test the rewriting.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D137845
2022-11-11 09:11:54 -08:00
bixia1
ccd1a5b9cb [mlir][sparse] Fix a bug in rewriting for the convert op.
The code to retrieve the number of entries isn't correct.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D137795
2022-11-10 14:30:51 -08:00
Aart Bik
e6cbb91483 [mlir][sparse] skip zeros during dense2sparse
This enables the full matmul integration test with runtime_lib=true/false!

Background:
https://github.com/llvm/llvm-project/issues/51657

Reviewed By: bixia

Differential Revision: https://reviews.llvm.org/D137750
2022-11-09 20:54:27 -08:00
Aart Bik
a61a9a700a [mlir][sparse] first end-to-end matmul with codegen
(1) also fixes memory leak in sparse2dense rewriting
(2) still needs fix in dense2sparse by skipping zeros

Reviewed By: wrengr

Differential Revision: https://reviews.llvm.org/D137736
2022-11-09 13:40:05 -08:00
Peiming Liu
f45c21da0c [mlir][sparse] fix SSA chain issue in dense2sparse conversion.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137683
2022-11-09 02:03:33 +00:00
Peiming Liu
7175f9dde1 [mlir][sparse] extend foreach operation to iterator over sparse constant.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137679
2022-11-09 01:50:34 +00:00
Peiming Liu
4fa00ce15c [mlir][sparse] extend foreach operation to accept reduction arguments; fix sparse tensor rewriting patterns that do not propagate sparse tensor SSA properly.
This patch re-commit D137468 and D137463, which were reverted by mistakes.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137579
2022-11-07 21:40:30 +00:00
Stella Stamenova
99171078bb Revert "[mlir][sparse] extend foreach operation to accept reduction arguments."
This reverts commit 53d5d3401120f2aa741a73a5a9ba0ce012ca532c.

This is causing a build failure on the windows mlir bot that was previously hidden by another sparse tensor change that caused failures:

https://lab.llvm.org/buildbot/#/builders/13/builds/28006
2022-11-07 09:34:10 -08:00
Stella Stamenova
ec224e3b68 Revert "[mlir][sparse] fix sparse tensor rewriting patterns that do not propagate sparse tensor SSA properly."
This reverts commit 70508b614e6478ba2c3fc79e935e2c68e2d79b71.

This change depends on a reverted change that broke the windows mlir buildbot; reverting to bring remaining mlir bots to green
2022-11-07 09:00:08 -08:00
Peiming Liu
70508b614e [mlir][sparse] fix sparse tensor rewriting patterns that do not propagate sparse tensor SSA properly.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137468
2022-11-04 23:43:54 +00:00
Peiming Liu
53d5d34011 [mlir][sparse] extend foreach operation to accept reduction arguments.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137463
2022-11-04 23:34:16 +00:00
bixia1
3e724e529b [mlir][sparse] Fix windows build.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D137210
2022-11-01 17:00:02 -07:00
bixia1
eb877006a6 [mlir][sparse] Add rewriting rule for the convert operator.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D136301
2022-11-01 15:57:34 -07:00
Peiming Liu
63baab8b39 [mlir][sparse] Fold invariant op only when it has only one use.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D136990
2022-10-28 23:42:28 +00:00
Alexander Belyaev
b4db15a949 [mlir] Rename getInputs->getDpsInputs and getOutputs->getDpsInits in DPS interface.
https://discourse.llvm.org/t/rfc-interface-for-destination-style-ops/64056

Differential Revision: https://reviews.llvm.org/D136943
2022-10-28 15:41:12 +02:00
Peiming Liu
b0f8057e4c [mlir][sparse] use loop emitter to generate loop in sparsification
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D136185
2022-10-26 00:27:56 +00:00
Jeff Niu
dc63ca78f7 [mlir][sparse] Fix build warning (NFC) 2022-10-21 13:13:07 -07:00
bixia1
e445349d2c [mlir][sparse] Add rewrite rule for the sparse_tensor.out operator.
Also fix the rewrite rule for sparse_tensor.new to reflect the recent change of
the runtime C interface and to use utilities for memref.alloca.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D135891
2022-10-21 12:20:53 -07:00
Peiming Liu
35b3a0ce8d [mlir][sparse] support foreach on dense tensor.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D136384
2022-10-21 00:12:37 +00:00
bixia1
cf7906cbbd [mlir][sparse] Fix getUnorderedCOOFromType for rank 1 tensor.
Previously, it used DimLevelType::SingletonNo to represent an unorder COO
tensor of rank 1 while it should use DimLevelType::CompressedNuNo.

Reviewed By: Peiming, wrengr

Differential Revision: https://reviews.llvm.org/D136387
2022-10-20 16:33:01 -07:00
wren romano
0e77b63bc0 [mlir][sparse] Use the runtime DimLevelType instead of a separate tablegen enum
This differential replaces all uses of SparseTensorEncodingAttr::DimLevelType with DimLevelType.  The next differential will break out a separate library for the DimLevelType enum, so that the Dialect code doesn't need to depend on the rest of the runtime

Depends On D135995

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D135996
2022-10-18 13:45:26 -07:00
bixia1
c1864ab953 [mlir][sparse] Add options to sparse-tensor-rewrite to disable rewriting rules for operators foreach and convert.
This is to help simplify FileCheck tests for sparse-tensor-rewrite.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D136093
2022-10-18 09:27:32 -07:00
bixia1
933e7e2393 [mlir][sparse] Change getUnorderedCOOFromType to propagate the overhead types.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D136112
2022-10-18 08:01:16 -07:00
Alexander Belyaev
a7cccb9cbb [mlir] Simplify DestinationStyleOpInterface.
Differential Revision: https://reviews.llvm.org/D135348
2022-10-17 12:43:41 +02:00
Oleg Shyshkov
c38d9cf20e [mlir] Remove iterator_types() method from LinalgStructuredInterface.
`getIteratorTypesArray` should be used instead. It's a better substitute for all the current usages of the interface.

The current `ArrayAttr iterator_types()` has a few problems:
* It creates an assumption operation has iterators types as an attribute, but it's not always the case. Sometime iterator types can be inferred from other attribute, or they're just static.
* ArrayAttr is an obscure contained and required extracting values in the client code.
* Makes it hard to migrate iterator types from strings to enums ([RFC](https://discourse.llvm.org/t/rfc-enumattr-for-iterator-types-in-linalg/64535/9)).

Concrete ops, like `linalg.generic` will still have iterator types as an attribute if needed.

As a side effect, this change helps a bit with migration to prefixed accessors.

Differential Revision: https://reviews.llvm.org/D135765
2022-10-13 07:52:43 +00:00
Kazu Hirata
b2674de375 [mlir] Fix a warning
This patch fixes:

  mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorRewriting.cpp:587:27:
  error: comparison of integers of different signs: 'int64_t' (aka
  'long') and 'uint64_t' (aka 'unsigned long')
  [-Werror,-Wsign-compare]
2022-10-12 15:25:17 -07:00