647 Commits

Author SHA1 Message Date
Peiming Liu
db7f639b90 [mlir][sparse] fix a crash when generating sparse convolution with nchw input
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D151744
2023-05-30 20:16:54 +00:00
Kun Wu
235fbe792b [mlir] [sparse] [gpu] adding transpose support to spmm spmv
Reviewed By: aartbik, wrengr

Differential Revision: https://reviews.llvm.org/D151259
2023-05-26 17:07:09 +00:00
Tres Popp
68f58812e3 [mlir] Move casting calls from methods to function calls
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.

Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.

Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
  for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443

Implementation:
This patch updates all remaining uses of the deprecated functionality in
mlir/. This was done with clang-tidy as described below and further
modifications to GPUBase.td and OpenMPOpsInterfaces.td.

Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
   additional check:
   main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
   and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
   them to a pure state.

```
ninja -C $BUILD_DIR clang-tidy

run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
               -header-filter=mlir/ mlir/* -fix

rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```

Differential Revision: https://reviews.llvm.org/D151542
2023-05-26 10:29:55 +02:00
Aart Bik
22caafc9f3 [mlir][sparse][gpu] end to end test for matmul
(1) minor bug fix in copy back [always nice to run stuff ;-)]
(2) run with and without lib (even though some fall back to CPU)

Reviewed By: wrengr

Differential Revision: https://reviews.llvm.org/D151507
2023-05-25 16:10:22 -07:00
Peiming Liu
f7b8b005ff [mlir][sparse] fix bugs when computing the memory size when lowering pack op.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D151481
2023-05-25 19:19:52 +00:00
Matthias Springer
3d90c8126f [mlir][sparse] Fix incorrect API usage in RewritePatterns
Incorrect API usage was detected by
`MLIR_ENABLE_EXPENSIVE_PATTERN_API_CHECKS`.

Differential Revision: https://reviews.llvm.org/D151302
2023-05-25 08:52:47 +02:00
Aart Bik
bcb698bfdc [mlir][sparse][gpu] various cuSparse refinements
(1) keep all cuSparse ops on single stream without wait() in right order
(2) use more type precise memref types for COO
(3) use ToTensor on resulting memref (even though it folds away again)

Reviewed By: K-Wu

Differential Revision: https://reviews.llvm.org/D151404
2023-05-24 22:32:52 -07:00
Kun Wu
86bf710cf7 [mlir] [gpu] [sparse] refined SparseHandle type
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D151014
2023-05-24 10:16:07 -07:00
Peiming Liu
b2e6b73544 [mlir][sparse] extend unpack operation to unpack arbitrary encodings.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D151174
2023-05-23 22:34:01 +00:00
Aart Bik
b75d6a40f1 [mlir][sparse][gpu] recognize SpMM cuSparse during sparsification
Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D150715
2023-05-19 17:22:59 -07:00
wren romano
d824af535c [mlir][sparse] Improving error-detection for STEA::get{Pos,Crd}Type
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D150995
2023-05-19 16:24:38 -07:00
Peiming Liu
de56088866 [mlir][sparse] Support packing external data into arbitrary sparse tensor encoding.
We previously only support packing two array (values and coordinates) into COO tensors.
This patch allows packing inputs into arbitrary sparse tensor format.

It also deletes the "implicit" data canonicalization performed inside sparse compiler,
but instead requires users to canonicalize the data before passing it to the sparse compiler.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D150916
2023-05-19 17:41:49 +00:00
wren romano
867e19648d [mlir][sparse] Adding SparseTensorEncodingAttr::getDimSlice
This helps catch segfaults and OOB.

Reviewed By: aartbik, Peiming

Differential Revision: https://reviews.llvm.org/D150917
2023-05-18 17:06:01 -07:00
wren romano
d9a2f89bee [mlir][sparse] Adjusting error message wording, to better match new field names
This is a followup to D150330, split out because it's not purely mechanical.

Depends On D150330

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D150409
2023-05-18 15:02:08 -07:00
Peiming Liu
afe78db770 [mlir][sparse] Make sparse_tensor::StorageLayout publicly available.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D150739
2023-05-18 20:29:46 +00:00
wren romano
50a7c77389 [mlir][sparse] Calling SparseTensorType::get{Pos,Crd}Type directly
The `SparseTensorType` versions of these methods have some special handling to ensure that they work for unannotated tensors; whereas, the `stt.getEncoding().get{Pos,Crd}Type()` idiom can cause segfaults.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D150815
2023-05-18 12:28:54 -07:00
wren romano
a0615d020a [mlir][sparse] Renaming the STEA field dimLevelType to lvlTypes
This commit is part of the migration of towards the new STEA syntax/design.  In particular, this commit includes the following changes:
* Renaming compiler-internal functions/methods:
  * `SparseTensorEncodingAttr::{getDimLevelType => getLvlTypes}`
  * `Merger::{getDimLevelType => getLvlType}` (for consistency)
  * `sparse_tensor::{getDimLevelType => buildLevelType}` (to help reduce confusion vs actual getter methods)
* Renaming external facets to match:
  * the STEA parser and printer
  * the C and Python bindings
  * PyTACO

However, the actual renaming of the `DimLevelType` itself (along with all the "dlt" names) will be handled in a separate commit.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D150330
2023-05-17 14:24:09 -07:00
Anlun Xu
6116ca67ab [mlir][sparse] Add sparse rewriting rules for tensor::ReshapeOp
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D149564
2023-05-16 14:56:33 -07:00
Aart Bik
195621aacb [mlir][sparse][gpu] set cubin flag when building for cuda
Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D150692
2023-05-16 10:50:22 -07:00
Peiming Liu
ad469385ab [mlir][sparse] Add a helper class to help lowering operations with/without function calls
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D150477
2023-05-16 17:22:44 +00:00
Aart Bik
ee42e23614 [mlir][sparse][gpu] first implementation of the GPU libgen approach
The sparse compiler now has two prototype strategies for GPU acceleration:

* CUDA codegen: this converts sparsified code to CUDA threads
* CUDA libgen: this converts pre-sparsified code to cuSPARSE library calls

This revision introduces the first steps required for the second approach.

Reviewed By: ThomasRaoux

Differential Revision: https://reviews.llvm.org/D150170
2023-05-15 08:49:38 -07:00
Tres Popp
c1fa60b4cd [mlir] Update method cast calls to function calls
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.

Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.

Context:

* https://mlir.llvm.org/deprecation/ at "Use the free function variants for dyn_cast/cast/isa/…"
* Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443

Implementation:
This follows a previous patch that updated calls
`op.cast<T>()-> cast<T>(op)`. However some cases could not handle an
unprefixed `cast` call due to occurrences of variables named cast, or
occurring inside of class definitions which would resolve to the method.
All C++ files that did not work automatically with `cast<T>()` are
updated here to `llvm::cast` and similar with the intention that they
can be easily updated after the methods are removed through a
find-replace.

See https://github.com/llvm/llvm-project/compare/main...tpopp:llvm-project:tidy-cast-check
for the clang-tidy check that is used and then update printed
occurrences of the function to include `llvm::` before.

One can then run the following:
```
ninja -C $BUILD_DIR clang-tidy

run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
                 -export-fixes /tmp/cast/casts.yaml mlir/*\
                 -header-filter=mlir/ -fix

rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```

Differential Revision: https://reviews.llvm.org/D150348
2023-05-12 11:21:30 +02:00
Tres Popp
5550c82189 [mlir] Move casting calls from methods to function calls
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.

Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.

Caveats include:
- This clang-tidy script probably has more problems.
- This only touches C++ code, so nothing that is being generated.

Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
  for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443

Implementation:
This first patch was created with the following steps. The intention is
to only do automated changes at first, so I waste less time if it's
reverted, and so the first mass change is more clear as an example to
other teams that will need to follow similar steps.

Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
   additional check:
   https://github.com/llvm/llvm-project/compare/main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
   and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
   them to a pure state.
4. Some changes have been deleted for the following reasons:
   - Some files had a variable also named cast
   - Some files had not included a header file that defines the cast
     functions
   - Some files are definitions of the classes that have the casting
     methods, so the code still refers to the method instead of the
     function without adding a prefix or removing the method declaration
     at the same time.

```
ninja -C $BUILD_DIR clang-tidy

run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
               -header-filter=mlir/ mlir/* -fix

rm -rf $BUILD_DIR/tools/mlir/**/*.inc

git restore mlir/lib/IR mlir/lib/Dialect/DLTI/DLTI.cpp\
            mlir/lib/Dialect/Complex/IR/ComplexDialect.cpp\
            mlir/lib/**/IR/\
            mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp\
            mlir/lib/Dialect/Vector/Transforms/LowerVectorMultiReduction.cpp\
            mlir/test/lib/Dialect/Test/TestTypes.cpp\
            mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp\
            mlir/test/lib/Dialect/Test/TestAttributes.cpp\
            mlir/unittests/TableGen/EnumsGenTest.cpp\
            mlir/test/python/lib/PythonTestCAPI.cpp\
            mlir/include/mlir/IR/
```

Differential Revision: https://reviews.llvm.org/D150123
2023-05-12 11:21:25 +02:00
Aart Bik
1e0966cd6c [mlir][sparse] add util for ToCoordinatesBuffer for COO AoS
Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D150382
2023-05-11 10:43:31 -07:00
Peiming Liu
0e1429add5 [mlir][sparse] fix build error.
Reviewed By: aartbik, vzakhari

Differential Revision: https://reviews.llvm.org/D149874
2023-05-04 17:30:13 +00:00
Peiming Liu
36c95ee739 [mlir][sparse] group tensor id and levels into pairs in loop emitter
This addressed some unresolved comments in https://reviews.llvm.org/D142930

Reviewed By: aartbik, wrengr

Differential Revision: https://reviews.llvm.org/D148565
2023-05-04 16:15:42 +00:00
wren romano
55a74af71b [mlir][sparse] Adding a new overload of getOpaquePointerType
(This will be used in future patches, but is split off for easier reviewing)

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D149805
2023-05-03 18:39:20 -07:00
Aart Bik
9a018a7b48 [mlir][sparse] relax constraints on tensor.cast with pre-rewriting
Reviewed By: wrengr

Differential Revision: https://reviews.llvm.org/D149489
2023-05-01 16:03:44 -07:00
Peiming Liu
4ca1932776 [mlir][sparse] fix compiler warning in release build.
Differential Revision: https://reviews.llvm.org/D149607
2023-05-01 19:02:43 +00:00
Peiming Liu
d4db528938 [mlir][sparse] extend unpack operation to support unpacking a batched COO type
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D149103
2023-05-01 18:17:29 +00:00
Aart Bik
dbdb4affa0 [mlir][sparse] avoid slice rewriting when conditions are not met
Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D148964
2023-04-21 16:09:08 -07:00
Aart Bik
86888e420c [mlir][sparse][gpu] generate proper memcpy in/out host and device
The host registration is a convenient way to get CUDA kernels
running, but it may be slow and does not work for all buffer
(like global constants). This revision uses the proper alloc
copy dealloc chains for buffers, using asynchronous chains
to increase overlap. The host registration mechanism is
kept under a flag for the output, just for experimentation
purposes while this project ramps up.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D148682
2023-04-21 09:30:42 -07:00
Peiming Liu
a7cfcc686b [mlir][sparse] fix crash when generating coiteration loop with compressed-hi DLT.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D148842
2023-04-20 21:15:49 +00:00
Rahul Kayaith
6089d612a5 [mlir] Prevent implicit downcasting to interfaces
Currently conversions to interfaces may happen implicitly (e.g.
`Attribute -> TypedAttr`), failing a runtime assert if the interface
isn't actually implemented. This change marks the `Interface(ValueT)`
constructor as explicit so that a cast is required.

Where it was straightforward to I adjusted code to not require casts,
otherwise I just made them explicit.

Depends on D148491, D148492

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D148493
2023-04-20 16:31:54 -04:00
Peiming Liu
fd2211d84a use heap memory for position buffer allocated for PackOp.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D148818
2023-04-20 20:26:01 +00:00
Matthias Springer
4c48f016ef [mlir][Affine][NFC] Wrap dialect in "affine" namespace
This cleanup aligns the affine dialect with all the other dialects.

Differential Revision: https://reviews.llvm.org/D148687
2023-04-20 11:19:21 +09:00
Peiming Liu
7864d736cf [mlir][sparse] extend pack operation to support packing a batched COO type
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D148670
2023-04-20 01:35:30 +00:00
Peiming Liu
abd66d918a [mlir][sparse] support iteration over compressed-hi dimension level in loop emitter
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D148668
2023-04-20 00:57:08 +00:00
Peiming Liu
b9589545c4 [mlir][sparse] introduce a new compressed(hi) dimension level type
`compressed(hi)` is similar to `compressed`, but instead of reusing the previous position high as the current position low, it uses a pair of positions for each sparse index.

The patch only introduces the definition (syntax) but does not provide codegen implementation.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D148664
2023-04-18 23:26:11 +00:00
Peiming Liu
6a148c5aa7 [mlir][sparse] enable more sparse convolution kernels.
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D147670
2023-04-17 17:43:52 +00:00
Peiming Liu
65aaabc8d1 [mlir][sparse] use macro to build common operations in LoopEmitter
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D147553
2023-04-17 16:57:50 +00:00
Peiming Liu
2cd15925f4 [mlir][sparse] implement index redution on dense level (for CSR)
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D147550
2023-04-17 16:36:31 +00:00
Aart Bik
4889214a48 [mlir][sparse][gpu] generate single module, unique kernel names
This fixes a TODO in the first version.

Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D148406
2023-04-15 17:25:36 -07:00
Peiming Liu
5fd9d80135 [mlir][sparse] extend loop emitter to emit slice driven loops
Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D142930
2023-04-13 03:29:40 +00:00
Oleg Shyshkov
75ef84bf52 [mlir][bufferization] Make function boundary type convertion logic dynamic.
Having to choose from only static or dynamic layout for all function is limiting.

Differential Revision: https://reviews.llvm.org/D148074
2023-04-12 11:02:43 +02:00
wren romano
9d4df97ff0 [mlir][sparse] Canonicalizing arguments to genReshapeDstShape and foreachInSparseConstant
These functions don't need a`PatternRewriter`, they only need an `OpBuilder`.  And, the builder should be the first argument, before the `Location`, to match the style used everywhere else in MLIR.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D148059
2023-04-11 19:11:59 -07:00
Frederik Gossen
4a33c67995 Fix invalid op result access in SparseTensorCodegen
This trigges an assertion introduced in https://reviews.llvm.org/D147883

Differential Revision: https://reviews.llvm.org/D147950
2023-04-10 12:36:26 -04:00
wren romano
686ef4b4a8 [mlir][sparse] Simplifying Merger::expContainsTensor
`expContainsTensor` used to call `expIsTensor` to short-circuit the recursive calls; however, the very first thing `expContainsTensor` does is to check `expIsTensor`, so the short-circuiting code just causes the function to check that condition redundantly.

Depends On D146684

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D146688
2023-04-07 12:32:47 -07:00
Aart Bik
61c1ed86a5 [mlir][sparse][gpu] sparse GPU code generator pipeline setup
Reviewed By: Peiming

Differential Revision: https://reviews.llvm.org/D147571
2023-04-05 15:12:54 -07:00
Aart Bik
19466ebc7f [mlir][sparse][gpu] a first prototype sparse GPU code generator
This implements a proof-of-concept GPU code generator
to the sparse compiler pipeline, currently only capable
of generating CUDA threads for outermost parallel loops.

The objective, obviously, is to grow this concept
to a full blown GPU code generator, capable of the
right combinaton of code generation as well as exploiting
idiomatic kernels or vector specific libraries (think cuSparse).

Reviewed By: ThomasRaoux

Differential Revision: https://reviews.llvm.org/D147483
2023-04-05 11:32:06 -07:00