This patch disables 2 bf16 tests that are currently not supported on
AArch64. I've triaged these failures and opened [1] to track this. I
don't have a simple reproducer for dense_output_bf16.mlir, but it's
rather clear that both tests fail due to missing support for `bfloat`
operations in the AArch64 backend.
I'm not sure what the path forward to enable these tests on AArch64
should be. I think that there are two options:
* AArch64 backened gains capability to legalize these nodes containing
`bfloat` operands, or
* MLIR (similarly to Clang) is taught not to emit such nodes in the
first place.
[1] https://github.com/llvm/llvm-project/issues/58465
Differential Revision: https://reviews.llvm.org/D136273
Handle more cases of singleton DLT including direct sparse2sparse conversion. (Followup to D134096)
Depends On D134926
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D134933
This extension to the sparse tensor type system in MLIR
opens up a whole new set of sparse storage schemes, such as
block sparse storage (e.g. BCSR) and ELL (aka jagged diagonals).
This revision merely introduces the type extension and
initial documentation. The actual interpretation of the type
(reading in tensors, lowering to code, etc.) will follow.
Reviewed By: Peiming
Differential Revision: https://reviews.llvm.org/D135206
tensor.empty/linalg.init_tensor produces an uninititalized tensor that can be used as a destination operand for destination-style ops (ops that implement `DestinationStyleOpInterface`).
This change makes it possible to implement `TilingInterface` for non-destination-style ops without depending on the Linalg dialect.
RFC: https://discourse.llvm.org/t/rfc-add-tensor-from-shape-operation/65101
Differential Revision: https://reviews.llvm.org/D135129
The region within sparse_tensor.select is used as the runtime criteria
for whether to keep the existing value in the sparse tensor.
While the sparse element is provided to the comparison, indices may also
be used to decide on whether to keep the original value. This allows, for
example, to only keep the upper triangle of a matrix.
Reviewed by: aartbik
Differential Revision: https://reviews.llvm.org/D134761
This is a first step towards fully implementing the new dimension
level types and properties, illustrating with a fully functional
sorted COO of any dimension. Note that the sparsification part is
pretty complete. The required parts in the runtime support library
have been kept to a minimum, to avoid huge conflicts with Wren's
ongoing refactoring. The missing parts will be filled in later.
Reviewed By: Peiming
Differential Revision: https://reviews.llvm.org/D134096
Add sparse-buffer-rewrite pass to rewrite sparse primitives on buffers to MLIR
implementation.
Add sparse rewrite rule for the sort operator.
Add FileCheck test and integration test.
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D134627
Previously, the iteration graph is computed without priority. This patch add a heuristic when computing the iteration graph by starting with Reduction iterator when doing topo sort, which makes Reduction iterators (likely) appear as late in the sorted array as possible.
The current sparse compiler also failed to compile the newly added case.
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D133738
Add new option (enable-runtime-library) to sparse compiler pipeline, it allows us to decide whether we need to rewrite operations (e.g., concatenate, reshape) within sparsification (when using codegen) or convert them after sparsification (when using runtime library).
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D133597
Access pattern expansion is always done along the innermost stored
dimension, but this was incorrectly reordered due to using a
general utility typically used by original dimensions only.
Reviewed By: bixia
Differential Revision: https://reviews.llvm.org/D133472
The "sparsification" pass does not need the ability to use runtime values for
the dimension, so the only source for variability would have been user code.
Restricting the dimension to constants simplifies code generation.
Reviewed By: Peiming, wrengr
Differential Revision: https://reviews.llvm.org/D133458
bufferization.to_memref ops are not supported in One-Shot Analysis. They often trigger a failed assertion that can be confusing. Instead, scan for to_memref ops before running the analysis and immediately abort with a proper error message.
Differential Revision: https://reviews.llvm.org/D132027
Implement the new sparse_tensor.reduce operation which
accepts a starting identity value and a code block
describing how to perform the reduction.
Reviewed by: aartbik
Differential Revision: https://reviews.llvm.org/D130573
The end-to-end test for this new feature also exposed a bug
in LLVM IR lowering (since then, fixed), where we need to account
for the min-poison bit as extra argument.
declare i32 @llvm.abs.i32(i32 <src>, i1 <is_int_min_poison>)
Reviewed By: bixia
Differential Revision: https://reviews.llvm.org/D131712
c7ec6e19d5446a448f888b33f66316cf2ec6ecae made LLVM adhere to the x86
psABI and pass bf16 in SSE registers instead of GPRs. This breaks the
custom versions of runtime functions we have for bf16 conversion. A
great fix for this would be to use __bf16 types instead which carry the
right ABI, but that type isn't widely available.
Instead just pretend it's a 32 bit float on the ABI boundary and
carefully cast it to the right type.
Fixes#57042
Currently, dense tensors are initialized in Sparse Integration tests using
"buffer.tensor_alloc and scf.for" . This makes code harder to read and maintain.
This diff uses tensor.generate instead to initialize dense tensors.
Testing: Ran integration tests after building with -DLLVM_USE_SANITIZER=Address flag.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D131404
This rewriting was no longer functional after recent migration to one shot
bufferization. However, this revision makes it work again, with a CHECK test
to ensure fusion happens. Note that functionality is tested by several
integration tests.
Reviewed By: Peiming
Differential Revision: https://reviews.llvm.org/D130996
Sparse compiler failed on the provided test (when the sparse kernel is nested in a scf structrual operator).
Reviewed By: bixia
Differential Revision: https://reviews.llvm.org/D130609
https://reviews.llvm.org/D130023 added a memory leak in sparse_sampled_matmul.mlir
This diff fixes the memory leak.
Testing: Ran integration tests after building with -DLLVM_USE_SANITIZER=Address flag.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D130428
