We currently only support one level of aliases, which isn't great
in situations where an attribute/type can have multiple duplicated
components nested within it(e.g. debuginfo metadata). This commit
refactors alias generation to support nested aliases, which requires
changing alias grouping to take into account the depth of child
aliases, to ensure that attributes/types aren't printed before the
aliases they use.
The only real user facing change here was that we no longer print
0 as an alias suffix, which would be unnecessarily expensive to keep
in the new alias generation method (and isn't that valuable of a
behavior to preserve).
Differential Revision: https://reviews.llvm.org/D136541
This patch introduces the `vector.mask` operation and the MaskableOpInterface
as described in https://discourse.llvm.org/t/rfc-vector-masking-representation-in-mlir/64964.
The `vector.mask` operation is used to predicate the execution of operations
implementing the MaskableOpInterface. This interface will be implemented by maskable
operations and provides information about its masking constraints and semantics.
For now, only vector transfer and reduction ops implement the MaskableOpInterface
for illustration and testing purposes.
Reviewed By: nicolasvasilache, rriddle
Differential Revision: https://reviews.llvm.org/D134939
This commit adds a pattern to merge accumulator and result
`vector.transpose` ops into `vector.contract`. This kind of
pattern can be generated for NCHW convolution vectorization,
where we use transposes to convert the 1-D NCW convolution
into NWC during vectorization. Merging the transpose would
mean we can avoid materialize vector extract/insert for
transposes and it makes further vector level transformations
easier.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D135496
This reverts commit c16f3260a9255c7d9880f72de7d856f9ceeb1866.
There's a bug in the commit creates a scalar result with `ShapeCastOp`.
Reverting till that fix is done.
In https://reviews.llvm.org/D133883, we changed the
`FoldExtractSliceIntoTransferRead` pattern from requiring
full identity map to minor identity map. This effectively
allows rank reducing `vector.transfer_read` ops. However,
the logic for checking `tensor.extract_slice` rank reducing
still looks at the vector rank, which now could be smaller
than the `tensor.extract_slice`'s output tensor rank.
It ends up we can have incorrect index cacluation after
folding due to this double rank reducing behavior.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D134984
Many tests still depend on specific names of SSA values (!!).
This commit is a best effort cleanup that will set the stage for adding some pretty SSA result names.
Make sure we consider other subviews of the same buffer when doing store
to load forwarding or dead store elimination.
Differential Revision: https://reviews.llvm.org/D134576
Test does a "CHECK-NOT" against the function name when it should check
to ensure that the `vector.warp_execute_on_lane_0` is removed.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D134448
One of the vector transformation patterns has been indiscriminately
converting layouts to affine maps. Leverage the strided form when
possible.
Reviewed By: nicolasvasilache, dcaballe
Differential Revision: https://reviews.llvm.org/D134047
All relevant operations have been switched to primarily use the strided
layout, but still support the affine map layout. Update the relevant
tests to use the strided format instead for compatibility with how ops
now print by default.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D134045
Bufferization already makes the assumption that buffers pass function
boundaries in the strided form and uses the corresponding affine map layouts.
Switch it to use the recently introduced strided layout instead to avoid
unnecessary casts when bufferizing further operations to the memref dialect
counterparts that now largely rely on the strided layout attribute.
Depends On D133947
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D133951
Memref subview operation has been initially designed to work on memrefs with
strided layouts only and has never supported anything else. Port it to use the
recently added StridedLayoutAttr instead of extracting the strided from
implicitly from affine maps.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D133938
vecotr.transfer_read ops with minor identity indexing map is rank
reducing, with implicit leading unit dimensions. This should be
a natural extension to support in addition to full identity indexing
maps.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D133883
Simplify the lowering of warp_execute_on_lane0 of scf.if by making the
logic more generic. Also remove the assumption that the most inner
dimension is the dimension distributed.
Differential Revision: https://reviews.llvm.org/D133826
Add a new pattern to fold `vector.extract` over n-D constants that extract scalars.
The previous code handled ND splat constants only. The new pattern is conservative and does handle sub-vector constants.
This is to aid the `arith::EmulateWideInt` pass which emits a lot of 2-element vector constants.
Reviewed By: Mogball, dcaballe
Differential Revision: https://reviews.llvm.org/D133742
This revision significantly improves and tests the broadcast behavior of vector.warp_execute_on_lane_0.
Previously, the implementation of the broadcast behavior of vector.warp_execute_on_lane_0
assumed that the broadcasted value was always of scalar type.
This is not necessarily the case.
Differential Revision: https://reviews.llvm.org/D133767
The logic to figure out if a transfer op can be flattened wasn't
considering the shape being loaded therefore it was incorrectly assuming
some transfer ops were reading contigous data.
Differential Revision: https://reviews.llvm.org/D133544
CombiningKind was implemented before EnumAttr, so it reimplements the same behaviour with the custom code. Except for a few places, EnumAttr is a drop-in replacement.
Reviewed By: nicolasvasilache, pifon2a
Differential Revision: https://reviews.llvm.org/D133343
Running: `mlir-opt -test-vector-warp-distribute=rewrite-warp-ops-to-scf-if -canonicalize -verify-each=0`.
Prior to this revision, IR resembling the following would be produced:
```
%4 = "vector.load"(%3, %arg0) : (memref<1x32xf32, 3>, index) -> vector<1x1xf32>
```
This fails verification since it needs 2 indices to load but only 1 is provided.
Differential Revision: https://reviews.llvm.org/D133106
Introduce a new attribute to represent the strided memref layout. Strided
layouts are omnipresent in code generation flows and are the only kind of
layouts produced and supported by a half of operation in the memref dialect
(view-related, shape-related). However, they are internally represented as
affine maps that require a somewhat fragile extraction of the strides from the
linear form that also comes with an overhead. Furthermore, textual
representation of strided layouts as affine maps is difficult to read: compare
`affine_map<(d0, d1, d2)[s0, s1] -> (d0*32 + d1*s0 + s1 + d2)>` with
`strides: [32, ?, 1], offset: ?`. While a rudimentary support for parsing a
syntactically sugared version of the strided layout has existed in the codebase
for a long time, it does not go as far as this commit to make the strided
layout a first-class attribute in the IR.
This introduces the attribute and updates the tests that using the pre-existing
sugared form to use the new attribute instead. Most memref created
programmatically, e.g., in passes, still use the affine form with further
extraction of strides and will be updated separately.
Update and clean-up the memref type documentation that has gotten stale and has
been referring to the details of affine map composition that are long gone.
See https://discourse.llvm.org/t/rfc-materialize-strided-memref-layout-as-an-attribute/64211.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D132864
Currently vector.gather only supports reading memory into a 1-D result vector.
This patch extends it to support an n-D result vector with the indices, masks,
and passthroughs in n-D vectors.
As we are trying to vectorize tensor.extract with vector.gather
(https://github.com/iree-org/iree/issues/9198), it will need to gather the
elements into an n-D vector. Having vector.gather with n-D results allows us
to avoid flatten and reshape at the vectorization stage. The backends can then
decide the optimal ways to lower the vector.gather op.
Note that this is different from n-D gathering, which is about reading n-D
memory with the n-D indices. The indices here are still only 1-D offsets on
the base.
Reviewed By: dcaballe
Differential Revision: https://reviews.llvm.org/D131905
This commit adds support for 0-D vectors in ReductionOp.
Reviewed By: nicolasvasilache, dcaballe
Differential Revision: https://reviews.llvm.org/D131896
In addition to memref, accept ranked tensor as the base operand of vector.gather, similar to vector.trasnfer_read.
This will allow us to vectorize noncontiguous tensor.extract into vector.gather. Full discussion can be found here: https://github.com/iree-org/iree/issues/9198
Reviewed By: hanchung, dcaballe
Differential Revision: https://reviews.llvm.org/D130097
The test was using a missing prefix. Add the prefix and fix the naming.
Found by @csigg
Reviewed By: csigg
Differential Revision: https://reviews.llvm.org/D131428
Folding of transfer_write into transfer_read is already supported but
this requires the read and write to have the same permuation map.
After linalg vectorization it is common to have different ppermuation
map for write followed by read even though the cases could be
propagated.
This canonicalization handle cases where the permuation maps are
different but the data read and written match and replace the transfer
ops with broadcast and permuation
Differential Revision: https://reviews.llvm.org/D130135
Convert arith.cmpi to the canonical form with constants on the right side
to simplify further optimizations and open more opportunities for CSE.
Differential Revision: https://reviews.llvm.org/D129929
since vector.reduce support accumulator in all the cases remove the
assert assuming old definition.
Differential Revision: https://reviews.llvm.org/D129602
This allows vectorizing linalg reductions without changing the operation
order. Therefore this produce a valid vectorization even if operations
are not associative.
Differential Revision: https://reviews.llvm.org/D129535
Prevent creating multiple output for the same Value when distributing
operations out of WarpExecuteOnLane0Op. This avoid creating combinatory
explosion of outputs.
Differential Revision: https://reviews.llvm.org/D129465
