The revision clears a long-due TODO, which supports the lowering when
transfer_read/write ops have mask via inserting a vector.shape_cast op
for the masked value.
---------
Signed-off-by: hanhanW <hanhan0912@gmail.com>
When the conversion destination type is i8 (e.g., extsi i4->i8),
RewriteAlignedSubByteIntExt was unconditionally creating a new
ConversionOp with identical source and result types (vector<8xi8> ->
vector<8xi8>), which is invalid IR. Similarly,
RewriteAlignedSubByteIntTrunc was creating arith.trunci from
vector<Nxi8> to vector<Nxi8> when the source was already i8.
Fix by:
- In RewriteAlignedSubByteIntExt: replace the op directly with
subByteExt when it already has the destination type, instead of wrapping
in a new conversion op.
- In RewriteAlignedSubByteIntTrunc: skip the intermediate truncation to
i8 when the source is already i8, passing srcValue directly to the
i8->i4 rewrite logic.
This matches the existing test expectations in
vector-rewrite-subbyte-ext-and-trunci.mlir.
Assisted-by: Claude Code
Fix a failure present with MLIR_ENABLE_EXPENSIVE_PATTERN_API_CHECKS=ON.
ScanToArithOps::matchAndRewrite created the result arith.constant before
checking whether the reduction dimension is scalable. When the dimension
was scalable, the pattern returned notifyMatchFailure() after IR was
already modified, violating MLIR_ENABLE_EXPENSIVE_PATTERN_API_CHECKS.
Fix: move the reductionScalableDims check to before the
arith::ConstantOp creation.
Assisted-by: Claude Code
Fix a failure present with MLIR_ENABLE_EXPENSIVE_PATTERN_API_CHECKS=ON.
BroadcastOpLowering::matchAndRewrite created ub::PoisonOp before
checking whether a scalable outer dimension prevents the
stretch-not-at-start case. When that check triggered, the pattern
returned failure() after IR was already modified, violating
MLIR_ENABLE_EXPENSIVE_PATTERN_API_CHECKS.
Fix: move the scalable dimension check to before the PoisonOp creation.
Assisted-by: Claude Code
Fix a failure present with MLIR_ENABLE_EXPENSIVE_PATTERN_API_CHECKS=ON.
Many MLIR APIs end up using a range of the same Type / Value repeated N
times, due to the (function of the) dimensionality of the problem.
Allocating a vector of N identical element is wasteful.
Add `Repeated<T>` as PointerUnion variants in TypeRange and ValueRange,
enabling O(1) storage for repeated elements. Size remains 2 pointers (16
bytes on 64-bit) for both range types. This required variable-width
`PointerUnion` encoding added in
https://github.com/llvm/llvm-project/pull/188167 on 32-bit systems.
Also update several MLIR dialects and conversions to exercise the new
code.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds support for flattening multi-dimensional scalar vector transfers.
The addition prevents pattern crashes on such inputs and allows for
cleaner lowering of scalar vectors.
The `OneDimMultiReductionToTwoDim` pattern had some issues. For the
input program:
```mlir
func.func @rank1_multi_reduction(%arg0: vector<8xf32>, %acc: f32) -> f32 {
%0 = vector.multi_reduction <add>, %arg0, %acc [0] : vector<8xf32> to f32
return %0 : f32
}
```
* when lowering using the inner-parallel strategy, the compiler would
essentially produce scalar code:
```mlir
func.func @rank1_multi_reduction(%arg0: vector<8xf32>, %arg1: f32) -> f32 {
%0 = vector.shape_cast %arg0 : vector<8xf32> to vector<1x8xf32>
%1 = vector.broadcast %arg1 : f32 to vector<1xf32>
%2 = vector.transpose %0, [1, 0] : vector<1x8xf32> to vector<8x1xf32>
%3 = vector.extract %2[0] : vector<1xf32> from vector<8x1xf32>
%4 = arith.addf %3, %1 : vector<1xf32>
%5 = vector.extract %2[1] : vector<1xf32> from vector<8x1xf32>
%6 = arith.addf %5, %4 : vector<1xf32>
... (repeats for all 8 elements) ...
%17 = vector.extract %2[7] : vector<1xf32> from vector<8x1xf32>
%18 = arith.addf %17, %16 : vector<1xf32>
%19 = vector.extract %18[0] : f32 from vector<1xf32>
return %19 : f32
}
```
* when lowering using the inner-reduction strategy, the compiler would
first unnecessarily transform it into a 2-D multi_reduction operation
<1x8xf32> and then extract an <8xf32> vector and apply reduction. The
canonicalization and folding would lead to the following final result:
```mlir
func.func @rank1_multi_reduction(%arg0: vector<8xf32>, %arg1: f32) -> f32 {
%0 = vector.reduction <add>, %arg0, %arg1 : vector<8xf32> into f32
return %0 : f32
}
```
Now, after this change:
* when lowering the compiler now produces for both strategies in one
step.
```
func.func @rank1_multi_reduction(%arg0: vector<8xf32>, %arg1: f32) -> f32 {
%0 = vector.reduction <add>, %arg0, %arg1 : vector<8xf32> into f32
return %0 : f32
}
```
This pattern is also useful for an ongoing refactoring that is happening
in the multi_reduction patterns. It is the only pattern that increases
multi_reduction in rank and would lead to an infinite loop when
attempting to reach a fixed point once we generalize other unrolling
patterns.
Assisted-by: Claude
The initial implementation of `shape_cast` distribution only focused on
scenarios with collapsing shape casts. Within downstream pipelines such
as IREE, commit 962a9a3 exposes an issue with this implementation, where
the rank-expanding cast ops (stemming from the new `vector.broadcast`
canonicalization) silently fall through to the "collapsing-or-no-op"
logic. This brings about bugs with rank mismatches and firing validation
assertions when distributing rather common reshaping sequences
encountered after CSE/ canonicalization, such as below:
```
// Example 1: gather op
%weight = arith.constant dense_resource<__elided__> : tensor<256xi8>
%c0 = arith.constant 0 : index
...
%expand = vector.shape_cast <...> : vector<1xindex> to vector<1x1xindex>
%gather = vector.gather %weight[%c0] [%expand], <...>, <...> : memref<256xi8>, vector<1x1xindex>, vector<1x1xi1>, vector<1x1xi8> into vector<1x1xi8>
%collapse_back = vector.shape_cast %gather : vector<1x1xi8> to vector<1xi8>
// Example 2: multi-reduction
%expand = vector.shape_cast <...>: vector<1x96xi32> to vector<1x2x48xi32>
%reduce = vector.multi_reduction <add>, %expand, <...> [1, 2]: vector<1x2x48xi32> to vector<1x1xi32>
%collapse = vector.shape_cast %reduce : vector<1x1xi32> to vector<1xi32>
```
This commit adds initial handling of expanding `shape_cast`s, going
through the three scenarios:
- if all "excess" dimensions in the front of the destination shape are
unit, it's clear that the work is not distributed across those, so we
strip the same number of unit dimensions from the per-lane yielded type;
- if the source type within the warp code is of rank 1, we still
determine the corresponding output type cleanly by multiplying the
dimensions of the per-lane yield type;
- if neither of the above are true, explicitly fail the pattern for such
expanding `shape_cast`'s. Dimension-specific distribution parameters are
deemed ambiguous, at least from within this pattern's context.
---------
Signed-off-by: Artem Gindinson <gindinson@roofline.ai>
The updated name better captures what the pattern does and matches the
coresponding `populat*` hook,
`populateVectorMultiReductionFlatteningPatterns`, that only contains
this pattern.
Refactor the following patterns to inherit from
`MaskableOpRewritePattern`:
* `TwoDimMultiReductionToReduction`
* `TwoDimMultiReductionToElementWise`
This improves code reuse, enables small simplifications, and unifies the
structure of the patterns. Add high-level comments to clarify the
overall lowering strategy.
Prepares for future refactoring (e.g. #182301) and helps maintain a
uniform implementation.
With the new finer grained populate methods introduced in
8dde3051504cb9ae42e654bbce39001f3946beea (#180750), there was a
discussion about refactoring tests such that only one of the patterns
applies at a time. This commit starts this process by adding the
structure for one of these populate methods. The goal is for the
populate methods to have their own file (each showing inner and outer
reduction); deprecating populateVectorMultiReductionLoweringPatterns and
ApplyLowerMultiReduction; and removing the test file for
mlir/test/Dialect/Vector/vector-multi-reduction-lowering.mlir
Essentially an NFC. It also adds a new transform op for testing the
dialect and which downstream projects may choose to use.
Assisted-By: claude-4.5-sonnet
Thiese commits add three more populate methods for
`vector.multi_reduction`'s lowering patterns:
* populateVectorMultiReductionTransformationPatterns
* populateVectorMultiReductionFlatteningPatterns
* populateVectorMultiReductionUnrollingPatterns
These methods have a
finer level of granularity and allow users to select between unrolling,
flattening, and applying transformations that would set up operations
for unrolling and flattening.
The previous populateVectorMultiReductionLoweringPatterns method
is rewritten in terms of these new methods.
There is no existing `vector.reduce` op in the vector dialect, but
multiple doc strings reference it. This change updates those instances
to the correct `vector.reduction` op.
The revision adds a new `assumeAligned` mode to the emulation, so
downstream projects can use simple path when it meets the requirements.
E.g., if the offset is always aligned with container's element type, we
can skip the check of front padding sizes.
---------
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Added rank‑0 handling to masked load/store emulation by reinterpreting
rank‑0 memrefs as 1‑D buffers with a synthetic index, preventing
empty‑indices crashes.
Fixes https://github.com/llvm/llvm-project/issues/131243
This PR adds unrolling for vector.constant_mask op based on the
targetShape. Each unrolled vector computes its local mask size in each
dimension (d) as:
min(max(originalMaskSize[d] - offset[d], 0), unrolledMaskSize[d]).
Fixes a crash in `ReorderCastOpsOnBroadcast` by ensuring the cast result
is a `VectorType` before applying the pattern.
A regression test has been added to
mlir/test/Dialect/Vector/vector-sink.mlir.
Fixes: #126371
This PR adds unrolling for vector.create_mask op based on the
targetShape. Each unrolled vector computes its local mask size in each
dimension (d) as:
min(max(originalMaskSize[d] - offset[d], 0), unrolledMaskSize[d]).
This PR adds pattern for unrolling shape_cast given a targetShape. This
PR is a follow up of #164010 which was very general and was using
inserts and extracts on each element (which is also
LowerVectorShapeCast.cpp is doing).
After doing some more research on use cases, we (me and @Jianhui-Li )
realized that the previous version in #164010 is unnecessarily generic
and doesn't fit our performance needs.
Our use case requires that targetShape is contiguous in both source and
result vector.
This pattern only applies when contiguous slices can be extracted from
the source vector and inserted into the result vector such that each
slice remains in vector form with targetShape (and not decompose to
scalars). In these cases, the unrolling proceeds as:
vector.extract_strided_slice -> vector.shape_cast (on the slice
unrolled) -> vector.insert_strided_slice
This PR makes the following improvements to `vector.scatter` and its
lowering pipeline:
- In addition to `memref`, accept a ranked `tensor` as the base operand
of `vector.scatter`, similar to `vector.transfer_write`.
- Implement bufferization support for `vector.scatter`, so that
tensor-based scatter ops can be fully lowered to memref-based forms.
It's worth to complete the functionality of map_scatter decomposition.
Full discussion can be found here:
https://github.com/iree-org/iree/issues/21135
---------
Signed-off-by: Ryutaro Okada <1015ryu88@gmail.com>
Uniform values should not be distributed during vector distribution.
Example would be a reduction result where reduction happens across
lanes.
However, current `getDistributedType` does not accept a zero result
affine map (i.e. no distributed dims) when describing the distributed
dimensions. This result in null type being returned and crashing the
vector distribution in some cases. An example case would be a `scf.for`
op (about to be distributed) in which one of the for result is a uniform
value and it does not have a user outside the warp op. This necessitates
querying the `getDistributedType` to figure our the distributed type of
this value.
The PR https://github.com/llvm/llvm-project/pull/162167 removed a
pattern to linearize vector.splat, without adding the equivalent pattern
for vector.broadcast. This PR adds such a pattern, hopefully brining
vector.broadcast up to full parity with vector.splat that has now been
removed.
---------
Signed-off-by: James Newling <james.newling@gmail.com>
In some cases, loop bounds (lower, upper and step) of `scf.for` can come
locally from the parent warp op the `scf.for`. Current logic will not
yield the loop bounds in the new warp op generated during lowering
causing sinked `scf.for` to have non dominating use.
In this PR, we have added logic to yield loop bounds by default (treat
them as other operands of `scf.for`) which fixes this bug.
This PR enhances the elementwise unrolling pattern to support higher
rank to lower rank unroll. The approach is to add leading unit dims to
lower rank targetShape to match the rank of original vector (because
ExtractStridedSlice requires same rank to extractSlices), extract slice,
reshape to targetShape's rank and perform the operation.
Use wrappers around `std::accumulate` to make the code more concise and
less bug-prone: https://github.com/llvm/llvm-project/pull/162129.
With `std::accumulate`, it's the initial value that determines the
accumulator type. `llvm::sum_of` and `llvm::product_of` pick the right
accumulator type based on the range element type.
Found some funny bugs like a local accumulate helper that calculated a
sum with initial value of 1 -- we didn't hit the bug because the code
was actually dead...
Change remaining OpBuilder methods to use `llvm::MaybeAlign` instead of
`uint64_t` for alignment parameters.
---------
Co-authored-by: Erick Ochoa Lopez <erick.ochoalopez@amd.com>
This PR moves the patterns that unroll vector.to_elements and
vector.from_elements into the file with other vector unrolling
operations. This PR also adds these unrolling patterns into the
`populateVectorUnrollPatterns`. And renames
`populateVectorToElementsLoweringPatterns`
`populateVectorFromElementsLoweringPatterns` to
`populateVectorToElementsUnrollPatterns`
`populateVectorFromElementsUnrollPatterns`.
This PR adds patterns to lower `vector.shuffle` with inputs with
different vector sizes more efficiently. The current LLVM lowering for
these cases degenerates to a sequence of `vector.extract` and
`vector.insert` operations. With this PR, the smaller input is promoted
to larger vector size by introducing an extra `vector.shuffle`.
Addresses: https://github.com/llvm/llvm-project/issues/115653
We already have utilities to flatten memrefs into 1-D. This change makes
memref flattening a prerequisite for vector narrow type emulation,
ensuring that emulation patterns only need to handle 1-D scenarios.
This patch updates the following ops to use `source` (instead of
`vector`) as the name for their source argument:
* `vector.extract`
* `vector.scalable.extract`
* `vector.extract_strided_slice`
This change ensures naming consistency with the "builders" for these Ops
that already use the name `source` rather than `vector`. It also
addresses part of:
* https://github.com/llvm/llvm-project/issues/131602
Specifically, it ensures that we use `source` and `dest` for read and
write operations, respectively (as opposed to `vector` and `dest`).
