Replace `SmallVector<Type/Value>(n, x)` with `Repeated<Type/Value>(n,
x)`. This avoids heap allocations for repeated values.
Also change `ExtractAddressComputations` rebuild callbacks from
`ArrayRef<Value>` to `ValueRange` to enable `Repeated<Value>`
passthrough.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop the `isCompatibleReturnTypes` override on `ExtractOp` that allowed
`vector.extract` to return a `vector<1xT>` when the natural inferred
return type is scalar `T` (and vice versa). Switch the op from
`InferTypeOpAdaptorWithIsCompatible` to `InferTypeOpAdaptor` to match.
RFC:
https://discourse.llvm.org/t/rfc-remove-implicit-bitcast-behavior-of-vector-extract/90178
Validate the operand type in ShuffleOp::inferReturnTypes to prevent a
crash when the operation is parsed with scalar types instead of vectors.
It will stop gracefully now.
Fixes#185587
Add a dedicated matcher for poison values in the UB dialect, similar to
`m_Constant()` for general constants. The matcher uses
`PoisonAttrInterface` for future extensibility.
Replace existing checks against `ub::PoisonAttr` with the new matcher.
Assisted-by: claude
### Summary
This PR adds two shape_cast-related canonicalization patterns for
`vector.to_elements` and `vector.from_elements`.
### Details
- Added` ToElements(ShapeCast(X)) -> ToElements(X)` as an in-place fold
in `ToElementsOp::fold`.
- Added `ShapeCast(FromElements(X)) -> FromElements(X)` as an
`OpRewritePattern` — it must be a pattern (not a `fold`) because we have
to create new op `FromElementsOp` with updated result type. This cannot
be done with a `fold`, because `fold` cannot create new ops and the
existing `FromElementsOp` result type differs from the `ShapeCastOp`
result type. Mutating the `FromElementsOp` (not root op) would violate
the `fold` contract and break other users.
- Added lit tests for the both ops (new `vector-to-elements.mlir`,
`vector-from-elements.mlir`)
---------
Signed-off-by: Alexandra Sidorova <asidorov@amd.com>
Fold `vector.extract(vector.constant_mask)` to `vector.constant_mask` if
possible.
If the static position is outside of the masked area, the pattern will
fold to a constant all-false value instead.
Dynamic positions are only supported if the mask covers the entire
vector in that dimension.
Assisted-by: Claude Code
---------
Signed-off-by: Lukas Sommer <lukas.sommer@amd.com>
Two related crashes were fixed in vector.mask handling:
1. MaskOp::fold() crashes with a null pointer dereference when the mask
is all-true and the mask body has no maskable operation (only a
vector.yield). getMaskableOp() returns nullptr in this case, and the
fold was calling nullptr->dropAllUses(). Fixed by returning failure()
when there is no maskable op, deferring to the canonicalizer.
2. CanonializeEmptyMaskOp creates an invalid arith.select when the mask
type is a vector (e.g., vector<1xi1>) but the result type is a scalar
(e.g., i32). arith.select with a vector condition requires the value
types to be vectors of the same shape. Fixed by bailing out when any
result type doesn't match the mask shape.
Regression tests are added for both cases.
Fixes#177833
This PR fixes `foldEmptyMaskOp` to return `failure` when folding an
empty vector.mask whose terminatorhas no operands. Previously this case
returned success without producing any folded results, which violates
the folding contract. Fixes#177825.
`BitCastOp::fold` called `Type::getIntOrFloatBitWidth()` on the source
element type without first verifying it satisfies `isIntOrFloat()`. When
the source vector has `index` element type (e.g. `vector<16xindex>`),
the assertion `only integers and floats have a bitwidth` fires.
Add an `srcElemType.isIntOrFloat()` guard to the condition so that the
constant-folding path is skipped for non-integer/float element types.
Fixes#177835
This change builds on #174336 and #175880, which introduced shared
VerificationUtils with verifyDynamicDimensionCount() and
verifyRanksMatch() methods.
This patch adds a new verifyElementTypesMatch() verification utility
that checks if two shaped types have matching element types and emits
consistent error messages. The utility is applied to several ops across
the MemRef and Vector dialects.
Based on the original PR
https://github.com/llvm/llvm-project/pull/140583, but without
vector.transpose -> vector.shape_cast.
This PR canonicalizes
%0 = vector.broadcast %arg0 : vector<4xi8> to vector<1x1x4xi8>
%2 = vector.extract %arg2[0] : vector<4xi8> from vector<1x4xi8>
to shape_cast. It was decided (see
https://github.com/llvm/llvm-project/pull/140583) that the
vector.transpose -> vector.shape_cast needs further consideration before
being added.
---------
Signed-off-by: James Newling <james.newling@gmail.com>
This PR fixes a crash by validating the type of the `kind` attribute.
For `vector.contract` and `vector.outerproduct`, the verifier now emits
an error when `kind` is not a CombiningKindAttr. Fixes#173555.
Commit
7e7ea9c535
added tensor support for scatter, but running the existing
canonicalization on tensors causes bugs, so we fix the canonicalization
with tensor output.
Closes https://github.com/llvm/llvm-project/issues/168695
---------
Signed-off-by: Ryutaro Okada <1015ryu88@gmail.com>
This patch changes the transfer_write -> transfer_read load store
forwarding canonicalization pattern to work based on permutation maps
and less on adhoc logic. The old logic couldn't canonicalize a simple
unit dim broadcast through transfer_write/transfer_read which is added
as a test in this patch.
This patch also details what would be needed to support cases which are
not yet implemented better.
This PR adds pattern for unrolling shape_cast given a targetShape. This
PR is a follow up of #164010 which was very general and was using
inserts and extracts on each element (which is also
LowerVectorShapeCast.cpp is doing).
After doing some more research on use cases, we (me and @Jianhui-Li )
realized that the previous version in #164010 is unnecessarily generic
and doesn't fit our performance needs.
Our use case requires that targetShape is contiguous in both source and
result vector.
This pattern only applies when contiguous slices can be extracted from
the source vector and inserted into the result vector such that each
slice remains in vector form with targetShape (and not decompose to
scalars). In these cases, the unrolling proceeds as:
vector.extract_strided_slice -> vector.shape_cast (on the slice
unrolled) -> vector.insert_strided_slice
This PR makes the following improvements to `vector.scatter` and its
lowering pipeline:
- In addition to `memref`, accept a ranked `tensor` as the base operand
of `vector.scatter`, similar to `vector.transfer_write`.
- Implement bufferization support for `vector.scatter`, so that
tensor-based scatter ops can be fully lowered to memref-based forms.
It's worth to complete the functionality of map_scatter decomposition.
Full discussion can be found here:
https://github.com/iree-org/iree/issues/21135
---------
Signed-off-by: Ryutaro Okada <1015ryu88@gmail.com>
This PR extends the `getVectorReductionOp` function, which is used by
the affine vectorizer, to also recognize and support
`xor/maxnumf/minnumf` reduction operations.
Fix missed `return` when folding splat ConstantOp, it could work well
probably because of good compatibility of
`foldExtractStridedSliceNonSplatConstant`.
## Description
Adds a new canonicalizer that folds
`vector.from_elements(vector.transpose))` => `vector.from_elements`.
This canonicalization reorders the input elements for
`vector.from_elements`, adjusts the output shape to match the effect of
the transpose op and eliminating its need.
## Testing
Added a 2D vector lit test that verifies the working of the rewrite.
---------
Signed-off-by: Keshav Vinayak Jha <keshavvinayakjha@gmail.com>
This PR adds a canonicalizer to vector.step that folds vector.step iff
the result of the fold is a splat value. An alternative would be to
always constant fold it, but that might result in some very
large/cumbersome constants.
I do wonder if vector.step might be better represented as some sort of
attribute in the arith dialect, like %step = arith.constant iota<32> :
vector<32xindex>.
---------
Signed-off-by: James Newling <james.newling@gmail.com>
Use wrappers around `std::accumulate` to make the code more concise and
less bug-prone: https://github.com/llvm/llvm-project/pull/162129.
With `std::accumulate`, it's the initial value that determines the
accumulator type. `llvm::sum_of` and `llvm::product_of` pick the right
accumulator type based on the range element type.
Found some funny bugs like a local accumulate helper that calculated a
sum with initial value of 1 -- we didn't hit the bug because the code
was actually dead...
Adds `::fold` for the new `vector.to_elements` op, folding `broadcast`
into `to_elements` or no-op wherever possible.
---------
Signed-off-by: keshavvinayak01 <keshavvinayakjha@gmail.com>
Signed-off-by: Keshav Vinayak Jha <keshavvinayakjha@gmail.com>
Co-authored-by: Jakub Kuderski <kubakuderski@gmail.com>
Fixes#159613
`vector.from_elements` crashes when processing float attributes with
integer result types (e.g., `llvm.mlir.constant(0.0 : f8E4M3FN) : i8`
from arith-to-llvm lowering):
```
Assertion `newType.getElementType() == curType.getElementType() && "expected the same element type"' failed.
```
## Implementation
- Rename `convertIntegerAttr` → `convertNumericAttr`
- Add float-to-integer conversion using `APFloat::convertToInteger()`
with exactness assertion
- Preserve existing integer-to-integer conversion behavior
Only implements conversions that pass LLVM verification. Other patterns
(int→float, float→float, bool→int) are rejected by LLVM verifier before
reaching this code, as documented in the attached verification failures.
The vector.from_elements constant folding was crashing when poison
values were present in the element list. The convertIntegerAttr function
was not properly handling poison attributes, leading to assertion
failures in dyn_cast operations.
This patch refactors convertIntegerAttr to take IntegerAttr directly,
moving poison detection to the caller using explicit
isa<ub::PoisonAttrInterface> checks. The function signature change
provides compile-time type safety while the early poison validation in
foldFromElementsToConstant prevents unsafe casting operations. The
folding now gracefully aborts when poison attributes are encountered,
preventing the crash while preserving correct folding for legitimate
mixed-type constants (int/float).
Fixes assertion: "dyn_cast on a non-existent value" when processing
ub.poison values in vector.from_elements operations.
This patch updates the following ops to use `source` (instead of
`vector`) as the name for their source argument:
* `vector.extract`
* `vector.scalable.extract`
* `vector.extract_strided_slice`
This change ensures naming consistency with the "builders" for these Ops
that already use the name `source` rather than `vector`. It also
addresses part of:
* https://github.com/llvm/llvm-project/issues/131602
Specifically, it ensures that we use `source` and `dest` for read and
write operations, respectively (as opposed to `vector` and `dest`).
This PR fixes two crashes / failures.
1. The `vector.broadcast` verifier did not take into account
`VectorElementTypeInterface` and was looking for int/index/float types.
2. The `vector.from_elements` folder attempted to create an invalid
`DenseElementsAttr`. Only int/float/index/complex types are supported.
## Description
This change introduces a new canonicalization pattern for the MLIR
Vector dialect that optimizes chains of insertions. The optimization
identifies when a vector is **completely** initialized through a series
of vector.insert operations and replaces the entire chain with a
single `vector.from_elements` operation.
Please be aware that the new pattern **doesn't** work for poison vectors
where only **some** elements are set, as MLIR doesn't support partial
poison vectors for now.
**New Pattern: InsertChainFullyInitialized**
* Detects chains of vector.insert operations.
* Validates that all insertions are at static positions, and all
intermediate insertions have only one use.
* Ensures the entire vector is **completely** initialized.
* Replaces the entire chain with a
single vector.from_elementts operation.
**Refactored Helper Function**
* Extracted `calculateInsertPosition` from
`foldDenseElementsAttrDestInsertOp` to avoid code duplication.
## Example
```
// Before:
%v1 = vector.insert %c10, %v0[0] : i64 into vector<2xi64>
%v2 = vector.insert %c20, %v1[1] : i64 into vector<2xi64>
// After:
%v2 = vector.from_elements %c10, %c20 : vector<2xi64>
```
It also works for multidimensional vectors.
```
// Before:
%v1 = vector.insert %cv0, %v0[0] : vector<3xi64> into vector<2x3xi64>
%v2 = vector.insert %cv1, %v1[1] : vector<3xi64> into vector<2x3xi64>
// After:
%0:3 = vector.to_elements %arg1 : vector<3xi64>
%1:3 = vector.to_elements %arg2 : vector<3xi64>
%v2 = vector.from_elements %0#0, %0#1, %0#2, %1#0, %1#1, %1#2 : vector<2x3xi64>
```
---------
Co-authored-by: Yang Bai <yangb@nvidia.com>
Co-authored-by: Andrzej Warzyński <andrzej.warzynski@gmail.com>
Fold `broadcast(shape_cast(x))` into `broadcast(x)` if the type of x is
compatible with broadcast's result type and the shape_cast only adds or removes ones in the leading dimensions.
---------
Co-authored-by: Andrzej Warzyński <andrzej.warzynski@gmail.com>
Co-authored-by: James Newling <james.newling@gmail.com>
This PR ensures parity in folding/canonicalizing of vector.broadcast
(from a scalar) and vector.splat. This means that by using
vector.broadcast instead of vector.splat (which is currently
deprecated), there is no loss in optimizations performed. All tests
which were previously checking folding/canonicalizing of vector.splat
are now done for vector.broadcast. The vector.splat canonicalization
tests are now in a separate file, ready for removal when, in the future,
we remove vector.splat completely.
This PR also adds a canonicalizer to vector.splat to always convert it
to vector.broadcast. This is to reduce the 'traffic' through
vector.splat.
There is a chance that this PR will break downstream users who create/expect
for vector.splat. Changing all such logic to work just vector.broadcast instead
should fix.
The folder `shape_cast(splat constant) -> splat constant` was first
introduced
[here](36480657d8 (diff-484cea976e0c96459027c951733bf2d22d34c5a0c0de6f577069870ef4588983R2600))
(Nov 2020). In that commit there is a comment to _Only handle splat for
now_. Based on that I assume the intention was to, at a later time,
support a general `shape_cast(constant) -> constant` folder. That is
what this PR does
One minor downside: It is possible with this folder end up with, instead
of 1 large constant and 1 shape_cast, 2 large constants:
```mlir
func.func @foo() -> (vector<4xi32>, vector<2x2xi32>) {
%cst = arith.constant dense<[1, 2, 3, 4]> : vector<4xi32> # 'large' constant 1
%0 = vector.shape_cast %cst : vector<4xi32> to vector<2x2xi32>
return %cst, %0 : vector<4xi32>, vector<2x2xi32>
}
```
gets folded with this new folder to
```mlir
func.func @foo() -> (vector<4xi32>, vector<2x2xi32>) {
%cst = arith.constant dense<[1, 2, 3, 4]> : vector<4xi32> # 'large' constant 1
%cst_0 = arith.constant dense<[[1, 2], [3, 4]]> : vector<2x2xi32> # 'large' constant 2
return %cst, %cst_0 : vector<4xi32>, vector<2x2xi32>
}
```
Notes on the above case:
1) This only effects the textual IR, the actual values share the same
context storage (I've verified this by checking pointer values in the
`DenseIntOrFPElementsAttrStorage`
[constructor](da5c442550/mlir/lib/IR/AttributeDetail.h (L59)))
so no compile-time memory overhead to this folding. At the LLVM
IR level the constant is shared, too.
2) This only happens when the pre-folded constant cannot be dead code
eliminated (i.e. when it has 2+ uses) which I don't think is common.
Implements the `inferResultRanges` method from the
`InferIntRangeInterface` interface for `vector.step`. The implementation
is similar to that of arith.constant, since the exact result values are
statically known.
Signed-off-by: Max Dawkins <max.dawkins@gmail.com>
Implements the `inferResultRanges` method from the
`InferIntRangeInterface` interface for `vector.transpose`. The result
ranges simply match the source ranges.
Signed-off-by: Max Dawkins <max.dawkins@gmail.com>
This PR uses `val.getDefiningOp<OpTy>()` to replace `dyn_cast<OpTy>(val.getDefiningOp())` , `dyn_cast_or_null<OpTy>(val.getDefiningOp())` and `dyn_cast_if_present<OpTy>(val.getDefiningOp())`.
In a later PR more shape_cast ops will appear. Specifically, broadcasts that
just prepend ones become shape_cast ops (i.e. volume preserving broadcasts
are canonicalized to shape_casts). This PR ensures that broadcast-like
shape_cast ops fold at least as well as broadcast ops.
This is done by modifying patterns that target broadcast ops, to target
'broadcast-like' ops. No new patterns are added, the patterns that exist
are just made to match on shape_casts where appropriate.
This PR also includes minor code simplifications: use
`isBroadcastableTo` to simplify `ExtractOpFromBroadcast` and simplify
how broadcast dims are detected in `foldExtractFromBroadcast`. These are
NFC.
---------
Co-authored-by: Andrzej Warzyński <andrzej.warzynski@gmail.com>
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
