Refactors intrinsic op interface to delegate initial operands mapping to
the dialect converter and allow intrinsic operands getters to only
perform last mile post-processing.
After each function is translated, both value and block maps are erased,
which makes the current mapping of blockaddresses to llvm blocks broken
- the patching happens only after *all* functions are translated.
Simplify the overall mapping, update comments, variable names and fix
the bug.
---------
Co-authored-by: Christian Ulmann <christianulmann@gmail.com>
This adds a token for a forward slash to the token definition list and
the methods to `AsmParser::parseSlash()` and
`AsmParser::parseOptionalSlash()`, similar to other tokens used as
operators (e.g., star, plus, etc.). This allows implementations of
attributes that contain arithmetic expressions to support operators with
a forward slash, e.g., a division.
The newly added check tests trigger the parsing of a slash in an
attribute.
As part of the work on transitioning bufferization dialect, ops, and
associated logic to operate on newly added type interfaces (see
00eaff3e9c897c263a879416d0f151d7ca7eeaff), rename the
bufferization.to_memref to highlight the generic nature of the op.
Bufferization process produces buffers while memref is a builtin type
rather than a generic term.
Preserve the current API (to_buffer still produces a memref), however,
as the new type interfaces are not used yet.
During bufferization, the callee of each `func::CallOp` / `CallableOpInterface` operation is retrieved by means of a symbol table that is temporarily built for the lookup purpose. The creation of the symbol table requires a linear scan of the operation body (e.g., a linear scan of the `ModuleOp` body). Considering that functions are typically called at least once, this leads to a scaling behavior that is quadratic with respect to the number of symbols. The problem is described in the following Discourse topic: https://discourse.llvm.org/t/quadratic-scaling-of-bufferization/86122/
This patch aims to partially address this scaling issue by leveraging the `SymbolTableCollection` class, whose instance is added to the `FuncAnalysisState` extension. Later modifications are also expected to address the problem in other methods required by `BufferizableOpInterface` (e.g., `bufferize` and `getBufferType`), which suffer of the same problem but do not provide access to any bufferization state.
This PR updates the FoldMemRefAliasOps to use `affine.linearize_index`
and `affine.delinearize_index` to perform the index computations needed
to fold a `memref.expand_shape` or `memref.collapse_shape` into its
consumers, respectively.
This also loosens some limitations of the pass:
1. The existing `output_shape` argument to `memref.expand_shape` is now
used, eliminating the need to re-infer this shape or call `memref.dim`.
2. Because we're using `affine.delinearize_index`, the restriction that
each group in a `memref.collapse_shape` can only have one dynamic
dimension is removed.
By analogy to some changess to the affine.apply lowering which put
`nsw`s on various multiplications, add appropritae overflow flags to the
multiplications and additions that're emitted when lowering
affine.delinearize_index and affine.linearize_index to arith ops.
This patch updates the following ArmSME ops to require that input and
output element types match:
* `arm_sme.tile_load`, `arm_sme.tile_store`,
`arm_sme.tile_load_slice`, `arm_sme.tile_store_slice`.
In addition, it ensures that the base memref operand for `tile_load` and
`tile_store` is always rank-2, aligning with the semantics of Arm SME
tiles (always rank-2). This change is effectively a follow-up to
#135151:
* "[mlir][vector] Tighten the semantics of vector.{load|store}"
The patch also updates `createLoadStoreForOverTileSlices` in
ArmSMEToSCF.cpp to fail when processing invalid tile stores like the
following:
```mlir
arm_sme.tile_store %arg0, %arg1[%c0] : memref<?x4xi8>, vector<[4]x[4]xi32>
```
This particular change fixes#118769. As noted in the TODO, we should
further extend op verification logic — I plan to address that in a
follow-up patch.
Improve mlir-query tool by implementing `getBackwardSlice` and
`getForwardSlice` matchers. As an addition `SetQuery` also needed to be
added to enable custom configuration for each query. e.g: `inclusive`,
`omitUsesFromAbove`, `omitBlockArguments`.
Note: backwardSlice and forwardSlice algoritms are the same as the ones
in `mlir/lib/Analysis/SliceAnalysis.cpp`
Example of current matcher. The query was made to the file:
`mlir/test/mlir-query/complex-test.mlir`
```mlir
./mlir-query /home/dbudii/personal/llvm-project/mlir/test/mlir-query/complex-test.mlir -c "match getDefinitions(hasOpName(\"arith.add
f\"),2)"
Match #1:
/home/dbudii/personal/llvm-project/mlir/test/mlir-query/complex-test.mlir:5:8:
%0 = linalg.generic {indexing_maps = [#map, #map], iterator_types = ["parallel", "parallel"]} ins(%arg0 : tensor<5x5xf32>) outs(%arg1 : tensor<5x5xf32>) {
^
/home/dbudii/personal/llvm-project/mlir/test/mlir-query/complex-test.mlir:7:10: note: "root" binds here
%2 = arith.addf %in, %in : f32
^
Match #2:
/home/dbudii/personal/llvm-project/mlir/test/mlir-query/complex-test.mlir:10:16:
%collapsed = tensor.collapse_shape %0 [[0, 1]] : tensor<5x5xf32> into tensor<25xf32>
^
/home/dbudii/personal/llvm-project/mlir/test/mlir-query/complex-test.mlir:13:11:
%c2 = arith.constant 2 : index
^
/home/dbudii/personal/llvm-project/mlir/test/mlir-query/complex-test.mlir:14:18:
%extracted = tensor.extract %collapsed[%c2] : tensor<25xf32>
^
/home/dbudii/personal/llvm-project/mlir/test/mlir-query/complex-test.mlir:15:10: note: "root" binds here
%2 = arith.addf %extracted, %extracted : f32
^
2 matches.
```
Three main changes:
- The pass createRequestCWrappersPass is renamed as
createLLVMRequestCWrappersPass
- createOptimizeForTargetPass is now under the LLVM namespace. It’s
unclear why the NVVM namespace was used initially, as all passes in
LLVMIR/Transforms/Passes.h consistently reside in the LLVM namespace.
- DuplicateFunctionEliminationPass is now in the func namespace.
Similar to vector ops, XeGPU ops need to be unrolled into smaller shapes
such that they can be dispatched into a hardware instruction. This PR
marks the initial phase of a series dedicated to incorporating unroll
patterns for XeGPU operations. In this installment, we introduce
patterns for the following operations:
1. createNd
2. updateNd
3. prefetchNd
4. loadNd
5. storeNd
6. dpas
…_reduce_matmul.
This patch exposes broadcast and transpose semantics on
'batch_reduce_matmul'. This is the last one in continuation of other two
variant of matmul ops.
The broadcast and transpose semantic are as follows:
Broadcast and Transpose semantics can be appiled by specifying the
explicit attribute 'indexing_maps' as shown below. This is a list
attribute, so must include maps for all arguments if specified.
Example Transpose:
```
linalg.batch_reduce_matmul indexing_maps = [
affine_map<(d0, d1, d2, d3) -> (d0, d3, d1)>, // transpose
affine_map<(d0, d1, d2, d3) -> (d0, d3, d2)>,
affine_map<(d0, d1, d2, d3) -> (d1, d2)>
]
ins(%arg0, %arg1 : memref<2x5x3xf32>,memref<2x5x7xf32>)
outs(%arg2: memref<3x7xf32>)
```
Example Broadcast:
```
linalg.batch_reduce_matmul indexing_maps = [
affine_map<(d0, d1, d2, d3) -> (d3)>, // broadcast
affine_map<(d0, d1, d2, d3) -> (d0, d3, d2)>,
affine_map<(d0, d1, d2, d3) -> (d1, d2)>
]
ins(%arg0, %arg1 : memref<5xf32>, memref<2x5x7xf32>)
outs(%arg2: memref<3x7xf32>)
```
Example Broadcast and Transpose:
```
linalg.batch_reduce_matmul indexing_maps = [
affine_map<(d0, d1, d2, d3) -> (d1, d3)>, // broadcast
affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>, // transpose
affine_map<(d0, d1, d2, d3) -> (d1, d2)>
]
ins(%arg0, %arg1 : memref<3x5xf32>, memref<2x7x5xf32>)
outs(%arg2: memref<3x7xf32>)
```
RFCs and related PR:
https://discourse.llvm.org/t/rfc-linalg-opdsl-constant-list-attribute-definition/80149https://discourse.llvm.org/t/rfc-op-explosion-in-linalg/82863https://discourse.llvm.org/t/rfc-mlir-linalg-operation-tree/83586https://github.com/llvm/llvm-project/pull/115319https://github.com/llvm/llvm-project/pull/122275
After the introduction of `OpAsmAttrInterface`, it is favorable to
migrate code using `OpAsmDialectInterface` for ASM alias generation,
which lives in `Dialect.cpp`, to use `OpAsmAttrInterface`, which lives
in `Attrs.td`. In this way, attribute behavior is placed near its
tablegen definition and people won't need to go through other files to
know what other (unexpected) hooks comes into play.
See #124721 for the interface itself and #128191 and #130479 for prior
migrations.
Note that `MLProgramOpAsmInterface` has no content now. However, if we
delete it, a failure related to dialect resource handling will occur
```
within split at llvm-project/mlir/test/IR/invalid-file-metadata.mlir:60 offset :7:7: error: unexpected error: unexpected 'resource' section for dialect 'ml_program'
```
To support resource such interface must be registered.
After the introduction of `OpAsmAttrInterface`, it is favorable to
migrate code using `OpAsmDialectInterface` for ASM alias generation,
which lives in `Dialect.cpp`, to use `OpAsmAttrInterface`, which lives
in `Attrs.td`. In this way, attribute behavior is placed near its
tablegen definition and people won't need to go through other files to
know what other (unexpected) hooks comes into play.
See #124721 for the interface itself and #128191 for prior migration for
Builtin Attributes.
See #131504 for the `genMnemonicAlias` tablegen field.
[mlir][vector] Standardize base Naming Across Vector Ops (NFC)
This change standardizes the naming convention for the argument
representing the value to read from or write to in Vector ops that
interface with Tensors or MemRefs. Specifically, it ensures that all
such ops use the name `base` (i.e., the base address or location to
which offsets are applied).
Updated operations:
* `vector.transfer_read`,
* `vector.transfer_write`.
For reference, these ops already use `base`:
* `vector.load`, `vector.store`, `vector.scatter`, `vector.gather`,
`vector.expandload`, `vector.compressstore`, `vector.maskedstore`,
`vector.maskedload`.
This is a non-functional change (NFC) and does not alter the semantics of these
operations. However, it does require users of the XFer ops to switch from
`op.getSource()` to `op.getBase()`.
To ease the transition, this PR temporarily adds a `getSource()` interface
method for compatibility. This is intended for downstream use only and should
not be relied on upstream. The method will be removed prior to the LLVM 21
release.
Implements #131602
Since the introduction of `OpAsm{Type,Attr}Interface` (#121187), it is
possible to generate alias in AsmPrinter solely from the type/attribute
itself without consulting the `OpAsmDialectInterface`. This means the
behavior can be put in tablegen file near the type/attribute definition.
A common pattern is to just use the type/attr mnemonic as the alias.
Previously, like #130479/#130481/#130483, this means adding a default
implementation to `extraClassDeclaration` in `LLVM_Attr` base class.
However, as attribute definition may override `extraClassDeclaration`,
it might be preferred to have a new field in tablegen to specify this
behavior.
This commit adds a `genMnemonicAlias` field to `AttrOrTypeDef`, when
enabled, makes `mlir-tblgen` emit a default implementation of `getAlias`
using mnemonic. When `OpAsm{Attr,Type}Interface` is not specified by the
user, `tblgen` will automatically add the interface.
For users wanting other alias behavior, they can ignore such field and
still use `extraClassDeclaration` way.
This clause requires an entire additional collection to keep track of
the gang 'kind' or 'type'. That work is maintained in the OpenACC
dialect functions. Otherwise, this is effectively the same as the
worker/vectors.
There are two kind of materialization callbacks: one for target
materializations and one for source materializations. The callback type
for target materializations is `TargetMaterializationCallbackFn`. This
commit renames the one for source materializations from
`MaterializationCallbackFn` to `SourceMaterializationCallbackFn`, for
consistency.
There used to be a single callback type for both kind of
materializations, but the materialization function signatures have
changed over time.
Also clean up a few places in the documentation that still referred to
argument materializations.
This patch implements worker and vector lowering for the loop construct,
which are fairly simple clauses, except that they also have a 'no
argument' form which requires a touch more work. Else, these are just
like a handful of other clauses where we just keep the device_type array
and operands in sync.
These two ended up being pretty similar in frontend implementation, and
fairly trivial when doing lowering. The collapse clause jsut results in
a normal device_type style attribute with some mild additional
complexity, and 'tile' just uses the current infrastructure for 'with
segments'.
The forms of the MemRef builder that took an integer argument instead of
an attribute have been deprecated for years now, and have almost no
upstream uses (the remaining ones are handled in this PR). Therefore,
remove them.
This adds a pass that converts memref.load/store into affine.load/store.
This is useful as those memref operators are ignored by passes like
--affine-scalrep as they don't implement the
Affine[Read/Write]OpInterface. Doing this allows you to put as much of
your program in affine form before you apply affine optimization passes.
This also slightly changes the implementation of affine::isValidDim. The
previous implementation allowed values from the iter_args of affine
loops to be used as valid dims. I think this doesn't make sense and what
was meant is just the induction vars. In the real world, there is little
reason to find an index in the iter_args, but I wrote that in my tests
and found out it was treated as an affine dim, so corrected that.
Co-authored-by: Oleksandr "Alex" Zinenko <git@ozinenko.com>
Rebased from #114032.
This change:
- Adds the `cvt.f32x2.to.f8x2`, `cvt.f16x2.to.f8x2`, and
`cvt.bf16x2.to.f8x2`
Ops to the NVVM dialect for the conversions to `.e4m3x2`, `e5m2x2`,
and `.ue8m0x2` types.
- Renames the recently added `cvt.to.f6x2` Op to `cvt.f32x2.to.f6x2`
for consistency with the other conversion Ops.
For more information, see PTX ISA:
https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt
LLVM IR currently [accepts](https://godbolt.org/z/nqnEsW1ja):
```
define void @incompatible_call_and_callee_types() {
call void @callee(i64 0)
ret void
}
define void @callee({ptr, i64}, i32) {
ret void
}
```
This currently fails to import. Even though these constructs are
dangerous and probably indicate some ODR violation (or optimization
bug), they are "valid" and should be imported into LLVM IR dialect. This
PR implements that by using an indirect call to represent it.
Translation already works nicely and outputs the same source llvm IR
file.
The error is now a warning, the tests in
`mlir/test/Target/LLVMIR/Import/import-failure.ll` already use `CHECK`
lines, so no need to add extra diagnostic tests.
Add one more of these module flags.
Unlike "CG Profile", LLVM proper does not verify the content of the
metadata, but returns a nullptr in case it's ill-formed (it's up to the
user to take action). This prompted me to implement warning checks,
preventing the importer to consume broken data.
This commit refactors the getStridesAndOffet() method on MemRefType to
just call `MemRefLayoutAttrInterface::getStridesAndOffset(shape,
strides& offset&)`, allowing downstream users and future layouts (ex, a
potential contiguous layout) to implement it without needing to patch
BuiltinTypes or without needing them to conform their affine maps to the
canonical strided form.
Adds AVX broadcast and conversion from F16 to packed F32 (similar to PR:
https://github.com/llvm/llvm-project/pull/136830). The instructions that
are added:
- VBCSTNESH2PS
- VCVTNEEPH2PS
- VCVTNEOPH2PS
This makes the dyn_cast<PassExecutionAction> work outside the dylib,
i.e. from the unit test binary, when the CMake setting
MLIR_LINK_MLIR_DYLIB is ON.
Fixes#138202
Create a wrapper around the new scaled MFMAs that operate on specific
element types and tile sizes.
See [Issue](https://github.com/iree-org/iree/issues/20616).
---------
Signed-off-by: Muzammiluddin Syed <muzasyed@amd.com>
In ConvertVectorToLLVM, the only option for setting alignment of
`vector.gather`, `vector.scatter`, and the `vector.load/store` ops was
to extract it from the datatype of the memref type. However, this is
insufficient for hardware backends requiring alignment of vector types.
This PR introduces the `use-vector-alignment` option to the
`ConvertVectorToLLVMPass`, which makes the pass use the alignment of the
vector type of these operations instead of the alignment of the memref
type.
---------
Co-authored-by: Lily Orth-Smith <lorthsmith@microsoft.com>
This PR introduces a new tool, mlir-irdl-to-cpp, that converts IRDL to
C++ definitions.
The C++ definitions allow use of the IRDL-defined dialect in MLIR C++
infrastructure, enabling the use of conversion patterns with IRDL
dialects for example. This PR also adds CMake utilities to easily
integrate the IRDL dialects into MLIR projects.
Note that most IRDL features are not supported. In general, we are only
able to define simple types and operations.
- The only type constraint supported is `irdl.any`.
- Variadic operands and results are not supported.
- Verifiers for the IRDL constraints are not generated.
- Attributes are not supported.
---------
Co-authored-by: Théo Degioanni <theo.degioanni.llvm.deluge062@simplelogin.fr>
Co-authored-by: Fehr Mathieu <mathieu.fehr@gmail.com>
Fix a compile error (with LLVM_ENABLE_WERROR=On) when building
`bin/flang`:
```
enumeration values 'FMaximum' and 'FMinimum' not handled in switch
```
caused by adding new LLVM IR instructions in #137701.
This wasn't picked up by the auto CI builders on GitHub, so I hadn't
realised until @kazutakahirata notified me about it.
These just add a standard 'device_type' flag to the acc.loop, so
implement that lowering. This also modifies the dialect to add helpers
for these as well, to be consistent with the previous ones.
