This patch is part of a larger initiative aimed at fixing floating-point `max` and `min` operations in MLIR: https://discourse.llvm.org/t/rfc-fix-floating-point-max-and-min-operations-in-mlir/72671.
This commit addresses Task 1.2 of the mentioned RFC. By renaming these operations, we align their names with LLVM intrinsics that have corresponding semantics.
Exposes the existing `get(ShapedType, StringRef, AsmResourceBlob)`
builder publicly (was protected) and adds a CAPI
`mlirUnmanagedDenseBlobResourceElementsAttrGet`.
While such a generic construction interface is a big help when it comes
to interop, it is also necessary for creating resources that don't have
a standard C type (i.e. f16, the f8s, etc).
Previously reviewed/approved as part of https://reviews.llvm.org/D157064
The `cache` directive may appear at the top of (inside of) a loop. It
specifies array elements or subarrays that should be fetched into the
highest level of the cache for the body of the loop.
The `cache` directive is modeled as a data entry operands attached to
the acc.loop operation.
Deprecate the `gpu-to-cubin` & `gpu-to-hsaco` passes in favor of the
`TargetAttr` workflow. This patch removes remaining upstream uses of the
aforementioned passes, including the option to use them in `mlir-opt`. A
future patch will remove these passes entirely.
The passes can be re-enabled in `mlir-opt` by adding the CMake flag: `-DMLIR_ENABLE_DEPRECATED_GPU_SERIALIZATION=1`.
This is a follow-on to D158753, and allows the lowering of a
transfer read/write of n-D vectors with a single trailing scalable dimension
to primitive vector ops.
The final conversion to LLVM depends on D158517 and D158752, without
these patches type conversion will fail (or an assert is hit in the LLVM
backend) if the final IR contains an array of scalable vectors.
This patch adds `transform.apply_patterns.vector.lower_create_mask`
which allows the lowering of vector.create_mask/constant_mask to be
tested independently of --convert-vector-to-llvm.
Reviewed By: c-rhodes, awarzynski, dcaballe
Differential Revision: https://reviews.llvm.org/D159482
Extend SPIR-V target serialization and deserialization to handle coop
matrix types. Add a roundtrip test. In addition to `FileCheck` checks,
the resulting spirv binary also passes `spir-val` (external tool).
Also fix a type attribute bug surfaced by the `CooperativeMatrixLength`
op.
Multiple matrix operand attributes will be handled in a future patch to
reduce the scope.
This patch changes vector type conversion to return failure on n-D
scalable vector types instead of asserting.
This is an alternative approach to #65261 that aims to enable lowering
of Vector ops directly to ArmSME intrinsics where possible, and seems
more consistent with other type conversions. It's trivial to hit the
assert at the moment and it could be interpreted as n-D scalable vector
types being a bug, when they're valid types in the Vector dialect.
By returning failure it will generally fail more gracefully,
particularly for release builds or other builds where assertions are
disabled.
This patch adds the option of building an optional symbol table for the
top operation in the `gpu-module-to-binary` pass. The table is not
created by default as most targets don't need it; instead, it is lazily
built. The table is passed through a callback in `TargetOptions`.
This patch is required to integrate #65539 .
The revert happened due to a build bot failure that threw 'CUDA_ERROR_UNSUPPORTED_PTX_VERSION'.
The failure's root cause was a pass using "+ptx76" for compilation and an old CUDA driver
on the bot. This commit relands the patch with "+ptx60".
Original Gh PR: #65768
Original commit message:
Migrate tests referencing `gpu-to-cubin` to the new compilation workflow
using `TargetAttrs`. The `test-lower-to-nvvm` pass pipeline was modified
to use the new compilation workflow to simplify the introduction of
future tests.
The `createLowerGpuOpsToNVVMOpsPass` function was removed, as it didn't
allow for passing all options available in the `ConvertGpuOpsToNVVMOp`
pass.
Migrate tests referencing `gpu-to-cubin` to the new compilation workflow
using `TargetAttrs`. The `test-lower-to-nvvm` pass pipeline was modified
to use the new compilation workflow to simplify the introduction of
future tests.
The `createLowerGpuOpsToNVVMOpsPass` function was removed, as it didn't
allow for passing all options available in the `ConvertGpuOpsToNVVMOp`
pass.
The current printer of `StringRefParameter` simply prints out the
content of the string as is without escaping it any way. This leads to
it generating invalid syntax, causing parser errors when read in again.
This PR fixes that by adding `printString` to `AsmPrinter`, allowing one
to print a string that can be parsed with `parseString`, using the same
escaping syntax as `StringAttr`.
This patch employs the updated promise mechanism to enforce Target
Attribute IR constraints. Due to this patch, TargetAttributes
implementations no longer have to be registered before executing
translation to LLVM IR in cases where they are not needed, like when
translating `gpu.binary` operations.
This commit provides a default implementation for all ops that implement
the `DestinationStyleOpInterface`. Result values of such ops are tied to
operand, and those have the same type.
- Fix order of operands/attributes
- Allow for stride to be any integer type
- Use ODS for parsing/printing
- Update examples and tests
- Fix a typo in SPIR-V tblgen code
According to the LLVM language reference, both volatile memory
operations and atomic operations (except unordered) do not simply read
memory but also perform write operations on arbitrary memory[0][1].
In the case of volatile memory operations, this is the case due to the
read possibly having target specific properties. A common real-world
situation where this happens is reading memory mapped registers on an
MCU for example. Atomic operations are more special. They form a kind of
memory barrier which from the perspective of the optimizer/lang-ref
makes writes from other threads visible in the current thread. Any kind
of synchronization can therefore conservatively be modeled as a
write-effect.
This PR therefore adjusts the side effects of `llvm.load` and
`llvm.store` to add unknown global read and write effects if they are
either atomic or volatile.
Regarding testing: I am not sure how to best test this change for
`llvm.store` and the "globalness" of the effect that isn't just a unit
test checking that the output matches exactly. For the time being, I
added a test making sure that `llvm.load` does not get DCEd in
aforementioned cases.
Related logic in LLVM proper:
3398744a61/llvm/lib/IR/Instruction.cpp (L638-L676)3398744a61/llvm/include/llvm/IR/Instructions.h (L258-L262)
[0] https://llvm.org/docs/LangRef.html#volatile-memory-accesses
[1] https://llvm.org/docs/Atomics.html#monotonic
This patch adds support for the zeroinitializer constant to LLVM
dialect. It's meant to simplify zero-initialization of aggregate types
in MLIR, although it can also be used with non-aggregate types.
Both `TileOp` and `TileToScfForOp` use the tiling interface and the
`tileUsingSCFForOp` method. This duplication was introduced in
44cfea0279
as a way to retire `linalg::tileLinalgOp,` now there is not more need
for this duplication, and it seems that `tileOp` has more recent
changes, thus retire `TileToScfForOp.`
This adds a native op trait SameOperandsAndResultRank
and associated verifier that checks that an operator's
operands and result types have same ranks if their ranks
are known.
Signed-off-by: Tai Ly <tai.ly@arm.com>
Change-Id: I2d536f77be10f3710d0c8d84c907ff492a984fda
Reviewed By: rsuderman
Differential Revision: https://reviews.llvm.org/D156369
This commit allows to omit insertion of the memref.dealloc operation
when linalg.structured.bufferize_to_allocation is run and makes this the
default behavior. This is desirable when the
buffer-deallocation-pipeline is run after bufferization to handle buffer
deallocation.
Currently, the NVPTX tool compilation path only calls `ptxas`; thus, the
GPU running the binary must be an exact match of the arch of the target,
or else the runtime throws an error due to the arch mismatch.
This patch adds a call to `fatbinary`, creating a fat binary with the
cubin object and the PTX code, allowing the driver to JIT the PTX at
runtime if there's an arch mismatch.
Return poison from foldBinary/unary if argument(s) is poison. Add ub dialect as dependency to affected dialects (arith, math, spirv, shape).
Add poison materialization to dialects. Add tests for some ops from each dialect.
Not all affected ops are covered as it will involve a huge copypaste.
Differential Revision: https://reviews.llvm.org/D159013
The substitution supported by `extraClassOf` is currently limited to
only the base instance, i.e. `Operation*`, `Type` or `Attribute`, which
limits the kind of checks you can perform in the `classof`
implementation.
Since prior to the user code, the interface concept is fetched, we can
use it to construct an instance of the interface, allowing use of its
methods in the `classof` check.
Since an instance of the interface allows access to the base class
methods through the `->` operator, I've gone ahead and replaced the
substitution of `$_op/$_type/$_attr` with an interface instance. This is
also consistent with `extraSharedClassDeclaration` and other methods
created in the interface class which do the same.
The OpenACC standard specifies an `atomic` construct in section 2.12 (of
3.3 spec), used to ensure that a specific location is accessed or
updated atomically. Four different clauses are allowed: `read`, `write`,
`update`, or `capture`. If no clause appears, it is as if `update` is
used.
The OpenMP specification defines the same clauses for `omp atomic`. The
types of expression and the clauses in the OpenACC spec match the OpenMP
spec exactly. The main difference is that the OpenMP specification is a
superset - it includes clauses for `hint` and `memory order`. It also
allows conditional expression statements. But otherwise, the expression
definition matches.
Thus, for OpenACC, we refactor and reuse the OpenMP implementation as
follows:
* The atomic operations are duplicated in OpenACC dialect. This is
preferable so that each language's semantics are precisely represented
even if specs have divergence.
* However, since semantics overlap, a common interface between the
atomic operations is being added. The semantics for the interfaces are
not generic enough to be used outside of OpenACC and OpenMP, and thus
new folders were added to hold common pieces of the two dialects.
* The atomic interfaces define common accessors (such as getting `x` or
`v`) which match the OpenMP and OpenACC specs. It also adds common
verifiers intended to be called by each dialect's operation verifier.
* The OpenMP write operation was updated to use `x` and `expr` to be
consistent with its other operations (that use naming based on spec).
The frontend lowering necessary to generate the dialect can also be
reused. This will be done in a follow up change.
One-Shot Bufferize correctly handles RaW conflicts around repetitive regions (loops). Specical handling is needed for parallel regions. These are a special kind of repetitive regions that can have additional RaW conflicts that would not be present if the regions would be executed sequentially.
Example:
```
%0 = bufferization.alloc_tensor()
scf.forall ... {
%1 = linalg.fill ins(...) outs(%0)
...
scf.forall.in_parallel {
tensor.parallel_insert_slice %1 into ...
}
}
```
A separate (private) buffer must be allocated for each iteration of the `scf.forall` loop.
This change adds a new interface method to `BufferizableOpInterface` to detect parallel regions. By default, regions are assumed to be sequential.
A buffer is privatized if an OpOperand bufferizes to a memory read inside a parallel region that is different from the parallel region where operand's value is defined.
Differential Revision: https://reviews.llvm.org/D159286
This patch adds attribute builders for all buildable attributes from the
builtin dialect that did not previously have any. These builders can be
used to construct attributes of a particular type identified by a string
from a Python argument without knowing the details of how to pass that
Python argument to the attribute constructor. This is used, for example,
in the generated code of the Python bindings of ops.
The list of "all" attributes was produced with:
(
grep -h "ods_ir.AttrBuilder.get" $(find ../build/ -name "*_ops_gen.py") \
| cut -f2 -d"'"
git grep -ho "^def [a-zA-Z0-9_]*" -- include/mlir/IR/CommonAttrConstraints.td \
| cut -f2 -d" "
) | sort -u
Then, I only retained those that had an occurence in
`mlir/include/mlir/IR`. In particular, this drops many dialect-specific
attributes; registering those builders is something that those dialects
should do. Finally, I removed those attrbiutes that had a match in
`mlir/python/mlir/ir.py` already and implemented the remaining ones. The
only ones that still miss a builder now are the following:
* Represent more than one possible attribute type:
- `Any.*Attr` (9x)
- `IntNonNegative`
- `IntPositive`
- `IsNullAttr`
- `ElementsAttr`
* I am not sure what "constant attributes" are:
- `ConstBoolAttrFalse`
- `ConstBoolAttrTrue`
- `ConstUnitAttr`
* `Location` not exposed by Python bindings:
- `LocationArrayAttr`
- `LocationAttr`
* `get` function not implemented in Python bindings:
- `StringElementsAttr`
This patch also fixes a compilation problem with
`I64SmallVectorArrayAttr`.
Reviewed By: makslevental, rkayaith
Differential Revision: https://reviews.llvm.org/D159403
Operations that can divide by zero and can have immediate undefined
behaviour should be marked as `NoMemoryEffect` rather than `Pure`.
Depends on D159456
Reviewed By: weiweichen
Differential Revision: https://reviews.llvm.org/D159457
It was common to see `value.getValue().empty()` or `value.size()`
instead of the idiomatic `value.empty()`.
Depends on D159455
Differential Revision: https://reviews.llvm.org/D159456
The primary motivation is to we have a simple mechanism to extract
values from attributes in folders and canon patterns without having to
re-fold constants or write nested conditions over attribute types.
Matching over attributes composes especially well with fold adaptors.
Update folds in Arith and SPIRV dialects to match over attributes, where
applicable.
Reviewed By: mehdi_amini, zero9178
Differential Revision: https://reviews.llvm.org/D159437
Introduce a simple conversion of a memref.alloc/dealloc pair into an
alloca in the same scope. Expose it as a transform op and a pattern.
Allocas typically lower to stack allocations as opposed to alloc/dealloc
that lower to significantly more expensive malloc/free calls. In
addition, this can be combined with allocation hoisting from loops to
further improve performance.
This patch pairs a promised interface with the object (Op/Attr/Type/Dialect) requesting the promise, ie:
```
declarePromisedInterface<MyAttr, MyInterface>();
```
Allowing to make fine grained promises. It also adds a mechanism to query if `Op/Attr/Type` has an specific promise returning true if the promise is there or if an implementation has been added. Finally it adds a couple of `Attr|TypeConstraints` that can be used in ODS to query if the promise or an implementation is there.
This patch tries to solve 2 issues:
1. Different entities cannot use the same promise.
```
declarePromisedInterface<MyInterface>();
// Resolves a promise.
MyAttr1::attachInterface<MyInterface>(ctx);
// Doesn't resolves a promise, as the previous attachment removed the promise.
MyAttr2::attachInterface<MyInterface>(ctx);
```
2. Is not possible to query if a promise has been declared.
Reviewed By: mehdi_amini
Differential Revision: https://reviews.llvm.org/D158464
There are two motivations for this change:
1. It considerably simplifies adding support for the realloc operation to the
new buffer deallocation pass by lowering the realloc such that no
deallocation operation is inserted and the deallocation pass itself can
insert that dealloc
2. The lowering is expressed on a higher level and thus easier to understand,
and the lowerings of the memref operations it is composed of don't have to
be duplicated in the MemRefToLLVM lowering (also see discussion in
https://reviews.llvm.org/D133424)
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D159430
This patch adds a new class Identifier to store identifiers in PresburgerSpace
and their types.
Identifiers were added earlier and were stored as a void pointer, and their type
in the form of mlir::TypeId in PresburgerSpace. To get an identifier, a user of
PresburgerSpace needed to know the type of identifiers. This was a problem for
users of PresburgerSpace like IntegerRelation, which want to work on
identifiers without knowing their type.
The Identifier class allows users like IntegerRelation to work on identifiers
without knowing their type, and also exposes an easier way to work with
Identifiers.
Reviewed By: arjunp
Differential Revision: https://reviews.llvm.org/D146909
The newly folded patterns get produced by type emulation code, so we
cannot rely on the Arith folders handling these folds.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D159394
Printing cyclic attributes and types currently has no first-class
support within the AsmPrinter and AsmParser. The workaround for this
issue used in all mutable attributes and types upstream has been to
create a `thread_local static SetVector` keeping track of currently
parsed and printed attributes.
This solution is not ideal readability wise due to the use of globals
and keeping track of state. Worst of all, this pattern had to be
reimplemented for every mutable attribute and type.
This patch therefore adds support for this pattern in `AsmPrinter` and
`AsmParser` replacing the use of this pattern. By calling
`tryStartCyclingPrint/Parse`, the mutable attribute or type are
registered in an internal stack. All subsequent calls to the function
with the same attribute or type will lead to returning failure. This way
the nesting can be detected and a short form printed or parsed instead.
Through the resetter returned by the call, the cyclic printing or
parsing region automatically ends on return.
