This commit renames 4 pattern rewriter API functions:
* `updateRootInPlace` -> `modifyOpInPlace`
* `startRootUpdate` -> `startOpModification`
* `finalizeRootUpdate` -> `finalizeOpModification`
* `cancelRootUpdate` -> `cancelOpModification`
The term "root" is a misnomer. The root is the op that a rewrite pattern
matches against
(https://mlir.llvm.org/docs/PatternRewriter/#root-operation-name-optional).
A rewriter must be notified of all in-place op modifications, not just
in-place modifications of the root
(https://mlir.llvm.org/docs/PatternRewriter/#pattern-rewriter). The old
function names were confusing and have contributed to various broken
rewrite patterns.
Note: The new function names use the term "modify" instead of "update"
for consistency with the `RewriterBase::Listener` terminology
(`notifyOperationModified`).
This extension has been superseded by SPV_KHR_cooperative_matrix which
is supported across major vendors GPU like Nvidia, AMD, and Intel.
Given that the KHR version has been supported for nearly half a year,
drop the NV-specific extension to reduce the maintenance burden and code
duplication.
see #73359
Declarative assemblyFormat ODS is more concise and requires less
boilerplate than filling out CPP interfaces.
Changes:
* updates the Ops defined in `SPIRVAtomicOps.td` to use assemblyFormat.
* Removes print/parse from`AtomcOps.cpp` which is now generated by
assemblyFormat
* Adds `Trait` to verify that a pointer operand `foo`'s pointee type
matches operand `bar`'s type
* * Updates error message expected in tests from new Trait
* Updates tests to updated format (largely using <operand> in place of
"operand")
Changes include:
- spirv serialization and deserialization needs handling in cases when
GlobalVariableOp initializer is defined using spirv SpecConstant or
SpecConstantComposite op, currently even though it allows SpecConst, it
only looked up in for GlobalVariable Map to find initializer symbol
reference, change is fixing this and extending the support to
SpecConstantComposite as an initializer.
- Adds tests to make sure GlobalVariable can be initialized using
specialized constants.
---------
Co-authored-by: Lei Zhang <antiagainst@gmail.com>
Add missing constant propogation folder for SNegate, [Logical]Not.
Implement additional folding when !(!x) for all ops.
This helps for readability of lowered code into SPIR-V.
Part of work for #70704
Add missing constant propogation folder for Bitwise[Or|And|Xor].
Move previous Bitwise[Or|And] fold implementations to
SPIRVCanonicalization for consistency.
Implement additional folding when lhs == rhs and rhs = 0 for Xor. As
well as, update an Xor testcase to account for this introduced folding.
This helps for readability of lowered code into SPIR-V.
Part of work for #70704
Add missing constant propogation folder for LeftShiftLogical,
RightShift[Logical|Arithmetic].
Implement additional folding when Shift value is 0.
This helps for readability of lowered code into SPIR-V.
Part of work for #70704
Add missing constant propogation folder for IAddCarry and
[S|U]MulExtended. Due to currently missing constant value for
spirv.struct the folding is done using canonicalization patterns.
Implement additional folding when rhs is 0 for all ops and when rhs is 1
for UMulExt.
This helps for readability of lowered code into SPIR-V.
Part of work for #70704
Add missing constant propogation folder for [S|U]Mod, [S|U]Div, SRem
Implement additional folding when rhs is 1 for all ops.
This helps for readability of lowered code into SPIR-V.
Part of work for #70704
This covers the following ops:
`spirv.GroupNonUniform` x {`Bitwise`, `Logical`} x {`And`, `Or`, `Xor`}
We need these to efficiently lower from the `gpu.subgroup_reduce` op.
Make the syntax more concise and aligned with the `spirv.Dot` syntax in
https://github.com/llvm/llvm-project/pull/73466.
Move some type verification from C++ to ODS.
Regexes to update existing code and tests:
`(\s*\{format\s+=\s+#spirv.packed_vector_format([^}]+)\})`
==>
`, $2`
`(spirv.[SU]+Dot[a-zA-Z]*[^:]+:)(\s*\(([^,]+),[^\)]+\))(.+)`
==>
`$1 $3$4`
MLIR can't really be const-correct (it would need a `ConstValue` class
alongside the `Value` class really, like `ArrayRef` and
`MutableArrayRef`). This is however making is more consistent: method
that are directly modifying the Value shouldn't be marked const.
This commit implements gpu::TargetAttrInterface for SPIR-V target
attribute. The plan is to use this to enable GPU compilation pipeline
for OpenCL kernels later.
The changes do not impact Vulkan shaders using milr-vulkan-runner.
New GPU Dialect transform pass spirv-attach-target is implemented for
attaching attribute from CLI.
gpu-module-to-binary pass now works with GPU module that has SPIR-V
module with OpenCL kernel functions inside.
LLVMStructTypes could be emitted with some null elements. This caused a
crash later in the LLVMDialect verifier.
We now use `convertTypes` and check that all types were successfully converted before passing them to the `LLVMStructType` constructor.
See #59990
WebGPU does not currently support extended arithmetic, this is an issue
when we want to lower from SPIR-V. This commit adds a pattern to
transform and emulate spirv.IAddCarry with spirv.IAdd operations
Fixes#65154
Variables that point to physical storage buffer require aliasing
decorations. This is specified by the `SPV_KHR_physical_storage_buffer`
extension.
Also add an example of a variable with a decoration attribute.
- Check `MakePointer*` load/store attribute values.
- Support coop matrix types in `MatrixTimesScalar` verification.
- Add test cases for all the remaining ops that accept coop matrix types.
- Split NV and KHR tests.
- Fix values of Matrix Operand bit enums.
- Add verification for the aligned Memory Operand attributes. Mark the
'Aligned' enumerant as not supported.
The target test passes validation with `spirv-val`.
Extend SPIR-V target serialization and deserialization to handle coop
matrix types. Add a roundtrip test. In addition to `FileCheck` checks,
the resulting spirv binary also passes `spir-val` (external tool).
Also fix a type attribute bug surfaced by the `CooperativeMatrixLength`
op.
Multiple matrix operand attributes will be handled in a future patch to
reduce the scope.
- Fix order of operands/attributes
- Allow for stride to be any integer type
- Use ODS for parsing/printing
- Update examples and tests
- Fix a typo in SPIR-V tblgen code
Return poison from foldBinary/unary if argument(s) is poison. Add ub dialect as dependency to affected dialects (arith, math, spirv, shape).
Add poison materialization to dialects. Add tests for some ops from each dialect.
Not all affected ops are covered as it will involve a huge copypaste.
Differential Revision: https://reviews.llvm.org/D159013
The primary motivation is to we have a simple mechanism to extract
values from attributes in folders and canon patterns without having to
re-fold constants or write nested conditions over attribute types.
Matching over attributes composes especially well with fold adaptors.
Update folds in Arith and SPIRV dialects to match over attributes, where
applicable.
Reviewed By: mehdi_amini, zero9178
Differential Revision: https://reviews.llvm.org/D159437
The newly folded patterns get produced by type emulation code, so we
cannot rely on the Arith folders handling these folds.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D159394
Printing cyclic attributes and types currently has no first-class
support within the AsmPrinter and AsmParser. The workaround for this
issue used in all mutable attributes and types upstream has been to
create a `thread_local static SetVector` keeping track of currently
parsed and printed attributes.
This solution is not ideal readability wise due to the use of globals
and keeping track of state. Worst of all, this pattern had to be
reimplemented for every mutable attribute and type.
This patch therefore adds support for this pattern in `AsmPrinter` and
`AsmParser` replacing the use of this pattern. By calling
`tryStartCyclingPrint/Parse`, the mutable attribute or type are
registered in an internal stack. All subsequent calls to the function
with the same attribute or type will lead to returning failure. This way
the nesting can be detected and a short form printed or parsed instead.
Through the resetter returned by the call, the cyclic printing or
parsing region automatically ends on return.
Functions are always callable operations and thus every operation
implementing the `FunctionOpInterface` also implements the
`CallableOpInterface`. The only exception was the FuncOp in the toy
example. To make implementation of the `FunctionOpInterface` easier,
this commit lets `FunctionOpInterface` inherit from
`CallableOpInterface` and merges some of their methods. More precisely,
the `CallableOpInterface` has methods to get the argument and result
attributes and a method to get the result types of the callable region.
These methods are always implemented the same way as their analogues in
`FunctionOpInterface` and thus this commit moves all the argument and
result attribute handling methods to the callable interface as well as
the methods to get the argument and result types. The
`FuntionOpInterface` then does not have to declare them as well, but
just inherits them from the `CallableOpInterface`.
Adding the inheritance relation also required to move the
`FunctionOpInterface` from the IR directory to the Interfaces directory
since IR should not depend on Interfaces.
Reviewed By: jpienaar, springerm
Differential Revision: https://reviews.llvm.org/D157988
ConversionPatterns do not (and should not) modify the type converter that they are using.
* Make `ConversionPattern::typeConverter` const.
* Make member functions of the `LLVMTypeConverter` const.
* Conversion patterns take a const type converter.
* Various helper functions (that are called from patterns) now also take a const type converter.
Differential Revision: https://reviews.llvm.org/D157601
This would mean allowing vector type conversion involving sub-byte
element types.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D157791
Add a method to the CallOpInterface to get a mutable operand range over
the function arguments. This allows to add, remove, or change the type
of call arguments in a generic manner without having to assume that the
argument operand range is at the end of the operand list, or having to
type switch on all supported concrete operation kinds.
Alternatively, a new OpInterface could be added which inherits from
CallOpInterface and appends it with the mutable variants of the base
interface.
There will be two users of this new function in the beginning:
(1) A few passes in the Arc dialect in CIRCT already use a downstream
implementation of the alternative case mentioned above: https://github.com/llvm/circt/blob/main/include/circt/Dialect/Arc/ArcInterfaces.td#L15
(2) The BufferDeallocation pass will be modified to be able to pass
ownership of memrefs to called private functions if the caller does not
need the memref anymore by appending the function argument list with a
boolean value per memref, thus enabling earlier deallocation of the
memref which can lead to lower peak memory usage.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D156675
1-element vectors are not valid in SPIR-V and fail `Bitcast` op verification.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D156207
Continue to work outlined in D155747 and split the main SPIR-V ops
implementation file into a few smaller and quicker to compile files.
Move control flow and memory ops to their own implementation files.
Create new `.cpp` files for tablegened code.
After this change, the `SPIRVOps.cpp` is 2k LoC-long and takes a
reasonable amount of time to compile.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D155883
Continue to work outlined in D155747 and split the main SPIR-V ops
implementation file into a few smaller and quicker to compile files.
This organization matches the op definition organizaion in `.td` files.
In this patch, extract atomic, cast/conversion, and group op
implementation into separate files.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D155777
The main op implementation file for SPIR-V grew past 5k LOC. This makes it
take a long time to compile and index with LSPs like clangd.
Pull out the first few SPIR-V extension ops into their own `.cpp` files,
just like we do with `.td` op definitions. This includes the
KHR/NV/Intel coop matrix and the integer dot prod extensions.
I plan to further split this in future revisions.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D155747
In SPIR-V, the capabilities for storage and compute are separate.
We have good handling of the storage side in general via MemRef
type conversion and various `memref` dialect ops.
Once the value was loaded properly, if the compute capability is
supported directly, we don't need to emulate like the storage side
with int32. However, we do need to make sure casting ops are
properly inserted to chain the flow to go back to the original
bitwidth.
Right now that is done in the each individual pattern directly,
which put lots of pressure that shouldn't be on the patterns and
causes duplication and trickiness w.r.t. capability check and such.
Instead, we should handle such casting within the SPIR-V conversion
framework using `addSourceMaterialization`, where we can check with
the target environment to make sure the corresponding compute
capability is allowed and then we can materialize and SPIR-V casting
op.
Along the way, we can drop all the duplicated cast materialization
registration in various places.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D155118
This is in preparation for adding the KHR version of the cooperative
matrix extension, `SPV_KHR_cooperative_matrix`, that comes with
equivalent ops and type. These are not cross-extension compatible,
so it's better to add prefixes/suffixes to the Nvidia one,
`SPV_NV_cooperative_matrix`, before adding the KHR counterparts.
In near future, I plan for these two extensions to co-exist in
the SPIR-V dialect, but we may want to remove the NV one at some point.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D154799
The SPIR-V spec does not specify the mangling for these variables, so
the conversion to SPIR-V should be flexible enough to allow adding a
custom prefix and suffix to the core name.
Differential Revision: https://reviews.llvm.org/D153951
Signed-off-by: Victor Perez <victor.perez@codeplay.com>
