When the affine.parallel op was introduced, affine utilities weren't
extended to handle it. Extending these is straightforward and natural
given that addAffineParallelOpDomain has also been added.
Update/complete memref region compute to account for affine.parallel
ops. Handle failure cleanly.
Add and expose utilities missing for affine.parallel to be consistent
with affine.for.
All of these allow various affine passes to work with a combination of
affine.parallel and affine.for ops.
Differential Revision: https://reviews.llvm.org/D145669
This is a clean up before fixing issues identified in this pass by
https://github.com/llvm/llvm-project/issues/61380 and similar issues.
- Move patterns definitions closer to declarations.
- Simplify pattern definitions.
- Drop hand-written pass constructor in favor of an auto-generated on.
- Fix typos in pass description.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D146077
This change does a bunch of renaming to clear up confusions in these files. In particular, this change:
* Renames variables and methods to clarify the "dim"/"lvl" distinction, and changes them to use the `Dimension`/`Level` types as appropriate.
* Introduces new typedefs
* `ExprId`, `LatPointId`, `LatSetId`: to clarify the interning design of the Merger.
* `LoopId`, `LoopOrd`: to clarify the distinction between arbitrary names for loop-variables, vs numeric identifiers based on the actual order of loop generation.
* `TensorId`
* (Future CLs will change these from typedefs to structs/classes, so that the typechecker can help avoid mixups.)
* Updates documentation to match the new terminology
* Adds additional assertions
* Adds `const` to local variables along the way
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D145756
This is for targets that do not support gather-like ops, e.g., SPIR-V.
Gather is expanded into lower-level vector ops with memory accesses
guarded with `scf.if`.
I also considered generating `vector.maskedload`s, but decided against
it to keep the `memref` and `tensor` codepath closer together. There's a
good chance that if a target doesn't support gather it does not support
masked loads either.
Issue: https://github.com/llvm/llvm-project/issues/60905
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D145942
Care is taken to order operands from least hoistable to most hoistable and to process subexpressions in the same
order.
This allows exposing more oppportunities for licm, cse and strength reduction.
Such a step should typically be applied while we still have loops in the IR and just before lowering affine ops to arith.
This is because the affine.apply canonicalization currently tries to maximally compose chains of affine.apply operations
and could undo the effects of these decompositions.
Depends on: D145784
Differential Revision: https://reviews.llvm.org/D145685
Reword the vector gather and scatter op description to make it
well-defined to have out-of-bounds indices when the corresponding mask
bits are false.
Update the code sample to avoid relying on C UB semantics to provide
informal semantics for vector.gather.
This change should be consistent with the existing interpretation of the
semantics in the codebase.
Issue: https://github.com/llvm/llvm-project/issues/60905
Reviewed By: dcaballe
Differential Revision: https://reviews.llvm.org/D145824
This is a convenient flag for context where we intend to summarize a top-level
operation without the full-blown regions it may hold.
Differential Revision: https://reviews.llvm.org/D145889
This iterator is similar to `ForwardIterator` but enumerates blocks according to their successor relationship. As a first use case, this new iterator is utilized in the dialect conversion framework.
Differential Revision: https://reviews.llvm.org/D144888
This component acts as an action handler that can be registered in the
MLIRContext. It is the main orchestration of the infrastructure, and implements
support for clients to hook there and snoop on or control the execution.
This is the basis to build tracing as well as a "gdb-like" control of the
compilation flow.
The ExecutionContext acts as a handler in the MLIRContext for executing an
Action. When an action is dispatched, it'll query its set of Breakpoints
managers for a breakpoint matching this action. If a breakpoint is hit, it
passes the action and the breakpoint information to a callback. The callback
is responsible for controlling the execution of the action through an enum
value it returns. Optionally, observers can be registered to be notified
before and after the callback is executed.
Differential Revision: https://reviews.llvm.org/D144812
This commit introduces the LLVM's visibility attribute and adds it to
both globals and functions.
Furthermore, this commit ensures that "thread_local" is printed in the
correct place and adds a test for that.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D145790
The inliner pass performs canonicalization when created programtically, run with `mlir-opt` with default options, or when explicitly specified. However, when running the pipeline resulting from a `-dump-pass-pipeline` on a default inline pass, the canonicalization is not performed as part of the inlining. This is because the default value for the `default-pipeline` option of the inline pass is an empty string, and this is selected during the dumping. When `InlinerPass::initializeOptions` detects the empty string, it sets the `defaultPipeline` to `nullptr`, which was previously set to canonicalize in the `InlinerPass` constructor, thus the canonicalization is not performed.
The added test checks if the inline pass performs canonicalization by default, and that the dumped `default-pipeline` is set to `canonicalize`.
Fixes: https://github.com/llvm/llvm-project/issues/60960
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D145066
A dialect can opt-in to handle versioning through the
`BytecodeDialectInterface`. Few hooks are exposed to the dialect to allow
managing a version encoded into the bytecode file. The version is loaded
lazily and allows to retrieve the version information while parsing the input
IR, and gives an opportunity to each dialect for which a version is present
to perform IR upgrades post-parsing through the `upgradeFromVersion` method.
Custom Attribute and Type encodings can also be upgraded according to the
dialect version using readAttribute and readType methods.
There is no restriction on what kind of information a dialect is allowed to
encode to model its versioning. Currently, versioning is supported only for
bytecode formats.
Reviewed By: rriddle, mehdi_amini
Differential Revision: https://reviews.llvm.org/D143647
This converts a specific form of `vector.reduction` to SPIR-V integer
dot product ops.
Add a new test pass to excercise this outside of the main vector to
spirv conversion pass.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D145760
Rationale:
The LLVM dialect supports passing fastmath flags from floating point ops to LLVMIR instructions. However, not all LLVM ops have the required attribute. This change adds support for fastmath flags to `llvm.intr.vector.reduce.{fmin,fmax}`. One scenario where this is useful is in lowering llvm.intr.vector.reduce.{fmax,fmin} to LLVMIR with `nnan` (NoNans) flag so it may be [[ 115c7beda7/llvm/lib/CodeGen/ExpandReductions.cpp (L159) | lowered to a shuffle reduction ]].
Changes:
- Make `LLVM_VecReductionF` implement the `FastmathFlagsInterface`; change is modeled on `LLVM_UnaryIntrOpF`
- Add an assembly format for `LLVM_VecReductionF` ops. The purpose is to keep existing functionality: avoid printing the fastmath flags attribute when it has its default value (`none`). Change is modeled on `LLVM_UnaryIntrOpBase`
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D145692
This change adds a new helper function `mlir::reifyResultShapes` that calls the corresponding interface method and also checks the result produced by the implementation when running in debug mode. Bugs due to incorrect interface implementations can be difficult to debug.
This helper function also reduces the amount of code needed at call sites: the cast to `ReifyRankedShapedTypeOpInterface` is done in the helper function.
Differential Revision: https://reviews.llvm.org/D145777
Part of https://discourse.llvm.org/t/rfc-switching-the-llvm-dialect-and-dialect-lowerings-to-opaque-pointers/68179
When this patch lands any downstream users with custom LLVM conversion passes not yet using opaque pointers will start either experiencing assertions being triggered, null pointer dereferences or at the very least verifier errors. These can be either fixed by switching to opaque pointers or simply disabling opaque pointers in both pass options of any upstream conversion passes and any uses of `LLVMTypeConverter` via the `LowerToLLVMOptions`.
Users using just MLIRs conversion passes to the LLVM Dialect should not experience any change in functionality except when inspecting the output from the passes.
Differential Revision: https://reviews.llvm.org/D145585
This pattern is not specific to nvgpu; I intend to use in SPIR-V codegen. `VectorTransforms` seems like a more generally useful place.
In addition:
- Fix a bug in the second condition (the dimensions were swapped for RHS).
- Add tests.
- Add support for externally provided filter functions, similar to other vector transforms.
- Prefer to transpose before zero/sign-extending inputs.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D145638
Add functions to set the verifier, printer, and parser of
dynamic attributes definitions, and dynamic type definitions.
This feature was already implemented for dynamic operations, but
is missing for attributes and types. This is necessary to define
attributes and types verifiers that refer to each others in a cyclic way.
Differential Revision: https://reviews.llvm.org/D144690
It also simplifies the implementation of the method. The map is not needed in the check.
Reviewed By: chelini
Differential Revision: https://reviews.llvm.org/D145522
This revision adds the inalloca attribute to the alloca operation in the LLVMIR dialect.
It also adds tests for import and export.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D145483
/data/jiefu/llvm-project/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorType.h:63:21: error: definition of implicit copy constructor for 'SparseTensorType' is deprecated because it has a user-declared copy assignment operator [-Werror,-Wdeprecated-copy]
SparseTensorType &operator=(const SparseTensorType &) = delete;
^
/data/jiefu/llvm-project/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorStorageLayout.h:239:9: note: in implicit copy constructor for 'mlir::sparse_tensor::SparseTensorType' first required here
: rType(stt), fields(fields) {
^
1 error generated.
The copy assignment is already implicitly deleted, but making it explicit helps clean up compilation error messages.
Reviewed By: Peiming
Differential Revision: https://reviews.llvm.org/D145606
Decompose conv_2d -> conv_1d.
This MR follows a similar approach to https://reviews.llvm.org/D112928.
This patch adds support to convert conv_2D operation with either unit height or unit width to conv_1D operation.
This is useful when 2D convolution is tiled to have a single dimension for either height or width and then can be vectorized once it is decomposed into 1D convolution.
This patch https://reviews.llvm.org/D145160 adds vector support for linalg.conv_1d operation and thereby allowing us to vectorize linalg.conv_2d operation after proper tiling.
This missing feature is reported here: https://discourse.llvm.org/t/vectorization-of-convolution-op/60458.
Reviewed By: hanchung
Differential Revision: https://reviews.llvm.org/D145162
Adds the -fopenmp-is-device flag to bbc and Flang's -fc1 (but not flang-new) and in addition adds an omp.is_device attribute onto the module when fopenmp is passed, this is a boolean attribute that is set to false or true dependent on if fopenmp-is-device is specified alongside the fopenmp flag on the commandline when invoking flang or bbc.
Reviewers:
awarzynski
kiranchandramohan
Differential Revision: https://reviews.llvm.org/D144864
The concept of the ActionManager acts as a sort of "Hub" that can receive
various types of action and dispatch them to a set of registered handlers.
One handler will handle the action or it'll cascade to other handlers.
This model does not really fit the current evolution of the Action tracing
and debugging: we can't foresee a good case where this behavior compose with
the use-case behind the handlers. Instead we simplify it with a single
callback installed on the Context.
Differential Revision: https://reviews.llvm.org/D144811
The old "pointer/index" names often cause confusion since these names clash with names of unrelated things in MLIR; so this change rectifies this by changing everything to use "position/coordinate" terminology instead.
In addition to the basic terminology, there have also been various conventions for making certain distinctions like: (1) the overall storage for coordinates in the sparse-tensor, vs the particular collection of coordinates of a given element; and (2) particular coordinates given as a `Value` or `TypedValue<MemRefType>`, vs particular coordinates given as `ValueRange` or similar. I have striven to maintain these distinctions
as follows:
* "p/c" are used for individual position/coordinate values, when there is no risk of confusion. (Just like we use "d/l" to abbreviate "dim/lvl".)
* "pos/crd" are used for individual position/coordinate values, when a longer name is helpful to avoid ambiguity or to form compound names (e.g., "parentPos"). (Just like we use "dim/lvl" when we need a longer form of "d/l".)
I have also used these forms for a handful of compound names where the old name had been using a three-letter form previously, even though a longer form would be more appropriate. I've avoided renaming these to use a longer form purely for expediency sake, since changing them would require a cascade of other renamings. They should be updated to follow the new naming scheme, but that can be done in future patches.
* "coords" is used for the complete collection of crd values associated with a single element. In the runtime library this includes both `std::vector` and raw pointer representations. In the compiler, this is used specifically for buffer variables with C++ type `Value`, `TypedValue<MemRefType>`, etc.
The bare form "coords" is discouraged, since it fails to make the dim/lvl distinction; so the compound names "dimCoords/lvlCoords" should be used instead. (Though there may exist a rare few cases where is is appropriate to be intentionally ambiguous about what coordinate-space the coords live in; in which case the bare "coords" is appropriate.)
There is seldom the need for the pos variant of this notion. In most circumstances we use the term "cursor", since the same buffer is reused for a 'moving' pos-collection.
* "dcvs/lcvs" is used in the compiler as the `ValueRange` analogue of "dimCoords/lvlCoords". (The "vs" stands for "`Value`s".) I haven't found the need for it, but "pvs" would be the obvious name for a pos-`ValueRange`.
The old "ind"-vs-"ivs" naming scheme does not seem to have been sustained in more recent code, which instead prefers other mnemonics (e.g., adding "Buf" to the end of the names for `TypeValue<MemRefType>`). I have cleaned up a lot of these to follow the "coords"-vs-"cvs" naming scheme, though haven't done an exhaustive cleanup.
* "positions/coordinates" are used for larger collections of pos/crd values; in particular, these are used when referring to the complete sparse-tensor storage components.
I also prefer to use these unabbreviated names in the documentation, unless there is some specific reason why using the abbreviated forms helps resolve ambiguity.
In addition to making this terminology change, this change also does some cleanup along the way:
* correcting the dim/lvl terminology in certain places.
* adding `const` when it requires no other code changes.
* miscellaneous cleanup that was entailed in order to make the proper distinctions. Most of these are in CodegenUtils.{h,cpp}
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D144773
This is a preparation for adding support for more infrastructure around the concept
of Action and make tracing Action more of a first class concept.
The doc will be updated later in a subsequent revision after the changes are
completed.
Action belongs to IR because of circular dependency: Actions are dispatched through
the MLIRContext but Action will learn to encapsulate IR construct.
Differential Revision: https://reviews.llvm.org/D144809
At the moment, we invoke `shouldExecute()` that way:
```
if (manager.shouldExecute<DebugAction>(currentOp) {
// apply a transformation
…
}
```
In this sequence, the manager isn’t involved in the actual execution
of the action and can’t develop rich instrumentations. Instead the API
could let the control to the handler itself:
```
// Execute the action under the control of the manager
manager.execute<DebugAction>(currentOp, [&]() {
// apply the transformation in this callback
…
});
```
This inversion of control (by injecting a callback) allows handlers to
implement potentially new interesting features: for example, snapshot
the IR before and after the action, or record an action execution time.
More importantly, it will allow to capture the nesting execution of
actions.
On the other side: handlers receives now a DebugAction object that wraps
generic information (tag and description especially) as well as
action-specific data.
Finally, the DebugActionManager is now enabled in release builds as
well.
Differential Revision: https://reviews.llvm.org/D144808
This allows for downstream *-opt tools to stay always aligned with the options
exposed by mlir-opt.
It aligns the "generic" options with the more "components" ones like the
pass manager options or the context options.
Differential Revision: https://reviews.llvm.org/D144782
* `RewriterBase::mergeBlocks` is simplified: it is implemented in terms of `mergeBlockBefore`.
* The signature of `mergeBlockBefore` is consistent with other API (such as `inlineRegionBefore`): an overload for a `Block::iterator` is added.
* Additional safety checks are added to `mergeBlockBefore`: detect cases where the resulting IR could be invalid (no more `dropAllUses`) or partly unreachable (likely a case of incorrect API usage).
* Rename `mergeBlockBefore` to `inlineBlockBefore`.
Differential Revision: https://reviews.llvm.org/D144969
Each user of the original value is modified in-place. Therefore, the corresponding notification should be triggered.
Also fixes a bug where `RewriterBase::mergeBlocks` did not notify the GreedyPatternRewriteDriver when replacing uses of block arguments. This function does not trigger "operation replaced" notifications, so the GreedyPatternRewriteDriver was not made aware of such IR changes.
Differential Revision: https://reviews.llvm.org/D144549
This allows users to provide custom `Iterator` templates. A new iterator will be added in a subsequent change.
Also rename `makeRange` to `makeIterable` and add a test case for the reverse iterator.
Differential Revision: https://reviews.llvm.org/D144887
`reifyResultShapes` now returns `OpFoldResult`s instead of `Value`s. This is often more efficient because many transformations immediately attempt to extract a constant from the reified values.
Differential Revision: https://reviews.llvm.org/D145250
scf.for loop was restricted to only operate on Index type since
splitting out from affine.for. Relax requirement to allow for signless
integer types additionally. This allows specifying explicitly different
bitwidths for different loops as well as specialize from index to iN
while still using scf.for.
Differential Revision: https://reviews.llvm.org/D145288
`parseAttribute` and `parseType` require null-terminated strings as
input, but this isn't great considering the argument type is
`StringRef`. This changes them to copy to a null-terminated buffer by
default, with a `isKnownNullTerminated` flag added to disable the
copying.
closes#58964
Reviewed By: rriddle, kuhar, lattner
Differential Revision: https://reviews.llvm.org/D145182
Love or hate it, but the vector.print operation was the very
first operation that actually made "end-to-end" CHECK integration
testing possible for MLIR. This revision adds support for
the -until recently- less common but important floating-point
types f16 and bf16.
This will become useful for accelerator specific testing (e.g. NVidia GPUs)
Reviewed By: wrengr
Differential Revision: https://reviews.llvm.org/D145207
