When the affine.parallel op was introduced, affine utilities weren't
extended to handle it. Extending these is straightforward and natural
given that addAffineParallelOpDomain has also been added.
Update/complete memref region compute to account for affine.parallel
ops. Handle failure cleanly.
Add and expose utilities missing for affine.parallel to be consistent
with affine.for.
All of these allow various affine passes to work with a combination of
affine.parallel and affine.for ops.
Differential Revision: https://reviews.llvm.org/D145669
Previously, we generate function calls to compare values for sorting. It turns
out that the compiler doesn't inline those function calls. We now directly
generate inlined code. Also, modify the code for comparing values to use less
number of branches.
This improves all sort implementation in general. For arabic-2005.mtx CSR, the
improvement is around 25%.
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D145442
Add conversion for integer multiplication in scf reductions in the
SCF to OpenMP dialect conversion.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D145948
This commit swaps back the `tosa.reshape` op used at the end of the
lowering for reduce ops with the op `tensor.expand_shape`. This is
needed to properly support dynamically-sized tensors. In such cases,
lowering directly to `tensor.expand_shape` allows us to control which
dimension gets expanded at the end using the knowledge of the
reduction. This would not be possible when using `tosa.reshape`, since
the op does not have a way of knowing that we are only unsqueezing a
single dimension.
Note: this change had previously been performed in
https://reviews.llvm.org/D133877.
Reviewed By: rsuderman
Differential Revision: https://reviews.llvm.org/D145986
The initial implementaiton of value handle update when the payload
operation defining the values associated with value handles was being
replaced required the replacement operation to have the same number of
results. This is not strictly necessary. The replacement operation may
have more results, or less results provided that there are no handles to
the results that have no equivalent in the replacement op.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D145254
This is for targets that do not support gather-like ops, e.g., SPIR-V.
Gather is expanded into lower-level vector ops with memory accesses
guarded with `scf.if`.
I also considered generating `vector.maskedload`s, but decided against
it to keep the `memref` and `tensor` codepath closer together. There's a
good chance that if a target doesn't support gather it does not support
masked loads either.
Issue: https://github.com/llvm/llvm-project/issues/60905
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D145942
Care is taken to order operands from least hoistable to most hoistable and to process subexpressions in the same
order.
This allows exposing more oppportunities for licm, cse and strength reduction.
Such a step should typically be applied while we still have loops in the IR and just before lowering affine ops to arith.
This is because the affine.apply canonicalization currently tries to maximally compose chains of affine.apply operations
and could undo the effects of these decompositions.
Depends on: D145784
Differential Revision: https://reviews.llvm.org/D145685
tosa.concat is lowered to tensor.insert_slice thus it should be in
TosaToTensor rather than in TosaToLinalg.
Reviewed By: rsuderman
Differential Revision: https://reviews.llvm.org/D145952
This is a convenient flag for context where we intend to summarize a top-level
operation without the full-blown regions it may hold.
Differential Revision: https://reviews.llvm.org/D145889
This iterator is similar to `ForwardIterator` but enumerates blocks according to their successor relationship. As a first use case, this new iterator is utilized in the dialect conversion framework.
Differential Revision: https://reviews.llvm.org/D144888
This commit introduces the LLVM's visibility attribute and adds it to
both globals and functions.
Furthermore, this commit ensures that "thread_local" is printed in the
correct place and adds a test for that.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D145790
The inliner pass performs canonicalization when created programtically, run with `mlir-opt` with default options, or when explicitly specified. However, when running the pipeline resulting from a `-dump-pass-pipeline` on a default inline pass, the canonicalization is not performed as part of the inlining. This is because the default value for the `default-pipeline` option of the inline pass is an empty string, and this is selected during the dumping. When `InlinerPass::initializeOptions` detects the empty string, it sets the `defaultPipeline` to `nullptr`, which was previously set to canonicalize in the `InlinerPass` constructor, thus the canonicalization is not performed.
The added test checks if the inline pass performs canonicalization by default, and that the dumped `default-pipeline` is set to `canonicalize`.
Fixes: https://github.com/llvm/llvm-project/issues/60960
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D145066
A dialect can opt-in to handle versioning through the
`BytecodeDialectInterface`. Few hooks are exposed to the dialect to allow
managing a version encoded into the bytecode file. The version is loaded
lazily and allows to retrieve the version information while parsing the input
IR, and gives an opportunity to each dialect for which a version is present
to perform IR upgrades post-parsing through the `upgradeFromVersion` method.
Custom Attribute and Type encodings can also be upgraded according to the
dialect version using readAttribute and readType methods.
There is no restriction on what kind of information a dialect is allowed to
encode to model its versioning. Currently, versioning is supported only for
bytecode formats.
Reviewed By: rriddle, mehdi_amini
Differential Revision: https://reviews.llvm.org/D143647
This converts a specific form of `vector.reduction` to SPIR-V integer
dot product ops.
Add a new test pass to excercise this outside of the main vector to
spirv conversion pass.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D145760
Rationale:
The LLVM dialect supports passing fastmath flags from floating point ops to LLVMIR instructions. However, not all LLVM ops have the required attribute. This change adds support for fastmath flags to `llvm.intr.vector.reduce.{fmin,fmax}`. One scenario where this is useful is in lowering llvm.intr.vector.reduce.{fmax,fmin} to LLVMIR with `nnan` (NoNans) flag so it may be [[ 115c7beda7/llvm/lib/CodeGen/ExpandReductions.cpp (L159) | lowered to a shuffle reduction ]].
Changes:
- Make `LLVM_VecReductionF` implement the `FastmathFlagsInterface`; change is modeled on `LLVM_UnaryIntrOpF`
- Add an assembly format for `LLVM_VecReductionF` ops. The purpose is to keep existing functionality: avoid printing the fastmath flags attribute when it has its default value (`none`). Change is modeled on `LLVM_UnaryIntrOpBase`
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D145692
This change adds a new helper function `mlir::reifyResultShapes` that calls the corresponding interface method and also checks the result produced by the implementation when running in debug mode. Bugs due to incorrect interface implementations can be difficult to debug.
This helper function also reduces the amount of code needed at call sites: the cast to `ReifyRankedShapedTypeOpInterface` is done in the helper function.
Differential Revision: https://reviews.llvm.org/D145777
This has caused build failures when enabling opaque pointers for the GPU integration tests as could be seen here:
https://lab.llvm.org/buildbot/#/builders/220/builds/16946 and here https://lab.llvm.org/buildbot/#/builders/61/builds/40822
The gist of the issue was the use of a wrong pointer base type within a GEP. There sadly was no test coverage for either the generating of that GEP, nor is LLVM Dialects GEP verifier currently capable of catching such issues, so it went unnoticed until the integration tests actually attempted to convert it to LLVM IR.
Differential Revision: https://reviews.llvm.org/D145774
Part of https://discourse.llvm.org/t/rfc-switching-the-llvm-dialect-and-dialect-lowerings-to-opaque-pointers/68179
When this patch lands any downstream users with custom LLVM conversion passes not yet using opaque pointers will start either experiencing assertions being triggered, null pointer dereferences or at the very least verifier errors. These can be either fixed by switching to opaque pointers or simply disabling opaque pointers in both pass options of any upstream conversion passes and any uses of `LLVMTypeConverter` via the `LowerToLLVMOptions`.
Users using just MLIRs conversion passes to the LLVM Dialect should not experience any change in functionality except when inspecting the output from the passes.
Differential Revision: https://reviews.llvm.org/D145585
This change allows using fp8 pointers when exporting to LLVM, because we anyway export them as opaque pointers, however full support of fp8 types is not yet implemented on the LLVM side.
Differential Revision: https://reviews.llvm.org/D143008
This pattern is not specific to nvgpu; I intend to use in SPIR-V codegen. `VectorTransforms` seems like a more generally useful place.
In addition:
- Fix a bug in the second condition (the dimensions were swapped for RHS).
- Add tests.
- Add support for externally provided filter functions, similar to other vector transforms.
- Prefer to transpose before zero/sign-extending inputs.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D145638
The patch adds the lowering from Flang parse-tree to FIR+OpenMP. The
conversion code is also added in MLIR.
Reviewed By: kiranchandramohan
Differential Revision: https://reviews.llvm.org/D133442
Co-authored-by: Kiran Chandramohan <kiran.chandramohan@arm.com>
This revision adds the inalloca attribute to the alloca operation in the LLVMIR dialect.
It also adds tests for import and export.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D145483
Decompose conv_2d -> conv_1d.
This MR follows a similar approach to https://reviews.llvm.org/D112928.
This patch adds support to convert conv_2D operation with either unit height or unit width to conv_1D operation.
This is useful when 2D convolution is tiled to have a single dimension for either height or width and then can be vectorized once it is decomposed into 1D convolution.
This patch https://reviews.llvm.org/D145160 adds vector support for linalg.conv_1d operation and thereby allowing us to vectorize linalg.conv_2d operation after proper tiling.
This missing feature is reported here: https://discourse.llvm.org/t/vectorization-of-convolution-op/60458.
Reviewed By: hanchung
Differential Revision: https://reviews.llvm.org/D145162
Fix bug when pipelining while interleaving stages. Re-do the logic to
only consider cloned operands when updating the use-def chain.
Differential Revision: https://reviews.llvm.org/D145598
Right now the bindings assume that all DenseElementsAttrs correspond to tensor values,
making it impossible to create vector-typed constants. I didn't want to change the API
significantly, so I opted for reusing the current signature of `.get`. Its `type` argument
now accepts both element types (in which case `shape` and `signless` can be specified too),
or a shaped type, which specifies the full type of the created attr (`shape` cannot be specified
in that case).
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D145053
Vectorization of `tensor.extract` using contiguous loads
(`vector.transfer_read`) was introduced in [1]. This patch updates and
refines the existing logic (so that more cases of contiguous can be
identified), as well as adds more tests.
Specifically, contiguous load operations are identified by making sure
that:
1. non-trailing indices for `tensor.extract` are loop invariant (so,
e.g., there are no "jumps" from one row to the other between
iterations),
2. the trailing index for `tensor.extract` increments by 1 with every
loop iteration (so that it's always adjacent elements that are
loaded).
This patch introduces:
* `isLoopInvariantIdx` for step 1., and
* `isContiguousLoadIdx` for step 2.
These new methods replace:
* `isContiguousLoadIdx`, and `isBasedOnIndexOp`.
Both approaches lead to similar end-result (none of the existing tests
required updating). However, with the updated approach, it's much easier
to treat the trailing and non-trailing indices separately and to add
more cases for which contiguous loads can be used.
[1] https://reviews.llvm.org/D141998
Differential Revision: https://reviews.llvm.org/D145385
