-- This commit is the eighth in the series of adding matchers
for linalg.*conv*/*pool*. Refer:
https://github.com/llvm/llvm-project/pull/163724
-- In this commit all variants of Pooling ops have been added.
Signed-off-by: Abhishek Varma <abhvarma@amd.com>
Introduce DefaultValuedEnumAttr, which similarly to DefaultValuedAttr
decorates an enum attribute to have a default value from a specific enum
case when not present. The default is constructed as the fully-qualified
enum case symbol.
In comparison to DefaultValuedAttr, this allows using a TableGen
EnumCase
variable instead of a raw string.
This change adds full support for the ptx `barrier.cta.red` instruction,
following the same conventions as are already used for
`barrier.cta.sync` and `barrier.cta.arrive`.
In addition this MR removes the following intrinsics which are no longer
needed:
* llvm.nvvm.barrier0.popc -->
llvm.nvvm.barrier.cta.red.popc.aligned.all(0, c)
* llvm.nvvm.barrier0.and -->
llvm.nvvm.barrier.cta.red.and.aligned.all(0, z)
* llvm.nvvm.barrier0.or -->
llvm.nvvm.barrier.cta.red.or.aligned.all(0, z)
If source strided memref is not fully static - at least one of shape,
strides, offset is kDynamic - use i64 source variant.
With this change, xegpu.create_nd_tdesc created by lowering from vector
dialect, can rely on getMixedOffsets, getMixedSize and getMixedStrides
to get relevant values.
Extend the `ControlDropUnitDims` struct to allow users of the
`linalg::dropUnitDims` function more control over the behavior of the
function.
The extended struct allows users to specify functions to control how the
operands are collapsed and how the result is expanded to the original
shape.
One example (and the motivation for this change) where this additional
control is useful is to allow collapsing of tensors with an encoding, as
demonstrated by the new test.
This is a breaking change. The new default behavior changes to abort the
transformation if one of the operands cannot be collapsed or if the
result cannot be expanded. This is the case for `memref` with
non-identity layout and `tensor`s with an encoding.
---------
Signed-off-by: Lukas Sommer <lukas.sommer@amd.com>
A sneaky operator precedence bug caused this resize operation to always
truncate to size 0 or 1:
```
probeAlias.resize(alias.size() + isdigit(alias.back()) ? 1 : 0);
```
Because `+` is associated more strongly than the ternary operator. This
eventually led to the asm printer repeating an alias name, generating
illegal IR.
It wasn't a problem in most cases because it required two things to
trigger:
- Two naturally generated aliases, one "xxx" the other "xxx1" (note the
trailing "1").
- A unique processing order such that we process "xxx", then "xxx1",
then "xxx" again. This can only happen if they happen to be at different
"alias depths", since otherwise the pre-sorting will make sure this
ordering never happens. See the added test case for how this works in
practice (I will also attach what the current code generates).
This PR fixes the operator precedence, but also moves the calculation
outside the loop since it never changes.
This PR builds on the anchor layout mechanism introduced in
https://github.com/llvm/llvm-project/pull/169267 and performs the
following refactoring:
1. Introduce getAnchorLayout() and setAnchorLayout() interface for
anchor ops to get and set layout attributes.
2. Add getLocalLayout() and setLocalLayout() utility functions, and
refactor workgroup/subgroup distribution patterns to use these APIs.
These utilities access the layout information directly and locally,
without relying on global propagation.
3. Introduce localPropagateLayoutsFromAnchor(), a utility used by
subgroup distribution to unify non-anchor layout setup.
This function is intended to be invoked upfront by all layout-based
passes (including workgroup/subgroup distribution and unrolling) to
propagate layouts from anchor ops to non-anchor ops.
After this step, patterns within the pass should exclusively use
getLocalLayout() / setLocalLayout().
4. Refactor getDistributeLayoutAttr() and setDistributeLayoutAttr() to
remove special-case handling. These APIs now operate in a uniform order:
anchor ops first, then non-anchor ops, and finally block arguments.
These APIs will be deprecated on long run.
5. Refactor patterns in wg/sg distribution, load optimization passes to
use get/setAnchorLayout() and get/setLocalLayout().
6. Update test cases to enforce that anchor ops must use—and only
use—anchor layouts.
The MLIR [GPU dialect
docs](https://mlir.llvm.org/docs/Dialects/GPU/#gpubarrier-gpubarrierop)
specify that gpu::BarrierOp should make *all memory accesses* visible to
all work items in the workgroup.
Current implementation uses only CLK_LOCAL_MEM_FENCE, which per the
[OpenCL
specification](https://registry.khronos.org/OpenCL/sdk/3.0/docs/man/html/barrier.html)
guarantees visibility of
only *local memory accesses*.
This PR changes the barrier conversion to use CLK_LOCAL_MEM_FENCE |
CLK_GLOBAL_MEM_FENCE,
ensuring both local and global memory operations are properly
synchronized per the MLIR spec.
This issue was discovered while investigating numerical instabilities on
Intel Battlemage,
where race conditions occurred due to incomplete memory synchronization.
Extend the `extractvalue` fold method to support extracting from
constant containers, such as `llvm.mlir.zero`, `llvm.mlir.undef`,
`llvm.mlir.poison`, and `llvm.mlir.constant` holding `ElementsAttr` or
`ArrayAttr`.
A `transform` pass to lower `BF16` type `vector.contract` to
`vector.fma` using `AVX2` BF16 packed operations:
- `vbcstnebf162ps` - Broadcasts BF16 into packed F32.
- `vcvtneebf162ps` - Convert packed BF16 even-indexed elements into
packed F32.
- `vcvtneobf162ps` - Convert packed BF16 odd-indexed elements into
packed F32 Data.
We have the most basic mbarrier.arrive supported on sm_80.
It supports: Space=cta, scope=cta, No relaxed, No explicit count.
This patch updates the lowering to the legacy intrinsic when all
these conditions are met, addressing a lowering issue for sm_80.
Signed-off-by: Durgadoss R <durgadossr@nvidia.com>
Add special-case handling for ShapeCast when it expands unit dimensions
for a succeeding broadcast op. In this scenario, distribution requires
the source layout to be a slice layout, and the result layout is first
normalized by setting the expanded unit dimensions to 1 before computing
the distributed result shape. In all other cases, ShapeCast is
distributed as usual.
This PR also updates the propagation rule for vectors with expanded unit
dimensions, allowing them to share the same layout as the result of a
broadcast op. This enables correct layout propagation back to the source
of the ShapeCast op, as that layout must ultimately be restored as the
parent layout of the slice layout.
This commit align the implementation of
`ConversionPatternRewriter::legalize` with its documentation:
```
/// Attempt to legalize the given region. This can be used within
...
LogicalResult legalize(Region *r);
```
This function now legalizes the entire region, including nested ops. The
implementation follows the same logic as the "main" traversal:
pre-order, forward-dominance.
- unify the i64->i32 and f64->f32 narrowing logic inside the shared
implementation
- register tosa::ConstOp in the non-aggressive rewrite set so standalone
constants are narrowed
---------
Signed-off-by: Vitalii Shutov <vitalii.shutov@arm.com>
Co-authored-by: Luke Hutton <Luke.Hutton@arm.com>
This commit allows tensor dimensions to be dynamic when the specified
target TOSA specification version is `1.1.draft` or higher. This is
because this version of the specification supports representation
operations that are dynamic until backend compile time.
