1047 Commits

Author SHA1 Message Date
Maksim Levental
54eee1e947
Reapply "[mlir][math] Add FP software implementation lowering pass: math-to-apfloat" (#172714) (#172716)
Reapply https://github.com/llvm/llvm-project/pull/171221 - Fix builder
by linking `MLIRTransformUtils`. Also move headers to
`mlir/Conversion/ArithAndMathToAPFloat`.
2025-12-17 17:26:37 -08:00
Jianhui Li
2b9e47749c
[MLIR][XeGPU] Refactor Layout access interface (#172125)
This PR builds on the anchor layout mechanism introduced in
https://github.com/llvm/llvm-project/pull/169267 and performs the
following refactoring:

1. Introduce getAnchorLayout() and setAnchorLayout() interface for
anchor ops to get and set layout attributes.

2. Add getLocalLayout() and setLocalLayout() utility functions, and
refactor workgroup/subgroup distribution patterns to use these APIs.
These utilities access the layout information directly and locally,
without relying on global propagation.

3. Introduce localPropagateLayoutsFromAnchor(), a utility used by
subgroup distribution to unify non-anchor layout setup.
This function is intended to be invoked upfront by all layout-based
passes (including workgroup/subgroup distribution and unrolling) to
propagate layouts from anchor ops to non-anchor ops.
After this step, patterns within the pass should exclusively use
getLocalLayout() / setLocalLayout().

4. Refactor getDistributeLayoutAttr() and setDistributeLayoutAttr() to
remove special-case handling. These APIs now operate in a uniform order:
anchor ops first, then non-anchor ops, and finally block arguments.
These APIs will be deprecated on long run. 

5. Refactor patterns in wg/sg distribution, load optimization passes to
use get/setAnchorLayout() and get/setLocalLayout().

6. Update test cases to enforce that anchor ops must use—and only
use—anchor layouts.
2025-12-17 12:04:58 -08:00
Maksim Levental
621fe03eaa
Revert "[mlir][math] Add FP software implementation lowering pass: math-to-apfloat" (#172714)
Reverts llvm/llvm-project#171221

Broken builder https://lab.llvm.org/buildbot/#/builders/138/builds/23270
2025-12-17 10:52:43 -08:00
Maksim Levental
7f1a30ebd2
[mlir][math] Add FP software implementation lowering pass: math-to-apfloat (#171221)
Add APFloat software implementation for `math.fma`, `math.abs`,
`math.isnan`, `math.isfinite`, `math.isinf`, `math.isnormal` for reduced
precision (`fp4*`, `fp6*`, `fp8*`).
2025-12-17 18:37:13 +00:00
Maksim Levental
8d5ade8feb
[mlir] enable APFloatWrappers on MacOS (#172070) 2025-12-12 11:34:23 -08:00
Sang Ik Lee
b8ddbc4f03
[MLIR][XeVM] gpu.printf test: use correct runtime. (#170754)
gpu printf test was not using the runtime required by lit.local.cfg
All other tests in the directory are correctly using level zero runtime.
But gpu printf test is using sycl runtime.
2025-12-08 08:14:56 -08:00
Matthias Springer
8378a6fa4f
[mlir][arith] Fix build after #171024 (#171057)
Fix build after #171024.
2025-12-07 21:48:00 +01:00
Matthias Springer
5dbd049662
[mlir][arith] arith-to-apfloat: Add vector support (#171024)
Add support for vectorized operations such as `arith.addf ... :
vector<4xf4E2M1FN>`. The computation is scalarized: scalar operands are
extracted with `vector.to_elements`, multiple scalar computations are
performed and the result is inserted back into a vector with
`vector.from_elements`.
2025-12-07 20:55:48 +01:00
Mehdi Amini
d02471ae5e [MLIR] Partially disable test/Integration/GPU/CUDA/async.mlir
This test is flaky, needs investigation.

See #170833
2025-12-05 03:54:04 -08:00
Sohaib Iftikhar
b31a398bcf
[MLIR][NVVM] Fix wmma test after d3edc94d (#170659)
See discussion on #169061
2025-12-04 13:37:29 +00:00
Sang Ik Lee
c379f7cc01
[MLIR][XeGPU] Add integration with XeGPU load / store ops to / from memref subview. (#170385)
Add XeGPU integration test for missing usage case: base memory from
memref subview.
2025-12-03 09:22:18 -08:00
Jakub Kuderski
ad656d3a19
[mlir][linalg][arm] Fix use of fill in arm integration tests (#170143)
Follow up to
https://github.com/llvm/llvm-project/pull/169567#issuecomment-3596220014
2025-12-01 14:19:07 +00:00
Ryan Holt
b27301ff5d
[mlir][linalg] Re-enable linalg runtime verification test (#170129)
Test seems to pass after re-enabling without any additional changes.
2025-12-01 08:52:20 -05:00
Giacomo Castiglioni
d3edc94d11
[MLIR][GPU] subgroup_mma fp64 extension - take 2 (#169061)
This PR re-lands #165873.

This PR extends the gpu.subgroup_mma_* ops to support fp64 type.
The extension requires special handling during the lowering to nvvm due
to the return type for load ops for fragment a and b (they return a
scalar instead of a struct).

The original PR did not guard the new test based on the required
architecture (sm80) which lead to a failure on the cuda runners with T4
GPUs.
2025-12-01 07:39:59 -05:00
Matthias Springer
147c466bcd
[mlir][arith] Add support for min/max to ArithToAPFloat (#169760)
Add support for `arith.minnumf`, `arith.maxnumf`, `arith.minimumf`,
`arith.maximumf`.
2025-12-01 08:50:02 +00:00
Matthias Springer
05b1989551
[mlir][arith] Add support for negf to ArithToAPFloat (#169759)
Add support for `arith.negf`.
2025-12-01 08:28:23 +00:00
Matthias Springer
4d7abe5355
[mlir][arith] Add support for cmpf to ArithToAPFloat (#169753)
Add support for `arith.cmpf`.
2025-12-01 09:12:11 +01:00
Jakub Kuderski
0bd2f12753
[mlir][linalg] Restrict fill initial value type to output element type (#169567)
Disallow implicit casting, which is surprising, and, IME, usually
indicative of copy-paste errors.

Because the initial value must be a scalar, I don't expect this to
affect any data movement.
2025-11-30 09:51:37 -05:00
Matthias Springer
6ec686735c
[mlir][arith] Add support for sitofp, uitofp to ArithToAPFloat (#169284)
Add support for `arith.sitofp` and `arith.uitofp`.
2025-11-25 11:31:23 +09:00
Matthias Springer
3db8ed0500
[mlir][arith] Add support for fptosi, fptoui to ArithToAPFloat (#169277)
Add support for `arith.fptosi` and `arith.fptoui`.
2025-11-25 10:50:20 +09:00
Matthias Springer
78994706d8
[mlir][arith] Add support for extf, truncf to ArithToAPFloat (#169275)
Add support for `arith.extf` and `arith.truncf`. No support for custom
rounding modes yet.
2025-11-25 10:09:26 +09:00
Fabian Mora
8c3f59f1b2
Revert "[MLIR][GPU] subgroup_mma fp64 extension" (#169049)
Reverts llvm/llvm-project#165873

The revert is triggered by a failing integration test on a couple of
buildbots.
2025-11-21 10:02:59 -05:00
Giacomo Castiglioni
49995b2af0
[MLIR][GPU] subgroup_mma fp64 extension (#165873)
This PR extends the `gpu.subgroup_mma_*` ops to support fp64 type.
The extension requires special handling during the lowering to `nvvm`
due to the return type for load ops for fragment a and b (they return a
scalar instead of a struct).
2025-11-21 09:07:43 -05:00
Matthias Springer
951ab04d6c
[mlir][NVVM] Add no-rollback option to NVVM lowering passes (#168477)
Add pass options to run lowerings to NVVM without pattern rollback. This
makes the dialect conversions easier to debug and improves
performance/memory usage.
2025-11-18 13:47:28 +08:00
Matthias Springer
7a53d33e7c
[mlir] Add FP software implementation lowering pass: arith-to-apfloat (#167848)
Reland pass and fix linker errors.

---------

Co-authored-by: Maksim Levental <maksim.levental@gmail.com>
2025-11-13 18:35:30 +09:00
Maksim Levental
140e07c862
Revert "Reland yet again: [mlir] Add FP software implementation lowering pass: arith-to-apfloat" (#167834)
Reverts llvm/llvm-project#167608

Broken builder https://lab.llvm.org/buildbot/#/builders/52/builds/12781
2025-11-12 23:02:21 -08:00
Matthias Springer
73e70e0c88
[mlir][linalg] Fix Linalg runtime verification test (#167814)
This integration test has been broken for a while. This commit partially
fixes it.

- Use `CHECK` + `CHECK-NEXT` to ensure that the correct error lines are
matched together.
- Move all `CHECK-NOT` to the end. Having a `CHECK` with the same string
does not make sense after a `CHECK-NOT`.
- Add a missing `CHECK: ERROR` for one of the test cases.
- Deactivate `reverse_from_3`, which is broken, and put a TODO.
2025-11-13 12:40:43 +09:00
Maksim Levental
0bba1e7658
Reland yet again: [mlir] Add FP software implementation lowering pass: arith-to-apfloat (#167608)
Fix both symbol visibility issue in the mlir_apfloat_wrappers lib and the linkage issue in ArithToAPFloat.
2025-11-12 17:57:53 -08:00
Hanumanth
81964597f9
[mlir][tensor] Fix runtime verification for tensor.extract_slice for empty tensor slices (#166569)
I hit another runtime verification issue (similar to
https://github.com/llvm/llvm-project/pull/164878) while working with
TFLite models. The verifier is incorrectly rejecting
`tensor.extract_slice` operations when extracting an empty slice
(size=0) that starts exactly at the tensor boundary.

The current runtime verification unconditionally enforces `offset <
dim_size`. This makes sense for non-empty slices, but it's too strict
for empty slices, causing false positives that lead to spurious runtime
assertions.

**Simple example that demonstrates the issue:**

```mlir
func.func @extract_empty_slice(%tensor: tensor<?xf32>, %offset: index, %size: index) {
  // When called with: tensor size=10, offset=10, size=0
  // Runtime verification fails: "offset 0 is out-of-bounds"
  %slice = tensor.extract_slice %tensor[%offset] [%size] [1] 
    : tensor<?xf32> to tensor<?xf32>
  return
}
```

For the above example, the check evaluates `10 < 10` which is false, so
verification fails. However, I believe this operation should be valid -
we're extracting zero elements, so there's no actual out-of-bounds
access.

**Real-world repro from the TensorFlow Lite models:**

This issue manifests while lowering TFLite models and a lot of our
system tests are failing due to this. Here's a simplified version
showing the problematic pattern:

In this code, `%extracted_slice_0` becomes an empty tensor when SSA
value `%15` reaches 10 (on the final loop iteration), making `%16 = 0`.
The operation extracts zero elements along dimension 0, which is
semantically valid but fails runtime verification.

```mlir
func.func @simplified_repro_from_tensorflowlite_model(%arg0: tensor<10x4x1xf32>) -> tensor<10x4x1xf32> {
  %c0 = arith.constant 0 : index
  %c1 = arith.constant 1 : index
  %c2 = arith.constant 2 : index
  %c10 = arith.constant 10 : index
  %c-1 = arith.constant -1 : index
  
  %0 = "tosa.const"() <{values = dense<0> : tensor<i32>}> : () -> tensor<i32>
  %1 = "tosa.const"() <{values = dense<1> : tensor<i32>}> : () -> tensor<i32>
  %2 = "tosa.const"() <{values = dense<10> : tensor<i32>}> : () -> tensor<i32>
  %3 = "tosa.const"() <{values = dense<-1> : tensor<2xi32>}> : () -> tensor<2xi32>
  %4 = "tosa.const"() <{values = dense<0> : tensor<2xi32>}> : () -> tensor<2xi32>
  %5 = "tosa.const"() <{values = dense<0.000000e+00> : tensor<1x4x1xf32>}> : () -> tensor<1x4x1xf32>
  %c4_1 = tosa.const_shape  {values = dense<1> : tensor<1xindex>} : () -> !tosa.shape<1>
  
  %6:2 = scf.while (%arg1 = %0, %arg2 = %arg0) 
    : (tensor<i32>, tensor<10x4x1xf32>) -> (tensor<i32>, tensor<10x4x1xf32>) {
    %7 = tosa.greater %2, %arg1 : (tensor<i32>, tensor<i32>) -> tensor<i1>
    %extracted = tensor.extract %7[] : tensor<i1>
    scf.condition(%extracted) %arg1, %arg2 : tensor<i32>, tensor<10x4x1xf32>
  } do {
  ^bb0(%arg1: tensor<i32>, %arg2: tensor<10x4x1xf32>):
    %7 = tosa.add %arg1, %1 : (tensor<i32>, tensor<i32>) -> tensor<i32>
    
    // First slice
    %8 = tosa.reshape %arg1, %c4_1 : (tensor<i32>, !tosa.shape<1>) -> tensor<1xi32>
    %9 = tosa.concat %8, %3 {axis = 0 : i32} : (tensor<1xi32>, tensor<2xi32>) -> tensor<3xi32>
    
    %extracted_0 = tensor.extract %9[%c0] : tensor<3xi32>
    %10 = index.casts %extracted_0 : i32 to index
    %11 = arith.cmpi eq, %10, %c-1 : index
    %12 = arith.select %11, %c10, %10 : index
    
    %extracted_slice = tensor.extract_slice %arg2[0, 0, 0] [%12, 4, 1] [1, 1, 1] 
      : tensor<10x4x1xf32> to tensor<?x4x1xf32>
    
    // Second slice - this is where the failure occurs
    %13 = tosa.reshape %7, %c4_1 : (tensor<i32>, !tosa.shape<1>) -> tensor<1xi32>
    %14 = tosa.concat %13, %4 {axis = 0 : i32} : (tensor<1xi32>, tensor<2xi32>) -> tensor<3xi32>
    
    %extracted_1 = tensor.extract %14[%c0] : tensor<3xi32>
    %15 = index.castu %extracted_1 : i32 to index
    %16 = arith.subi %c10, %15 : index  // size = 10 - offset
    
    %extracted_2 = tensor.extract %14[%c1] : tensor<3xi32>
    %17 = index.castu %extracted_2 : i32 to index
    
    %extracted_3 = tensor.extract %14[%c2] : tensor<3xi32>
    %18 = index.castu %extracted_3 : i32 to index
    
    // On the last loop iteration: %15=10, %16=0
    // %extracted_slice_0 becomes an empty tensor
    // Runtime verification fails: "offset 0 is out-of-bounds"
    %extracted_slice_0 = tensor.extract_slice %arg2[%15, %17, %18] [%16, 4, 1] [1, 1, 1] 
      : tensor<10x4x1xf32> to tensor<?x4x1xf32>
    
    %19 = tosa.concat %extracted_slice, %5, %extracted_slice_0 {axis = 0 : i32} 
      : (tensor<?x4x1xf32>, tensor<1x4x1xf32>, tensor<?x4x1xf32>) -> tensor<10x4x1xf32>
    
    scf.yield %7, %19 : tensor<i32>, tensor<10x4x1xf32>
  }
  
  return %6#1 : tensor<10x4x1xf32>
}
```
**The fix:**

Make the offset check conditional on slice size:
- Empty slice (size == 0): allow `0 <= offset <= dim_size`
- Non-empty slice (size > 0): require `0 <= offset < dim_size`


**Question for reviewers:**
Should we also relax the static verifier to allow this edge case?
Currently, the static verifier rejects the following IR:

```mlir
%tensor = arith.constant dense<1.0> : tensor<10xf32>
%slice = tensor.extract_slice %tensor[10] [0] [1] : tensor<10xf32> to tensor<0xf32>
```
Since we're allowing it at runtime for dynamic shapes, it seems
inconsistent to reject it statically. However, I wanted to get feedback
before making that change - this PR focuses only on the runtime
verification fix for dynamic shapes.

P.S. We have a similar issue with `memref.subview`. I will send a
separate patch for the issue.

Co-authored-by: Hanumanth Hanumantharayappa <hhanuman@ah-hhanuman-l.dhcp.mathworks.com>
2025-11-12 08:37:15 +09:00
Hanumanth
a664f584f9
[mlir][memref] Fix runtime verification for memref.subview for empty memref subviews (#166581)
This PR applies the same fix from #166569 to `memref.subview`. That PR
fixed the issue for `tensor.extract_slice`, and this one addresses the
identical problem for `memref.subview`.

The runtime verification for `memref.subview` incorrectly rejects valid
empty subviews (size=0) starting at the memref boundary.

**Example that demonstrates the issue:**

```mlir
func.func @subview_with_empty_slice(%memref: memref<10x4x1xf32, strided<[?, ?, ?], offset: ?>>, 
                                     %dim_0: index, 
                                     %dim_1: index, 
                                     %dim_2: index,
                                     %offset: index) {
    // When called with: offset=10, dim_0=0, dim_1=4, dim_2=1
    // Runtime verification fails: "offset 0 is out-of-bounds"
    %subview = memref.subview %memref[%offset, 0, 0] [%dim_0, %dim_1, %dim_2] [1, 1, 1] :
        memref<10x4x1xf32, strided<[?, ?, ?], offset: ?>> to
        memref<?x?x?xf32, strided<[?, ?, ?], offset: ?>>
    return
}
```

When `%offset=10` and `%dim_0=0`, we're creating an empty subview (zero
elements along dimension 0) starting at the boundary. The current
verification enforces `offset < dim_size`, which evaluates to `10 < 10`
and fails. I feel this should be valid since no memory is accessed.

**The fix:**

Same as #166569 - make the offset check conditional on subview size:
- Empty subview (size == 0): allow `0 <= offset <= dim_size`
- Non-empty subview (size > 0): require `0 <= offset < dim_size`

Please see #166569 for motivation and rationale.

---

Co-authored-by: Hanumanth Hanumantharayappa <hhanuman@ah-hhanuman-l.dhcp.mathworks.com>
2025-11-12 08:36:41 +09:00
Maksim Levental
e67ac07881
Revert "Reapply "Reapply "[mlir] Add FP software implementation lowering pass: arith-to-apfloat (#166618)" (#167431)"" (#167549)
Reverts llvm/llvm-project#167436 to fix sanitizers
2025-11-11 09:29:09 -08:00
Maksim Levental
75751f33a8
Reapply "Reapply "[mlir] Add FP software implementation lowering pass: arith-to-apfloat (#166618)" (#167431)" (#167436)
Reland https://github.com/llvm/llvm-project/pull/166618 by fixing
missing symbol issues by explicitly loading
`--shared-libs=%mlir_apfloat_wrappers` as well as
`--shared-libs=%mlir_c_runner_utils`.
2025-11-11 08:50:37 -08:00
Maksim Levental
93b3c1b7e1
Revert "Reapply "[mlir] Add FP software implementation lowering pass: arith-to-apfloat (#166618)" (#167431)" (#167435)
This reverts commit 0e639ae6e030ade849fa7a09cb7dc40b42f25373.
2025-11-10 17:40:51 -08:00
Maksim Levental
0e639ae6e0
Reapply "[mlir] Add FP software implementation lowering pass: arith-to-apfloat (#166618)" (#167431)
Reland https://github.com/llvm/llvm-project/pull/166618 with
MLIRFuncUtils linked in.
2025-11-10 17:13:29 -08:00
Maksim Levental
70d379fd7a
Revert "[mlir] Add FP software implementation lowering pass: arith-to-apfloat (#166618)" (#167429)
This reverts commit 222f4e494a0cd9515c242fd083c2776772734385.
2025-11-10 16:54:44 -08:00
Maksim Levental
222f4e494a
[mlir] Add FP software implementation lowering pass: arith-to-apfloat (#166618)
This commit adds a new pass that lowers floating-point `arith`
operations to calls into the execution engine runtime library. Currently
supported operations: `addf`, `subf`, `mulf`, `divf`, `remf`.

All floating-point types that have an APFloat semantics are supported.
This includes low-precision floating-point types such as `f4E2M1FN` that
cannot execute natively on CPUs.

This commit also improves the `vector.print` lowering pattern to call
into the runtime library for floating-point types that are not supported
by LLVM. This is necessary to write a meaningful integration test.

The way it works is 

```mlir
func.func @full_example() {
  %a = arith.constant 1.4 : f8E4M3FN
  %b = func.call @foo() : () -> (f8E4M3FN)
  %c = arith.addf %a, %b : f8E4M3FN
  vector.print %c : f8E4M3FN
  return
}
```

gets transformed to

```mlir
func.func private @__mlir_apfloat_add(i32, i64, i64) -> i6
func.func @full_example() {
  %cst = arith.constant 1.375000e+00 : f8E4M3FN
  %0 = call @foo() : () -> f8E4M3FN
  // bitcast operand A to integer of equal width
  %1 = arith.bitcast %cst : f8E4M3FN to i8
  // zext A to i64
  %2 = arith.extui %1 : i8 to i64
  // same for operand B
  %3 = arith.bitcast %0 : f8E4M3FN to i8
  %4 = arith.extui %3 : i8 to i64
  // get the llvm::fltSemantics(f8E4M3FN) as an enum
  %c10_i32 = arith.constant 10 : i32
  // call the impl against APFloat in mlir_apfloat_wrappers
  %5 = call @__mlir_apfloat_add(%c10_i32, %2, %4) : (i32, i64, i64) -> i64
  // "cast" back to the original fp type
  %6 = arith.trunci %5 : i64 to i8
  %7 = arith.bitcast %6 : i8 to f8E4M3FN
  vector.print %7 : f8E4M3FN
}
```

Note, `llvm::fltSemantics(f8E4M3FN)` is emitted by the pattern each time
an `arith` op is transformed, thereby making the call to
`__mlir_apfloat_add` correct (i.e., no name mangling on type necessary).


RFC:
https://discourse.llvm.org/t/rfc-software-implementation-for-unsupported-fp-types-in-convert-arith-to-llvm/88785

---------

Co-authored-by: Matthias Springer <me@m-sp.org>
2025-11-10 16:21:39 -08:00
Aleksei Nurmukhametov
d30bd27e7f
[mlir][complex] Fix exp accuracy (#164952)
This ports openxla/stablehlo#2682 implementation by @pearu.

Three tests were added to
`Integration/Dialect/Complex/CPU/correctness.mlir`. I also verified
accuracy using XLA's complex_unary_op_test and its MLIR emitters.
2025-10-28 15:01:15 +01:00
Hanumanth
cbe7c49e93
[mlir][memref] Fix runtime verification for memref.subview when size dimension value is 0 (#164897)
Previously, the runtime verification pass would insert assertion
statements with conditions that always evaluate to false for
semantically valid `memref.subview` operations where one of the
dimensions had a size of 0.

The `memref.subview` runtime verification logic was unconditionally
generating checks for the position of the last element (`offset + (size
- 1) * stride`). When `size` is 0, this causes the assertion condition
to always be false, leading to runtime failures even though the
operation is semantically valid.

This patch fixes the issue by making the `lastPos` check conditional.
The offset is always verified, but the endpoint check is only performed
when `size > 0` to avoid generating spurious assert statements.

This issue was discovered through a LiteRT model, where a dynamic shape
calculation resulted in a zero-sized dimension being passed to
`memref.subview`. The following is a simplified IR snippet from the
model. After running the runtime verification pass, an assertion that
always fails is generated because the SSA value `%5` becomes 0.

```mlir
module {
  memref.global "private" constant @__constant_2xi32 : memref<2xi32> = dense<-1> {alignment = 64 : i64}
  memref.global "private" constant @__constant_1xi32 : memref<1xi32> = dense<0> {alignment = 64 : i64}
  func.func @simpleRepro(%arg0: memref<10x4x1xf32, strided<[?, ?, ?], offset: ?>>) -> memref<?x?x?xf32, strided<[?, ?, ?], offset: ?>> {
    %c2 = arith.constant 2 : index
    %c4 = arith.constant 4 : index
    %c1 = arith.constant 1 : index
    %c10 = arith.constant 10 : index
    %c0 = arith.constant 0 : index
    %c-1 = arith.constant -1 : index
    %0 = memref.get_global @__constant_1xi32 : memref<1xi32>
    %1 = memref.get_global @__constant_2xi32 : memref<2xi32>
    %alloca = memref.alloca() {alignment = 64 : i64} : memref<3xi32>
    %subview = memref.subview %alloca[0] [1] [1] : memref<3xi32> to memref<1xi32, strided<[1]>>
    memref.copy %0, %subview : memref<1xi32> to memref<1xi32, strided<[1]>>
    %subview_0 = memref.subview %alloca[1] [2] [1] : memref<3xi32> to memref<2xi32, strided<[1], offset: 1>>
    memref.copy %1, %subview_0 : memref<2xi32> to memref<2xi32, strided<[1], offset: 1>>
    %2 = memref.load %alloca[%c0] : memref<3xi32>
    %3 = index.casts %2 : i32 to index
    %4 = arith.cmpi eq, %3, %c-1 : index
    %5 = arith.select %4, %c10, %3 : index
    %6 = memref.load %alloca[%c1] : memref<3xi32>
    %7 = index.casts %6 : i32 to index
    %8 = arith.cmpi eq, %7, %c-1 : index
    %9 = arith.select %8, %c4, %7 : index
    %10 = memref.load %alloca[%c2] : memref<3xi32>
    %11 = index.casts %10 : i32 to index
    %12 = arith.cmpi eq, %11, %c-1 : index
    %13 = arith.select %12, %c1, %11 : index
    %subview_1 = memref.subview %arg0[0, 0, 0] [%5, %9, %13] [1, 1, 1] : memref<10x4x1xf32, strided<[?, ?, ?], offset: ?>> to memref<?x?x?xf32, strided<[?, ?, ?], offset: ?>>
    return %subview_1 : memref<?x?x?xf32, strided<[?, ?, ?], offset: ?>>
  }
}
```

P.S. This is a similar issue to the one fixed for `tensor.extract_slice`
in https://github.com/llvm/llvm-project/pull/164878

---------

Co-authored-by: Hanumanth Hanumantharayappa <hhanuman@ah-hhanuman-l.dhcp.mathworks.com>
2025-10-27 11:43:45 -07:00
Hanumanth
a6788b5246
[mlir][tensor] Fix runtime verification for tensor.extract_slice when size dimension value is 0 (#164878)
Previously, the runtime verification pass would insert assertion
statements with conditions that always evaluate to false for
semantically valid `tensor.extract_slice` operations where one of the
dimensions had a size of 0.

The `tensor.extract_slice` runtime verification logic was
unconditionally generating checks for the position of the last element
(`offset + (size - 1) * stride`). When `size` is 0, this causes the
assertion condition to always be false, leading to runtime failures even
though the operation is semantically valid.

This patch fixes the issue by making the `lastPos` check conditional.
The offset is always verified, but the endpoint check is only performed
when `size > 0` to avoid generating spurious assert statements.

This issue was discovered through LiteRT model, where a dynamic shape
calculation resulted in a zero-sized dimension being passed to
`tensor.extract_slice`.

The following is a simplified IR snippet from the model. After running
the runtime verification pass, an assertion that always fails is
generated because the SSA value `%3` becomes 0.

```mlir
func.func @simple_repro_from_liteRT_model(%arg0: tensor<10x4x1xf32>) -> tensor<?x?x?xf32> {
  %cst = arith.constant dense<0> : tensor<1xi32>
  %cst_0 = arith.constant dense<-1> : tensor<2xi32>
  %c-1 = arith.constant -1 : index
  %c0 = arith.constant 0 : index
  %c10 = arith.constant 10 : index
  %c1 = arith.constant 1 : index
  %c4 = arith.constant 4 : index
  %c2 = arith.constant 2 : index
  %0 = tensor.empty() : tensor<3xi32>
  %inserted_slice = tensor.insert_slice %cst into %0[0] [1] [1] : tensor<1xi32> into tensor<3xi32>
  %inserted_slice_1 = tensor.insert_slice %cst_0 into %inserted_slice[1] [2] [1] : tensor<2xi32> into tensor<3xi32>
  %extracted = tensor.extract %inserted_slice_1[%c0] : tensor<3xi32>
  %1 = index.casts %extracted : i32 to index
  %2 = arith.cmpi eq, %1, %c-1 : index
  %3 = arith.select %2, %c10, %1 : index
  %extracted_2 = tensor.extract %inserted_slice_1[%c1] : tensor<3xi32>
  %4 = index.casts %extracted_2 : i32 to index
  %5 = arith.cmpi eq, %4, %c-1 : index
  %6 = arith.select %5, %c4, %4 : index
  %extracted_3 = tensor.extract %inserted_slice_1[%c2] : tensor<3xi32>
  %7 = index.casts %extracted_3 : i32 to index
  %8 = arith.cmpi eq, %7, %c-1 : index
  %9 = arith.select %8, %c1, %7 : index
  %extracted_slice = tensor.extract_slice %arg0[0, 0, 0] [%3, %6, %9] [1, 1, 1] : tensor<10x4x1xf32> to tensor<?x?x?xf32>
  return %extracted_slice : tensor<?x?x?xf32>
}
```

The issue can be reproduced more simply with the following test case,
where `dim_0` is `0`. When the runtime verification pass is applied to
this code with `dim_0 = 0`, it generates an assertion that will always
fail at runtime.

```mlir
func.func @extract_slice_zero_size_dim(%arg0: tensor<10x4x1xf32>,
                                      %dim_0: index,
                                      %dim_1: index,
                                      %dim_2: index) {
  %slice = tensor.extract_slice %arg0[0, 0, 0] [%dim_0, %dim_1, %dim_2] [1, 1, 1]
    : tensor<10x4x1xf32> to tensor<?x?x?xf32>
  return
}

func.func @test_zero_size_extraction() {
  %input = arith.constant dense<1.0> : tensor<10x4x1xf32>
  // Define slice dimensions: 0x4x1 (zero-size in first dimension)
  %dim_0 = arith.constant 0 : index
  %dim_1 = arith.constant 4 : index
  %dim_2 = arith.constant 1 : index
  func.call @extract_slice_zero_size_dim(%input, %dim_0, %dim_1, %dim_2)
    : (tensor<10x4x1xf32>, index, index, index) -> ()
  return
}
```

P.S. We probably have a similar issue with `memref.subview`. I will
check this and send a separate PR for the issue.

---------

Co-authored-by: Hanumanth Hanumantharayappa <hhanuman@ah-hhanuman-l.dhcp.mathworks.com>
2025-10-27 11:43:18 -07:00
Sang Ik Lee
150145486e
[MLIR][GPU] Generalize gpu.printf op lowering to LLVM call pattern. (#164297)
Existing pattern for lowering gpu.printf op to LLVM call uses fixed
function name and calling convention.
Those two should be exposed as pass option to allow supporting Intel
Compute Runtime for GPU.

Also adds gpu.printf op pattern to GPU to LLVMSPV pass.
It may appear out of place, but integration test is added to XeVM
integration test as that is the current best folder for testing with
Intel Compute Runtime.
Test should be moved in the future if a better test folder is added.
2025-10-23 08:32:53 -07:00
Hanumanth
d08cbc1cdd
[mlir][linalg] Fix Linalg runtime verification pass to handle tensors with dimensions of size 0 (#163791)
Runtime verification on Linalg structured ops unconditionally computed
`end - 1` to determine the last iteration index before composing
indexing maps. This caused spurious "negative index" assertion failures
while operating on empty tensors (tensors with a dimension of size 0).

The issue occurs because: 

1. Empty tensors create loop ranges [0, 0) with zero trip count 

2. Computing end - 1 = 0 - 1 = -1 creates a fictitious negative index 

3. The negative index check triggers even though no loop iterations
occur

The fix is to guard all runtime verification with a check that ensures
all loop ranges are non-empty (start < end) before performing any index
arithmetic.

Example MLIR that previously failed:
```mlir
func.func @fill_empty() -> tensor<0xi32> {
  %c0 = arith.constant 0 : i32
  %empty = tensor.empty() : tensor<0xi32>
  %filled = linalg.fill ins(%c0 : i32) outs(%empty : tensor<0xi32>) -> tensor<0xi32>
  return %filled : tensor<0xi32>
}
```

---------

Co-authored-by: Hanumanth Hanumantharayappa <hhanuman@ah-hhanuman-l.dhcp.mathworks.com>
2025-10-22 15:19:40 +02:00
Andrzej Warzyński
33f812dbae
[mlir][sve] Extend the e2e test for linalg.pack + linalg.unpack (#158094)
Adds vectorization of `linalg.unpack` to:
* test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-scalable-inner-tile.mlir

The original configuration that didn't use vectorization is preserved as
reference and also to make sure we exercise all relevant code paths
(*). To this end, the TD sequence has be refactored out into a dedicated
file. The payload IR has not changed.

NOTE - ATM only vectorization of `linalg.unpack` is supported.
Vectorizing `linalg.pack` is left as a TODO.

(*) It remains valid to use scalable tile sizes in Linalg Ops even when
vectorization is not enabled.
2025-10-21 12:26:41 +01:00
Md Abdullah Shahneous Bari
89d1143a90
[mlir][gpu]Add GPUToXeVM lowering pipeline pass. (#161216)
It's the default GPU to XeVM lowering pipeline. It starts by lowering
GPU code to the specified compilation target (default is fatbin), then
lowers the host code.
If XeGPU ops are used, it expects the MLIR code to have XeGPU ops
already embedded in gpu code.
2025-10-17 14:41:20 -05:00
Erick Ochoa Lopez
55d4d2ee0d
[mlir][spirv] Fix test (NFC) (#163413)
This test had a CHECK-RAW command. The intention behind this command
appears to be to avoid using the regular expression matching
capabilities. However, this was interpretted as a comment by FileCheck.
In order to check for literal strings the {LITERAL} modifier should be
used.
https://llvm.org/docs/CommandGuide/FileCheck.html#directive-modifiers
2025-10-14 12:09:27 -04:00
James Newling
01c7ef6556
[MLIR][Vector] Fix test following vector.splat removal (#162892)
This is a fix for the failing integration test (see
https://lab.llvm.org/buildbot/#/builders/177/builds/22398) reported in
https://github.com/llvm/llvm-project/pull/162167.
2025-10-10 10:23:27 -07:00
James Newling
ea291d0e8c
[MLIR][Vector] Remove vector.splat (#162167)
vector.splat has been deprecated (user: please use the very similar vector.broadcast instead) 
with the last PR landing about 6 weeks ago.

The discourse discussion is at
https://discourse.llvm.org/t/rfc-mlir-vector-deprecate-then-remove-vector-splat/87143/1
The last PR was #152230

This PR completely removes vector.splat. In addition to removing vector.splat from VectorOps.td, it

- Updates the few remaining places where vector::SplatOp is created (now vector::BroadcastOp is created)
- Removes temporary patterns where vector.splat is replaced by vector.broadcast

The only place 'vector.splat' appears is now the files

https://github.com/llvm/llvm-project/blob/main/mlir/utils/tree-sitter-mlir/test/corpus/op.txt
 and

https://github.com/llvm/llvm-project/blob/main/mlir/utils/tree-sitter-mlir/dialect/vector.js

---------

Signed-off-by: James Newling <james.newling@gmail.com>
2025-10-10 09:58:18 -07:00
Hanchenng Wu
4e6ee0b674
[MLIR] Fix test failures for generate-runtime-verification pass from PR #160331 (#162533)
[MLIR] Fix test failures for generate-runtime-verification pass from PR #160331
    
PR #160331 introduced a mistake that removed the error message for
generate-runtime-verification
pass, leading to test failures during
`test-build-check-mlir-build-only-check-mlir`.
    
This patch restores the missing error message.
    
In addition, for related tests, the op strings used in FileChecks are
updated with the same op
formats as used in input mlirs.
    
Verified locally.
    
Fixes post-merge regression from:
https://github.com/llvm/llvm-project/pull/160331
2025-10-08 23:31:55 +01:00
Andrzej Warzyński
3233c94789
[mlir][test] Fix SVE e2e test (#160269)
The original RUN line assumed that we are running on a host with SVE.
That's not necessarily the case. Failing bot:
* https://lab.llvm.org/buildbot/#/builders/121/builds/1661
2025-09-23 12:25:55 +01:00
Durgadoss R
fa366b4e9f
[MLIR][NVVM] Update TMA Load Op (#156347)
This patch includes im2col and gather mode
support for the TMA Load Op. The lowering is
also updated to intrinsics except when a Predicate
is given. This completes the Blackwell additions
on this Op.

* NVVM Dialect has support for Shared::Cluster
   address-space now. So, this patch also updates the
   Op to use AS(7) instead of AS(3). The corresponding
   inline-ptx based unit tests are also updated.
*  lit tests are added for all combinations.

Signed-off-by: Durgadoss R <durgadossr@nvidia.com>
2025-09-23 13:03:35 +05:30
Jorn Tuyls
faf5f28fc2
[mlir][arith][transforms] Fix f4E2M1FN to f32 cast (#160121)
The signed i4 bitcast was used when setting the exponent and mantissa
and instead the sign should be omitted in the comparisons.

Without this, for example the following incorrect conversion from `-0.5`
f4 to `-3.0` f32 will happen:

| Binary | F4E2M1 | f32[23:32] | f32
| 1001   | -0.5    | ~~1 1000 000 01~~ | ~~-3.0~~

**Walkthrough:**
Bits 23 and 24 are set based on:
```
Value isHalf =
        arith::CmpIOp::create(b, arith::CmpIPredicate::eq, i4BitsNoSign, c0x1);
```
Because `1001 (i4) != 1`, bit 23 and 24 are set to the leading two bits
of `1001 << 2`, which is `01`. The correct bits are `00`.

Bits 25 through 31 are set based on the i4 value being greater or equal
to 4:
```
Value useLargerExp =
        arith::CmpIOp::create(b, arith::CmpIPredicate::uge, i4BitsNoSign, c0x4);
```
As `1001` is a negative i4 value, this is false and those bits are
incorrectly set to `1000 000` instead of `0111 111`.
2025-09-22 12:15:26 -04:00