Fixes#151786
The original `ceilf` expansion lowers to `fptosi`, which produces poison
for Inf, and any subsequent use leads to undefined behavior. This patch
adds a safe path, similar to the existing `round` expansion, for large
or special inputs and avoids the UB.
Since #180397, all elements of a `DenseIntOrFPElementsAttr` are padded
to full bytes. This enables additional simplifications: whether a
`DenseIntOrFPElementsAttr` is a splat or not can now be inferred from
the size of the buffer. This was not possible before because a single
byte sometimes contained multiple `i1` elements.
Discussion:
https://discourse.llvm.org/t/denseelementsattr-i1-element-type/62525
This PR allows the expand op converter to consider the NoNaN fastmath
attribute to disable the runtime checks for NaNs in E8M0 types. Default
behaviour is still the same.
The OCP document provides all-ones as NaN for E8M0, but for pre-MX I8
quantization, the checks for NaNs are prohibitively expensive,
especially if the hardware doesn't have native support for that type.
Remove restriction in affine analysis utility for checking slice
validity. This was unnecessarily bailing out still after the underlying
methods were extended. This update enables fusion of affine nests with
symbolic bounds.
Fixes: https://github.com/llvm/llvm-project/issues/61784
Based on and revived from https://reviews.llvm.org/D148559 from
@anoopjs.
This patch moves the definitions of memory effects for the data
entry/exit operations into C++ code. The main reason for this
is to modify the effects of [first]private and reduction
operations: they should not access `CurrentDeviceIdResource`
when they are located inside a compute construct.
The ODS to C++ migration was done with AI assistance. I reviewed
these changes and made sure it was an NFC change. After that
I modified [first]private and reduction implementations.
Thiese commits add three more populate methods for
`vector.multi_reduction`'s lowering patterns:
* populateVectorMultiReductionTransformationPatterns
* populateVectorMultiReductionFlatteningPatterns
* populateVectorMultiReductionUnrollingPatterns
These methods have a
finer level of granularity and allow users to select between unrolling,
flattening, and applying transformations that would set up operations
for unrolling and flattening.
The previous populateVectorMultiReductionLoweringPatterns method
is rewritten in terms of these new methods.
The OpenACC remark emission utilities previously only accepted Twine for
message construction. However, complex remarks often require additional
logic to build messages, such as resolving variable names. This results
in unnecessary work when remarks are disabled.
Add an overload that accepts a lambda for message generation, which is
only invoked when remark emission is enabled. Update ACCLoopTiling to
use this lazy API for tile size reporting.
Additionally, getVariableName now returns numeric strings for constant
integer values. This is also being used by ACCLoopTiling along with the
lazy remark update.
- fixing incorrect assertion and related function name
- MPI_comm_split is not pure
- simplifying/standardizing permutation in all_gather
---------
Co-authored-by: Rolf Morel <rolfmorel@gmail.com>
Currently, UniformQuantizedType only supports built-in MLIR storage
types such as Integer. LLM quantization research introducing feature of
using NF4 as a low precision datatype (see
https://arxiv.org/pdf/2305.14314). There is a growing need to make the
system extensible and maintainable as more types are added. Ensuring
that MLIR can natively support NF4 through a clean, extensible interface
is essential for both current and future quantization workflows.
**Current Approach and Its Limitations:**
- The present implementation relies on dynamic checks (e.g., type
switches or if-else chains) to determine the storage type and retrieve
type-specific information for legality checks.
- This approach works for a small, fixed set of types, but as the number
of supported types grows, the code becomes harder to read, maintain, and
extend.
**Proposed Interface-Based Approach:**
- Define a StorageTypeInterface that specifies the required methods any
storage type must implement to be used in UniformQuantizedType.
- Each storage type (Integer, Float8E5M2, Float8E4M3FN, and new types
like NF4) would implement this interface, encapsulating their
type-specific logic.
- When UniformQuantizedType needs to check legality or retrieve
information, it can use MLIR’s dyn_cast mechanism to check if the type
implements the interface and then call the required methods.
- This design decouples UniformQuantizedType from the specifics of each
storage type, making it easy to add new types (such as NF4) without
modifying the core logic or introducing more type checks.
**Benefits:**
- Extensibility: New storage types can be added by simply implementing
the interface, without touching the core UniformQuantizedType logic.
- Readability: The code is cleaner, as it avoids large switch statements
or if-else chains.
- Maintainability: Type-specific logic is encapsulated within each type,
reducing the risk of errors and making the codebase easier to understand
and update.
The verifiers of these attributes are supposed to verify additional
constraints which usually require the invariants, nested ops to be
verified first. Move it to the end of verification so that we don't
operate on malformed operations.
This PR adds validation in the `gpu.launch` parser to ensure the launch
configuration provides exactly 3 arguments. Emit a parser error when the
argument count is not 3. Fixes#176426.
Extend operands when computing ub - lb to avoid overflow in signed
arithmetic. E.g., i8: ub=127, lb=-128 yields 255, which overflows
without extension.
The PR modifies the subgroup distribution pass to only sink
insert_strided_slice operation if it becomes the last op before yield.
It avoids sinking insert_strided_slice multiple times and cause
potential issue in worst case.
### whats the problem
mlir-opt could crash while verifying amdgpu.dpp when its operands had
vector
types, such as ARM SME tile vectors produced by arm_sme.get_tile.
The crash occurred during IR verification, before any lowering or passes
ran.
### why it happens
DPPOp::verify() called Type::getIntOrFloatBitWidth() on the operand
type.
When the operand was a VectorType, this hit an assertion because only
scalar
integer and float types have a bitwidth.
### whats the fix
Query the bitwidth on the element type using getElementTypeOrSelf()
instead of
the container type.
Add a regression test to ensure amdgpu.dpp verification no longer
asserts on
vector operand types.
Fixes#178128
-- vector.outerproduct requires lhs/rhs to have same element type as the
result.
-- This commit adds a fix to promote lhs/rhs to have result's element
type when vectorizing conv1D slice to vector.outerproduct.
-- This is along the similar lines of what happens when we are
vectorizing conv1D slice to vector.contract - the corresponding
CHECK line was incorrect and this commit fixes that too.
Signed-off-by: Abhishek Varma <abhvarma@amd.com>
This PR calls recoverTemporaryLayout before the XeGPUWgtoSgDistribute &
XeGPUBlocking Pass to recover all the temporary operand layout which
might be required by the transformation patterns for checks and
verification
Fix two issues brough by PR179016:
1. unused variable if build the option with
"DLLVM_ENABLE_ASSERTIONS=OFF"
2. Recover modification to recoverTemporaryLayouts() brought by
PR176737. Unintentionally lost during the merging process.
This commit introduces the `!amdgpu.ds_barrier_state` type and
operations on that type, including extracting its components and (more
importantly) provides wrappers around the upcoming barrier-management
instructions that will be added in gfx1250.
This commit is loosely based on work done for Triton, but this commit
provides slightly more lower level-primitives (namely a known-atomic
load for getting the barrier state instead of providing a `wait`
operation that includes an entire spin-loop, though if people want one
we could consider adding it.) These operations will allow LDS barriers
to be interacted with in a more type-safe manner.
The types and operations use the Ds naming scheme to match the
underlying instructions and to avoid confusion with the "LDS barrier"
already present in the AMDGPU dialect that was a workaround for LLVM's
memory fencing support.
(To summarize a potential usage pattern, one can use a pair of these
barriers to communicate between wave(s) in a workgroup that load data
into memory and a separate wave(s) that compute with that data.)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This patch puts together a lot more of the CIR infrastructure for
function attributes, plus adds a bunch of 'TODO' messages for areas that
have been skipped.
Along the way, we also implement 8 attributes in some way: -Convergent
gets a little more work, to make the `noconvergent` C attribute have an
effect
-optsize/minsize are implemented, sourced from the command line
-nobuiltin is a call-only attribute that tells not to replace the
individual call with a builtin. This is a touch confusing, since
no-builtins is an attribute that means "don't replace anything in the
body of this function with builtins (from this list)". The spelling
confusion is existing, and it seems that changing the names away from
LLVM would be confusing.
-save_reg_params & zero_call_used_regs are boht pretty simple registers
-temp-func-name just passes a string to LLVM, consistent with existing
implementation.
-default-func-attrs is a difficult one. It takes command line arguments
and passes them as LLVM-IR attributes directly on functions/calls. In
the dialect, we are capturing these in their own attribute to pass them
on correctly. However, this is one we cannot recover from LLVM-IR for
obvious reasons, so we instead choose to let the 'passthrough' mechanism
work for those.
This PR refactors layout propagation into two distinct components:
result/anchor layout setup and source layout inference from the result.
For operations that require a specific result layout due to semantic or
hardware constraints, the propagation logic explicitly sets up the
result or anchor layout. Otherwise, it infers the source layout from the
backward-propagated consumer layout.
The result or anchor layout may differ from the backward-propagated
consumer layout; any such discrepancies are resolved via the existing
layout-conflict mechanism.
**This PR introduces the following utility functions:**
Source layout inference:
> inferBroadcastSourceLayout()
> inferMultiReductionSourceLayout()
> inferBitCastSourceLayout()
> inferShapeCastSourceLayout()
> inferInsertStridedSliceSourceLayout()
Result / anchor layout setup:
> setupMultiReductionResultLayout()
> setupBitCastResultLayout()
> setupInsertStridedSliceResultLayout()
> setupLoadMatrixAnchorLayout()
> setupStoreMatrixAnchorLayout()
> setupLoadGatherAnchorLayout()
> setupStoreScatterAnchorLayout()
Part of subgroup distribution related code changes are separated and
created as PR https://github.com/llvm/llvm-project/pull/179018/changes.
Remove xevm to llvm conversion pass from convert to llvm as it is a
backend dependent conversion.
And add legalization pattern for splitting large vector load that are
eventually split into smaller
vectors by shufflevector. shufflevector can be replaced with a smaller
load in such case.
Updating memref.cast check regarding if input and output are valid for
casting.
Currently in case of casting between dynamic and static dims with
different strides, the return value of the check is not symmetric and
depends if casting for dynamic to static or vice versa. Updating the
check logic to make this symmetric.
`shard.allgather` concatenates along a specified gather-axis. However,
`mpi.allgather` always concatenates along the first dimension and there
is no MPI operation that allows gathering along an arbitrary axis.
Hence, if gather-axis!=0, we need to create a temporary buffer where we
gather along the first dimension and then copy from that buffer to the
final output along the specified gather-axis. This is not ideal by far.
Along the way also
- fixing computation of memref size in mpitollvm
- adding a simple canonicalization pattern for comm_size for easier
debugging
- adding more tests
Upgrade the barrier eliminiation pass to account for the address spaces
of accessed memory when deciding which barriers to eliminiate. In
particular, a loop that only reads and writes global memory that has a
workgoup-memory-fencing barrier inside of it will now have that barrier
marked for elimiination, as the global memory traffic is not being
synchronized by the barrier.
The pass is also adjusted to ignore barriers whose memory fencing list
is [], as those do not synchronize memory and therefore the logic in
this pass would potentially incorrectly remove them after proving that
fact.
---------
Co-authored-by: Jakub Kuderski <kubakuderski@gmail.com>
Update the `WrapFuncInClassPass` pass so that, by default, the generated
method is named `operator()()` rather than `execute()`. This makes the
pass more generic, instead of catering to specific users expecting an
`execute()` method.
To preserve the original behaviour, add a new pass option to override
the method name: `func-name`. For example:
```bash
mlir-opt file.mlir -wrap-emitc-func-in-class=func-name=execute
```
Additionally, make a couple of small editorial changes:
* Rename `populateFuncPatterns` to `populateWrapFuncInClass` to make it
clear that the corresponding pattern is specific to the
`WrapFuncInClass` pass.
* Remove `// CHECK: module {` to reduce test noise.
For context, this change was proposed on Discourse:
* https://discourse.llvm.org/t/rfc-emitc-support-for-mlgo
This commit improves `resolveBroadcastShape` when some input dims are
dynamic. Previously, this would result in shape inference failing,
meaning the output type of the operation would not be updated. Now the
output shape is correctly inferred, selecting the static dim if > 1,
otherwise selecting the dynamic dim.
The alloc_size attribute takes the argument number(normalized to the
index!) of the element size and count, for things like 'malloc' or
'calloc'.
This ends up being slightly more complicated than others, as this has
data that we have to decide on a format for. LLVM chooses to pack both
of these 32 bit values into a single i64, but unpacks it for the purpose
of input/output. The second value, the number of elements, is optional.
This patch uses a DenseI32ArrayAttr to store them for the LLVMIR
dialect, which gets us the packed nature, but doesn't require us doing
any work to unpack it.
This PR updates tensor.expand_shape and tensor.collapse_shape ODS
definitions to require ranked tensor operands/results by switching from
AnyTensor to AnyRankedTensor.
Fixes https://github.com/llvm/llvm-project/issues/178228
Change `getConstLoopTripCounts` to return `SmallVector<llvm::APInt>`
instead
of `SmallVector<int64_t>` to properly handle signedness and prevent
potential
overflow issues. Update all call sites to use APInt methods and uint64_t
for
intermediate calculations.
- Use APInt::isOne() instead of direct comparison with 1
- Store trip counts in uint64_t to avoid overflow in modulo operations
- Remove TODOs about signedness and overflow issues that are now fixed
Fixes#178506
There is no existing `vector.reduce` op in the vector dialect, but
multiple doc strings reference it. This change updates those instances
to the correct `vector.reduction` op.
