7c07cb6542a0c5e4340e09a9a247e3e5123c6567 introduced a variable created
in an if statement that is only used in an assertion. Per the coding
guidelines, mark it [[maybe_unused]].
This PR continues work from
https://github.com/llvm/llvm-project/pull/178290
Added local helper functions to avoid dependency on LLVM APIs.
---------
Co-authored-by: Jakub Kuderski <kubakuderski@gmail.com>
The OpenACC remark emission utilities previously only accepted Twine for
message construction. However, complex remarks often require additional
logic to build messages, such as resolving variable names. This results
in unnecessary work when remarks are disabled.
Add an overload that accepts a lambda for message generation, which is
only invoked when remark emission is enabled. Update ACCLoopTiling to
use this lazy API for tile size reporting.
Additionally, getVariableName now returns numeric strings for constant
integer values. This is also being used by ACCLoopTiling along with the
lazy remark update.
- fixing incorrect assertion and related function name
- MPI_comm_split is not pure
- simplifying/standardizing permutation in all_gather
---------
Co-authored-by: Rolf Morel <rolfmorel@gmail.com>
Currently, UniformQuantizedType only supports built-in MLIR storage
types such as Integer. LLM quantization research introducing feature of
using NF4 as a low precision datatype (see
https://arxiv.org/pdf/2305.14314). There is a growing need to make the
system extensible and maintainable as more types are added. Ensuring
that MLIR can natively support NF4 through a clean, extensible interface
is essential for both current and future quantization workflows.
**Current Approach and Its Limitations:**
- The present implementation relies on dynamic checks (e.g., type
switches or if-else chains) to determine the storage type and retrieve
type-specific information for legality checks.
- This approach works for a small, fixed set of types, but as the number
of supported types grows, the code becomes harder to read, maintain, and
extend.
**Proposed Interface-Based Approach:**
- Define a StorageTypeInterface that specifies the required methods any
storage type must implement to be used in UniformQuantizedType.
- Each storage type (Integer, Float8E5M2, Float8E4M3FN, and new types
like NF4) would implement this interface, encapsulating their
type-specific logic.
- When UniformQuantizedType needs to check legality or retrieve
information, it can use MLIR’s dyn_cast mechanism to check if the type
implements the interface and then call the required methods.
- This design decouples UniformQuantizedType from the specifics of each
storage type, making it easy to add new types (such as NF4) without
modifying the core logic or introducing more type checks.
**Benefits:**
- Extensibility: New storage types can be added by simply implementing
the interface, without touching the core UniformQuantizedType logic.
- Readability: The code is cleaner, as it avoids large switch statements
or if-else chains.
- Maintainability: Type-specific logic is encapsulated within each type,
reducing the risk of errors and making the codebase easier to understand
and update.
The verifiers of these attributes are supposed to verify additional
constraints which usually require the invariants, nested ops to be
verified first. Move it to the end of verification so that we don't
operate on malformed operations.
This PR adds validation in the `gpu.launch` parser to ensure the launch
configuration provides exactly 3 arguments. Emit a parser error when the
argument count is not 3. Fixes#176426.
Extend operands when computing ub - lb to avoid overflow in signed
arithmetic. E.g., i8: ub=127, lb=-128 yields 255, which overflows
without extension.
Whoops, turns out I was off by 1 on how many bits are in the counts and
phases ind these new LDS barriers. This commit fixes this.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The PR modifies the subgroup distribution pass to only sink
insert_strided_slice operation if it becomes the last op before yield.
It avoids sinking insert_strided_slice multiple times and cause
potential issue in worst case.
This is a follow-up PR of #169045 and the second part of #179086.
In #179086, we added support for defining regions in Python-defined ops,
but its usefulness was quite limited because we still couldn’t mark an
op as a `Terminator` or `NoTerminator`. In this PR, we port the
`DynamicOpTrait` (introduced on the C++ side for `DynamicDialect` in
#177735) to Python, so we can dynamically attach traits to
Python-defined ops.
### whats the problem
mlir-opt could crash while verifying amdgpu.dpp when its operands had
vector
types, such as ARM SME tile vectors produced by arm_sme.get_tile.
The crash occurred during IR verification, before any lowering or passes
ran.
### why it happens
DPPOp::verify() called Type::getIntOrFloatBitWidth() on the operand
type.
When the operand was a VectorType, this hit an assertion because only
scalar
integer and float types have a bitwidth.
### whats the fix
Query the bitwidth on the element type using getElementTypeOrSelf()
instead of
the container type.
Add a regression test to ensure amdgpu.dpp verification no longer
asserts on
vector operand types.
Fixes#178128
-- vector.outerproduct requires lhs/rhs to have same element type as the
result.
-- This commit adds a fix to promote lhs/rhs to have result's element
type when vectorizing conv1D slice to vector.outerproduct.
-- This is along the similar lines of what happens when we are
vectorizing conv1D slice to vector.contract - the corresponding
CHECK line was incorrect and this commit fixes that too.
Signed-off-by: Abhishek Varma <abhvarma@amd.com>
This PR calls recoverTemporaryLayout before the XeGPUWgtoSgDistribute &
XeGPUBlocking Pass to recover all the temporary operand layout which
might be required by the transformation patterns for checks and
verification
Fix two issues brough by PR179016:
1. unused variable if build the option with
"DLLVM_ENABLE_ASSERTIONS=OFF"
2. Recover modification to recoverTemporaryLayouts() brought by
PR176737. Unintentionally lost during the merging process.
This commit introduces the `!amdgpu.ds_barrier_state` type and
operations on that type, including extracting its components and (more
importantly) provides wrappers around the upcoming barrier-management
instructions that will be added in gfx1250.
This commit is loosely based on work done for Triton, but this commit
provides slightly more lower level-primitives (namely a known-atomic
load for getting the barrier state instead of providing a `wait`
operation that includes an entire spin-loop, though if people want one
we could consider adding it.) These operations will allow LDS barriers
to be interacted with in a more type-safe manner.
The types and operations use the Ds naming scheme to match the
underlying instructions and to avoid confusion with the "LDS barrier"
already present in the AMDGPU dialect that was a workaround for LLVM's
memory fencing support.
(To summarize a potential usage pattern, one can use a pair of these
barriers to communicate between wave(s) in a workgroup that load data
into memory and a separate wave(s) that compute with that data.)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This patch puts together a lot more of the CIR infrastructure for
function attributes, plus adds a bunch of 'TODO' messages for areas that
have been skipped.
Along the way, we also implement 8 attributes in some way: -Convergent
gets a little more work, to make the `noconvergent` C attribute have an
effect
-optsize/minsize are implemented, sourced from the command line
-nobuiltin is a call-only attribute that tells not to replace the
individual call with a builtin. This is a touch confusing, since
no-builtins is an attribute that means "don't replace anything in the
body of this function with builtins (from this list)". The spelling
confusion is existing, and it seems that changing the names away from
LLVM would be confusing.
-save_reg_params & zero_call_used_regs are boht pretty simple registers
-temp-func-name just passes a string to LLVM, consistent with existing
implementation.
-default-func-attrs is a difficult one. It takes command line arguments
and passes them as LLVM-IR attributes directly on functions/calls. In
the dialect, we are capturing these in their own attribute to pass them
on correctly. However, this is one we cannot recover from LLVM-IR for
obvious reasons, so we instead choose to let the 'passthrough' mechanism
work for those.
This PR refactors layout propagation into two distinct components:
result/anchor layout setup and source layout inference from the result.
For operations that require a specific result layout due to semantic or
hardware constraints, the propagation logic explicitly sets up the
result or anchor layout. Otherwise, it infers the source layout from the
backward-propagated consumer layout.
The result or anchor layout may differ from the backward-propagated
consumer layout; any such discrepancies are resolved via the existing
layout-conflict mechanism.
**This PR introduces the following utility functions:**
Source layout inference:
> inferBroadcastSourceLayout()
> inferMultiReductionSourceLayout()
> inferBitCastSourceLayout()
> inferShapeCastSourceLayout()
> inferInsertStridedSliceSourceLayout()
Result / anchor layout setup:
> setupMultiReductionResultLayout()
> setupBitCastResultLayout()
> setupInsertStridedSliceResultLayout()
> setupLoadMatrixAnchorLayout()
> setupStoreMatrixAnchorLayout()
> setupLoadGatherAnchorLayout()
> setupStoreScatterAnchorLayout()
Part of subgroup distribution related code changes are separated and
created as PR https://github.com/llvm/llvm-project/pull/179018/changes.
Remove xevm to llvm conversion pass from convert to llvm as it is a
backend dependent conversion.
And add legalization pattern for splitting large vector load that are
eventually split into smaller
vectors by shufflevector. shufflevector can be replaced with a smaller
load in such case.
This PR adds a C API `MlirLlvmRawFdOstream` for `llvm::raw_fd_ostream`,
which cannot be safely replaced by `std::ofstream` on Windows.
`llvm::raw_fd_ostream` configures Win32 file sharing flags, allowing
other handles (e.g. Python temp file handles) to coexist, see details
[here](https://llvm.org/doxygen/Windows_2Path_8inc_source.html#l1281),
while `std::ofstream` disables file sharing by default.
Type conversion rule for single element vector and materialization
function to support the conversion has a mismatch.
Update materialization function to match the type conversion rule.
Add support for lowering vector.transfer_read to
gpu.subgroup_mma_load_matrix with transpose permutation_map
with non-minor dimensions e.g. (d0, d1, d2) -> (d2, d0)
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
Following work completed in #174386 and #174623, this patch adds support
for collapse to Taskloop. Collapse allows for the user to compress
multiple loop nests into a single loop, and for this to work with
Taskloop, there needs to be some changes to how we process the loops,
and the tasks that run them.
This patch brings Taskloop equivalent to OpenMP 4.5 support for MLIR and
Flang.
Updating memref.cast check regarding if input and output are valid for
casting.
Currently in case of casting between dynamic and static dims with
different strides, the return value of the check is not symmetric and
depends if casting for dynamic to static or vice versa. Updating the
check logic to make this symmetric.
This PR continues work from #178290
It cleans up multiple LLVM utilities in *.h files under
`mlir/Bindings/python`, along with the corresponding *.cpp files.
`shard.allgather` concatenates along a specified gather-axis. However,
`mpi.allgather` always concatenates along the first dimension and there
is no MPI operation that allows gathering along an arbitrary axis.
Hence, if gather-axis!=0, we need to create a temporary buffer where we
gather along the first dimension and then copy from that buffer to the
final output along the specified gather-axis. This is not ideal by far.
Along the way also
- fixing computation of memref size in mpitollvm
- adding a simple canonicalization pattern for comm_size for easier
debugging
- adding more tests
Upgrade the barrier eliminiation pass to account for the address spaces
of accessed memory when deciding which barriers to eliminiate. In
particular, a loop that only reads and writes global memory that has a
workgoup-memory-fencing barrier inside of it will now have that barrier
marked for elimiination, as the global memory traffic is not being
synchronized by the barrier.
The pass is also adjusted to ignore barriers whose memory fencing list
is [], as those do not synchronize memory and therefore the logic in
this pass would potentially incorrectly remove them after proving that
fact.
---------
Co-authored-by: Jakub Kuderski <kubakuderski@gmail.com>
Update the `WrapFuncInClassPass` pass so that, by default, the generated
method is named `operator()()` rather than `execute()`. This makes the
pass more generic, instead of catering to specific users expecting an
`execute()` method.
To preserve the original behaviour, add a new pass option to override
the method name: `func-name`. For example:
```bash
mlir-opt file.mlir -wrap-emitc-func-in-class=func-name=execute
```
Additionally, make a couple of small editorial changes:
* Rename `populateFuncPatterns` to `populateWrapFuncInClass` to make it
clear that the corresponding pattern is specific to the
`WrapFuncInClass` pass.
* Remove `// CHECK: module {` to reduce test noise.
For context, this change was proposed on Discourse:
* https://discourse.llvm.org/t/rfc-emitc-support-for-mlgo
This commit improves `resolveBroadcastShape` when some input dims are
dynamic. Previously, this would result in shape inference failing,
meaning the output type of the operation would not be updated. Now the
output shape is correctly inferred, selecting the static dim if > 1,
otherwise selecting the dynamic dim.
This patch adds support for `PredTypeTrait` and `PredAttrTrait` in type
and attribute definitions, enabling declarative predicate-based
verification similar to how `PredOpTrait` works for operations.
## Motivation
In 802bf02 (from 2021), `PredTypeTrait`/`PredAttrTrait` were defined in
TableGen but not implemented in the code generator. Using them causes
mlir-tblgen to crash with an assertion failure when trying to cast
`PredTrait` to `InterfaceTrait`. This patch fixes the crash and
implements the actual verification code generation.
## Usage
Use `$paramName` syntax in predicates to reference type/attribute
parameters:
```tablegen
def MyType : MyDialect_Type<"MyType",
[PredTypeTrait<"value must be positive", CPred<"$value > 0">>]> {
let parameters = (ins "unsigned":$value);
let mnemonic = "my_type";
let assemblyFormat = "`<` $value `>`";
}
```
This generates verification code in `verifyInvariantsImpl()`:
```cpp
if (!(value > 0)) {
emitError() << "failed to verify that value must be positive";
return ::mlir::failure();
}
```
The alloc_size attribute takes the argument number(normalized to the
index!) of the element size and count, for things like 'malloc' or
'calloc'.
This ends up being slightly more complicated than others, as this has
data that we have to decide on a format for. LLVM chooses to pack both
of these 32 bit values into a single i64, but unpacks it for the purpose
of input/output. The second value, the number of elements, is optional.
This patch uses a DenseI32ArrayAttr to store them for the LLVMIR
dialect, which gets us the packed nature, but doesn't require us doing
any work to unpack it.
Update symbol resolution to examine whether a nested symbol being
resolved is private, and fail in that case. This ensures that we
maintain invariants on symbol visibility that we depend on in
optimisations.
Consolidate the handling of nested symbols in the loop over nested
references. This is an NFC refactor to make it simpler to enforce symbol
visibility here.
