Since #180397, all elements of a `DenseIntOrFPElementsAttr` are padded
to full bytes. This enables additional simplifications: whether a
`DenseIntOrFPElementsAttr` is a splat or not can now be inferred from
the size of the buffer. This was not possible before because a single
byte sometimes contained multiple `i1` elements.
Discussion:
https://discourse.llvm.org/t/denseelementsattr-i1-element-type/62525
`DenseElementsAttr` stores elements in a `ArrayRef<char>` buffer, where
each element is padded to a full byte. Before this commit, there used to
be a special storage format for `i1` elements: they used to be densely
packed, i.e., 1 bit per element. This commit removes the dense packing
special case for `i1`.
This commit removes complexity from `DenseElementsAttr`. If dense
packing is needed in the future it could be implemented in a general way
that works for all element types (based on #179122).
Discussion:
https://discourse.llvm.org/t/denseelementsattr-i1-element-type/62525
For example:
```
error: 'tosa.dim' op illegal: requires [bf16, shape] but not included in the profile compliance [shape]
%0 = tosa.dim %arg0 {axis = 4 : i32} : (tensor<4x5x8x8x6x4xbf16>) -> !tosa.shape<1>
```
Here dim requires support to be declared for the BF16 and SHAPE
extensions, but only SHAPE was specified in the op declaration.
Previously, `NVVM_PTXBuilder_Op` included `BasicPtxBuilderOpInterface`
as part of the default value of the `traits` parameter. This meant any
subclass that provided an explicit traits list would silently replace
the default and lose the interface, defeating the purpose of the base
class. Callers had to redundantly re-specify the interface.
This patch moves the definitions of memory effects for the data
entry/exit operations into C++ code. The main reason for this
is to modify the effects of [first]private and reduction
operations: they should not access `CurrentDeviceIdResource`
when they are located inside a compute construct.
The ODS to C++ migration was done with AI assistance. I reviewed
these changes and made sure it was an NFC change. After that
I modified [first]private and reduction implementations.
Thiese commits add three more populate methods for
`vector.multi_reduction`'s lowering patterns:
* populateVectorMultiReductionTransformationPatterns
* populateVectorMultiReductionFlatteningPatterns
* populateVectorMultiReductionUnrollingPatterns
These methods have a
finer level of granularity and allow users to select between unrolling,
flattening, and applying transformations that would set up operations
for unrolling and flattening.
The previous populateVectorMultiReductionLoweringPatterns method
is rewritten in terms of these new methods.
This PR continues work from
https://github.com/llvm/llvm-project/pull/178290
Added local helper functions to avoid dependency on LLVM APIs.
---------
Co-authored-by: Jakub Kuderski <kubakuderski@gmail.com>
The OpenACC remark emission utilities previously only accepted Twine for
message construction. However, complex remarks often require additional
logic to build messages, such as resolving variable names. This results
in unnecessary work when remarks are disabled.
Add an overload that accepts a lambda for message generation, which is
only invoked when remark emission is enabled. Update ACCLoopTiling to
use this lazy API for tile size reporting.
Additionally, getVariableName now returns numeric strings for constant
integer values. This is also being used by ACCLoopTiling along with the
lazy remark update.
Previous commit message:
>Previous commit message:
>
>> Original commit message:
>>
>>>When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>>>
>>>When no SM or PTX version is specified, we now use PTX 3.2 (the
minimum for the default SM 3.0) instead of PTX 6.0.
>>
>>The following commits should fix the failures that arose when I
previously tried to land this commit:
>>
>>
>>9fc5fd0ad6
should address the llvm-nvptx*-nvidia-* build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
>>
>>
>>600514a637
should address the MLIR failures
>
>The previous commit was reverted with
d23cb79ba4
because the
[mlir-nvidia](https://lab.llvm.org/buildbot/#/builders/138/builds/24797)
and
[mlir-nvidia-gcc7](https://lab.llvm.org/buildbot/#/builders/116/builds/23929)
Buildbots were failing.
>
>Those tests failed because MLIR's default SM was 5.0, which caused
NVPTX
to target PTX ISA v4.0, which did not support the intrinsics used in the
failing tests.
>
>243f011577
should address this by bumping MLIR's default SM to 7.5. Now, using
MLIR's new default SM, NVPTX
targets the PTX ISA v6.3, which supports the intrinsics used in the
failing tests.
---
The previous commit was reverted with
e9b578a4d77025e18318efedd0f3f3764338d859
[because](https://github.com/llvm/llvm-project/pull/179304#issuecomment-3856301333)
the clang driver set the default PTX ISA version to v4.2 when no CUDA
installation is found. However, given our patch, we should not set a
default; instead, let the LLVM backend select the appropriate PTX ISA
version for the target SM.
Currently we use `DenseMap` in `DynamicOpTraitList` to store traits and
iterate over the `DenseMap`, and we found that the order is not fixed
and we also cannot control verification order.
In this PR we use `MapVector` to preserve the insertion order so that
the verification order over traits can be fixed and users can tune the
verification order.
- fixing incorrect assertion and related function name
- MPI_comm_split is not pure
- simplifying/standardizing permutation in all_gather
---------
Co-authored-by: Rolf Morel <rolfmorel@gmail.com>
Currently, UniformQuantizedType only supports built-in MLIR storage
types such as Integer. LLM quantization research introducing feature of
using NF4 as a low precision datatype (see
https://arxiv.org/pdf/2305.14314). There is a growing need to make the
system extensible and maintainable as more types are added. Ensuring
that MLIR can natively support NF4 through a clean, extensible interface
is essential for both current and future quantization workflows.
**Current Approach and Its Limitations:**
- The present implementation relies on dynamic checks (e.g., type
switches or if-else chains) to determine the storage type and retrieve
type-specific information for legality checks.
- This approach works for a small, fixed set of types, but as the number
of supported types grows, the code becomes harder to read, maintain, and
extend.
**Proposed Interface-Based Approach:**
- Define a StorageTypeInterface that specifies the required methods any
storage type must implement to be used in UniformQuantizedType.
- Each storage type (Integer, Float8E5M2, Float8E4M3FN, and new types
like NF4) would implement this interface, encapsulating their
type-specific logic.
- When UniformQuantizedType needs to check legality or retrieve
information, it can use MLIR’s dyn_cast mechanism to check if the type
implements the interface and then call the required methods.
- This design decouples UniformQuantizedType from the specifics of each
storage type, making it easy to add new types (such as NF4) without
modifying the core logic or introducing more type checks.
**Benefits:**
- Extensibility: New storage types can be added by simply implementing
the interface, without touching the core UniformQuantizedType logic.
- Readability: The code is cleaner, as it avoids large switch statements
or if-else chains.
- Maintainability: Type-specific logic is encapsulated within each type,
reducing the risk of errors and making the codebase easier to understand
and update.
Extend operands when computing ub - lb to avoid overflow in signed
arithmetic. E.g., i8: ub=127, lb=-128 yields 255, which overflows
without extension.
Now that we have an AMDGPU dialect type for the in-LDS barriers that the
tensor data mover can automatically visit, update the definition of the
tensor descriptor operations to use said types and document the behavior
of the barrier.
Whoops, turns out I was off by 1 on how many bits are in the counts and
phases ind these new LDS barriers. This commit fixes this.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This is a follow-up PR of #169045 and the second part of #179086.
In #179086, we added support for defining regions in Python-defined ops,
but its usefulness was quite limited because we still couldn’t mark an
op as a `Terminator` or `NoTerminator`. In this PR, we port the
`DynamicOpTrait` (introduced on the C++ side for `DynamicDialect` in
#177735) to Python, so we can dynamically attach traits to
Python-defined ops.
### whats the problem
mlir-opt could crash while verifying amdgpu.dpp when its operands had
vector
types, such as ARM SME tile vectors produced by arm_sme.get_tile.
The crash occurred during IR verification, before any lowering or passes
ran.
### why it happens
DPPOp::verify() called Type::getIntOrFloatBitWidth() on the operand
type.
When the operand was a VectorType, this hit an assertion because only
scalar
integer and float types have a bitwidth.
### whats the fix
Query the bitwidth on the element type using getElementTypeOrSelf()
instead of
the container type.
Add a regression test to ensure amdgpu.dpp verification no longer
asserts on
vector operand types.
Fixes#178128
This commit introduces the `!amdgpu.ds_barrier_state` type and
operations on that type, including extracting its components and (more
importantly) provides wrappers around the upcoming barrier-management
instructions that will be added in gfx1250.
This commit is loosely based on work done for Triton, but this commit
provides slightly more lower level-primitives (namely a known-atomic
load for getting the barrier state instead of providing a `wait`
operation that includes an entire spin-loop, though if people want one
we could consider adding it.) These operations will allow LDS barriers
to be interacted with in a more type-safe manner.
The types and operations use the Ds naming scheme to match the
underlying instructions and to avoid confusion with the "LDS barrier"
already present in the AMDGPU dialect that was a workaround for LLVM's
memory fencing support.
(To summarize a potential usage pattern, one can use a pair of these
barriers to communicate between wave(s) in a workgroup that load data
into memory and a separate wave(s) that compute with that data.)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
It appears that these all became `Optional` thanks to a series of
copy/paste errors going back years. CallOp doesn't have this 'feature',
and a "UnitAttr' should be enough as it is effectively already a
true/false value as it is.
So this patch removes the optional here to simplify the implementation,
and changes no tests, as the interface is identical anyway.
This patch puts together a lot more of the CIR infrastructure for
function attributes, plus adds a bunch of 'TODO' messages for areas that
have been skipped.
Along the way, we also implement 8 attributes in some way: -Convergent
gets a little more work, to make the `noconvergent` C attribute have an
effect
-optsize/minsize are implemented, sourced from the command line
-nobuiltin is a call-only attribute that tells not to replace the
individual call with a builtin. This is a touch confusing, since
no-builtins is an attribute that means "don't replace anything in the
body of this function with builtins (from this list)". The spelling
confusion is existing, and it seems that changing the names away from
LLVM would be confusing.
-save_reg_params & zero_call_used_regs are boht pretty simple registers
-temp-func-name just passes a string to LLVM, consistent with existing
implementation.
-default-func-attrs is a difficult one. It takes command line arguments
and passes them as LLVM-IR attributes directly on functions/calls. In
the dialect, we are capturing these in their own attribute to pass them
on correctly. However, this is one we cannot recover from LLVM-IR for
obvious reasons, so we instead choose to let the 'passthrough' mechanism
work for those.
This PR refactors layout propagation into two distinct components:
result/anchor layout setup and source layout inference from the result.
For operations that require a specific result layout due to semantic or
hardware constraints, the propagation logic explicitly sets up the
result or anchor layout. Otherwise, it infers the source layout from the
backward-propagated consumer layout.
The result or anchor layout may differ from the backward-propagated
consumer layout; any such discrepancies are resolved via the existing
layout-conflict mechanism.
**This PR introduces the following utility functions:**
Source layout inference:
> inferBroadcastSourceLayout()
> inferMultiReductionSourceLayout()
> inferBitCastSourceLayout()
> inferShapeCastSourceLayout()
> inferInsertStridedSliceSourceLayout()
Result / anchor layout setup:
> setupMultiReductionResultLayout()
> setupBitCastResultLayout()
> setupInsertStridedSliceResultLayout()
> setupLoadMatrixAnchorLayout()
> setupStoreMatrixAnchorLayout()
> setupLoadGatherAnchorLayout()
> setupStoreScatterAnchorLayout()
Part of subgroup distribution related code changes are separated and
created as PR https://github.com/llvm/llvm-project/pull/179018/changes.
Collapsing expand(collapse(src)) to cast(src) is supported in cases
where the source and result are cast compatible but not equal. When the
source has dynamic dimensions this leads to cases where the cast is
enabled even though certain dimensions cast from static to dynamic when
the dynamic size is not assured to be equal to the static size.
Currently blocking applying this folding when the source has dynamic
dimensions to preserve correctness.
In the future it could be possible to enable some cases of folding when
not all dimensions of the source are static.
Such cases could be when:
1) expand and collapse happened on non dynamic dims
2) expand and collapse on dynamic dims could be folded to no op
This PR adds a C API `MlirLlvmRawFdOstream` for `llvm::raw_fd_ostream`,
which cannot be safely replaced by `std::ofstream` on Windows.
`llvm::raw_fd_ostream` configures Win32 file sharing flags, allowing
other handles (e.g. Python temp file handles) to coexist, see details
[here](https://llvm.org/doxygen/Windows_2Path_8inc_source.html#l1281),
while `std::ofstream` disables file sharing by default.
Change Remark's StringRef members to std::string to ensure remarks own
their data, preventing dangling pointers when used with
RemarkEmittingPolicyFinal.
Previous commit message:
> Original commit message:
>
>>When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>>
>>When no SM or PTX version is specified, we now use PTX 3.2 (the
minimum for the default SM 3.0) instead of PTX 6.0.
>
>The following commits should fix the failures that arose when I
previously tried to land this commit:
>
>9fc5fd0ad6
should address the llvm-nvptx*-nvidia-* build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
>
>600514a637
should address the MLIR failures
---
The previous commit was reverted with
d23cb79ba497281de050ef609cb91b91058bf323 because the
[mlir-nvidia](https://lab.llvm.org/buildbot/#/builders/138/builds/24797)
and
[mlir-nvidia-gcc7](https://lab.llvm.org/buildbot/#/builders/116/builds/23929)
Buildbots were failing.
Those tests failed because MLIR's default SM was 5.0, which caused NVPTX
to target PTX ISA v4.0, which did not support the intrinsics used in the
failing tests.
243f011577193c99358ccc4142b296d4fa80ea11 should address this by bumping
MLIR's default SM to 7.5. Now, using MLIR's new default SM, NVPTX
targets the PTX ISA v6.3, which supports the intrinsics used in the
failing tests.
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
This PR continues work from #178290
It cleans up multiple LLVM utilities in *.h files under
`mlir/Bindings/python`, along with the corresponding *.cpp files.
`shard.allgather` concatenates along a specified gather-axis. However,
`mpi.allgather` always concatenates along the first dimension and there
is no MPI operation that allows gathering along an arbitrary axis.
Hence, if gather-axis!=0, we need to create a temporary buffer where we
gather along the first dimension and then copy from that buffer to the
final output along the specified gather-axis. This is not ideal by far.
Along the way also
- fixing computation of memref size in mpitollvm
- adding a simple canonicalization pattern for comm_size for easier
debugging
- adding more tests
Update the `WrapFuncInClassPass` pass so that, by default, the generated
method is named `operator()()` rather than `execute()`. This makes the
pass more generic, instead of catering to specific users expecting an
`execute()` method.
To preserve the original behaviour, add a new pass option to override
the method name: `func-name`. For example:
```bash
mlir-opt file.mlir -wrap-emitc-func-in-class=func-name=execute
```
Additionally, make a couple of small editorial changes:
* Rename `populateFuncPatterns` to `populateWrapFuncInClass` to make it
clear that the corresponding pattern is specific to the
`WrapFuncInClass` pass.
* Remove `// CHECK: module {` to reduce test noise.
For context, this change was proposed on Discourse:
* https://discourse.llvm.org/t/rfc-emitc-support-for-mlgo
The `BooleanConditionOrMatchingShape` trait was assuming that if the
condition was not i1, both condition and result must be `ShapedTypes`.
It would then call `AllShapesMatch` which performs a blind cast to
`ShapedType`, causing a crash when one of the operands was a scalar.
This PATCH fixes the problem.
Closes [#178230](https://github.com/llvm/llvm-project/issues/178230)
The alloc_size attribute takes the argument number(normalized to the
index!) of the element size and count, for things like 'malloc' or
'calloc'.
This ends up being slightly more complicated than others, as this has
data that we have to decide on a format for. LLVM chooses to pack both
of these 32 bit values into a single i64, but unpacks it for the purpose
of input/output. The second value, the number of elements, is optional.
This patch uses a DenseI32ArrayAttr to store them for the LLVMIR
dialect, which gets us the packed nature, but doesn't require us doing
any work to unpack it.
