This op is the inverse of all-gather. It is useful to have an explicit
concise representation instead of having a blob of slicing logic.
Add lowering for the op that slices from the tensor based on the
in-group process index.
Make resharding generate an all-slice instead of inserting the slicing
logic directly.
This patch introduces the new MPI dialect into MLIR. The Message Passing
Interface (MPI) is a widely-used standard for distributed programs to
exchange data. This PR goes together with a talk later at today's LLVM
Dev Meeting.
This is just a first, small patch to get going and add the necessary
base files, so that we can add more operations in further patches.
Here's the documentation as generated by `ninja mlir-doc`:
# 'mpi' Dialect
This dialect models the Message Passing Interface (MPI), version
4.0. It is meant to serve as an interfacing dialect that is targeted
by higher-level dialects. The MPI dialect itself can be lowered to
multiple MPI implementations and hide differences in ABI. The dialect
models the functions of the MPI specification as close to 1:1 as
possible
while preserving SSA value semantics where it makes sense, and uses
`memref` types instead of bare pointers.
This dialect is under active development, and while stability is an
eventual goal, it is not guaranteed at this juncture. Given the early
state, it is recommended to inquire further prior to using this dialect.
For an in-depth documentation of the MPI library interface, please refer
to official documentation such as the
[OpenMPI online documentation](https://www.open-mpi.org/doc/current/).
[TOC]
## Operation definition
### `mpi.comm_rank` (mpi::CommRankOp)
_Get the current rank, equivalent to `MPI_Comm_rank(MPI_COMM_WORLD,
&rank)`_
Syntax:
```
operation ::= `mpi.comm_rank` attr-dict `:` type(results)
```
Communicators other than `MPI_COMM_WORLD` are not supported for now.
This operation can optionally return an `!mpi.retval` value that can be
used
to check for errors.
#### Results:
| Result | Description |
| :----: | ----------- |
| `retval` | MPI function call return value
| `rank` | 32-bit signless integer
### `mpi.error_class` (mpi::ErrorClassOp)
_Get the error class from an error code, equivalent to the
`MPI_Error_class` function_
Syntax:
```
operation ::= `mpi.error_class` $val attr-dict `:` type($val)
```
`MPI_Error_class` maps return values from MPI calls to a set of
well-known
MPI error classes.
#### Operands:
| Operand | Description |
| :-----: | ----------- |
| `val` | MPI function call return value
#### Results:
| Result | Description |
| :----: | ----------- |
| `errclass` | MPI function call return value
### `mpi.finalize` (mpi::FinalizeOp)
_Finalize the MPI library, equivalent to `MPI_Finalize()`_
Syntax:
```
operation ::= `mpi.finalize` attr-dict (`:` type($retval)^)?
```
This function cleans up the MPI state. Afterwards, no MPI methods may
be invoked (excpet for MPI_Get_version, MPI_Initialized, and
MPI_Finalized).
Notably, MPI_Init cannot be called again in the same program.
This operation can optionally return an `!mpi.retval` value that can be
used
to check for errors.
#### Results:
| Result | Description |
| :----: | ----------- |
| `retval` | MPI function call return value
### `mpi.init` (mpi::InitOp)
_Initialize the MPI library, equivalent to `MPI_Init(NULL, NULL)`_
Syntax:
```
operation ::= `mpi.init` attr-dict (`:` type($retval)^)?
```
This operation must preceed most MPI calls (except for very few
exceptions,
please consult with the MPI specification on these).
Passing &argc, &argv is not supported currently.
This operation can optionally return an `!mpi.retval` value that can be
used
to check for errors.
#### Results:
| Result | Description |
| :----: | ----------- |
| `retval` | MPI function call return value
### `mpi.recv` (mpi::RecvOp)
_Equivalent to `MPI_Recv(ptr, size, dtype, dest, tag, MPI_COMM_WORLD,
MPI_STATUS_IGNORE)`_
Syntax:
```
operation ::= `mpi.recv` `(` $ref `,` $tag `,` $rank `)` attr-dict `:` type($ref) `,` type($tag) `,` type($rank)(`->` type($retval)^)?
```
MPI_Recv performs a blocking receive of `size` elements of type `dtype`
from rank `dest`. The `tag` value and communicator enables the library
to
determine the matching of multiple sends and receives between the same
ranks.
Communicators other than `MPI_COMM_WORLD` are not supprted for now.
The MPI_Status is set to `MPI_STATUS_IGNORE`, as the status object
is not yet ported to MLIR.
This operation can optionally return an `!mpi.retval` value that can be
used
to check for errors.
#### Operands:
| Operand | Description |
| :-----: | ----------- |
| `ref` | memref of any type values
| `tag` | 32-bit signless integer
| `rank` | 32-bit signless integer
#### Results:
| Result | Description |
| :----: | ----------- |
| `retval` | MPI function call return value
### `mpi.retval_check` (mpi::RetvalCheckOp)
_Check an MPI return value against an error class_
Syntax:
```
operation ::= `mpi.retval_check` $val `=` $errclass attr-dict `:` type($res)
```
This operation compares MPI status codes to known error class
constants such as `MPI_SUCCESS`, or `MPI_ERR_COMM`.
#### Attributes:
<table>
<tr><th>Attribute</th><th>MLIR Type</th><th>Description</th></tr>
<tr><td><code>errclass</code></td><td>::mlir::mpi::MPI_ErrorClassEnumAttr</td><td><details><summary>MPI
error class name</summary>{{% markdown %}}Enum cases:
* MPI_SUCCESS (`MPI_SUCCESS`)
* MPI_ERR_ACCESS (`MPI_ERR_ACCESS`)
* MPI_ERR_AMODE (`MPI_ERR_AMODE`)
* MPI_ERR_ARG (`MPI_ERR_ARG`)
* MPI_ERR_ASSERT (`MPI_ERR_ASSERT`)
* MPI_ERR_BAD_FILE (`MPI_ERR_BAD_FILE`)
* MPI_ERR_BASE (`MPI_ERR_BASE`)
* MPI_ERR_BUFFER (`MPI_ERR_BUFFER`)
* MPI_ERR_COMM (`MPI_ERR_COMM`)
* MPI_ERR_CONVERSION (`MPI_ERR_CONVERSION`)
* MPI_ERR_COUNT (`MPI_ERR_COUNT`)
* MPI_ERR_DIMS (`MPI_ERR_DIMS`)
* MPI_ERR_DISP (`MPI_ERR_DISP`)
* MPI_ERR_DUP_DATAREP (`MPI_ERR_DUP_DATAREP`)
* MPI_ERR_ERRHANDLER (`MPI_ERR_ERRHANDLER`)
* MPI_ERR_FILE (`MPI_ERR_FILE`)
* MPI_ERR_FILE_EXISTS (`MPI_ERR_FILE_EXISTS`)
* MPI_ERR_FILE_IN_USE (`MPI_ERR_FILE_IN_USE`)
* MPI_ERR_GROUP (`MPI_ERR_GROUP`)
* MPI_ERR_INFO (`MPI_ERR_INFO`)
* MPI_ERR_INFO_KEY (`MPI_ERR_INFO_KEY`)
* MPI_ERR_INFO_NOKEY (`MPI_ERR_INFO_NOKEY`)
* MPI_ERR_INFO_VALUE (`MPI_ERR_INFO_VALUE`)
* MPI_ERR_IN_STATUS (`MPI_ERR_IN_STATUS`)
* MPI_ERR_INTERN (`MPI_ERR_INTERN`)
* MPI_ERR_IO (`MPI_ERR_IO`)
* MPI_ERR_KEYVAL (`MPI_ERR_KEYVAL`)
* MPI_ERR_LOCKTYPE (`MPI_ERR_LOCKTYPE`)
* MPI_ERR_NAME (`MPI_ERR_NAME`)
* MPI_ERR_NO_MEM (`MPI_ERR_NO_MEM`)
* MPI_ERR_NO_SPACE (`MPI_ERR_NO_SPACE`)
* MPI_ERR_NO_SUCH_FILE (`MPI_ERR_NO_SUCH_FILE`)
* MPI_ERR_NOT_SAME (`MPI_ERR_NOT_SAME`)
* MPI_ERR_OP (`MPI_ERR_OP`)
* MPI_ERR_OTHER (`MPI_ERR_OTHER`)
* MPI_ERR_PENDING (`MPI_ERR_PENDING`)
* MPI_ERR_PORT (`MPI_ERR_PORT`)
* MPI_ERR_PROC_ABORTED (`MPI_ERR_PROC_ABORTED`)
* MPI_ERR_QUOTA (`MPI_ERR_QUOTA`)
* MPI_ERR_RANK (`MPI_ERR_RANK`)
* MPI_ERR_READ_ONLY (`MPI_ERR_READ_ONLY`)
* MPI_ERR_REQUEST (`MPI_ERR_REQUEST`)
* MPI_ERR_RMA_ATTACH (`MPI_ERR_RMA_ATTACH`)
* MPI_ERR_RMA_CONFLICT (`MPI_ERR_RMA_CONFLICT`)
* MPI_ERR_RMA_FLAVOR (`MPI_ERR_RMA_FLAVOR`)
* MPI_ERR_RMA_RANGE (`MPI_ERR_RMA_RANGE`)
* MPI_ERR_RMA_SHARED (`MPI_ERR_RMA_SHARED`)
* MPI_ERR_RMA_SYNC (`MPI_ERR_RMA_SYNC`)
* MPI_ERR_ROOT (`MPI_ERR_ROOT`)
* MPI_ERR_SERVICE (`MPI_ERR_SERVICE`)
* MPI_ERR_SESSION (`MPI_ERR_SESSION`)
* MPI_ERR_SIZE (`MPI_ERR_SIZE`)
* MPI_ERR_SPAWN (`MPI_ERR_SPAWN`)
* MPI_ERR_TAG (`MPI_ERR_TAG`)
* MPI_ERR_TOPOLOGY (`MPI_ERR_TOPOLOGY`)
* MPI_ERR_TRUNCATE (`MPI_ERR_TRUNCATE`)
* MPI_ERR_TYPE (`MPI_ERR_TYPE`)
* MPI_ERR_UNKNOWN (`MPI_ERR_UNKNOWN`)
* MPI_ERR_UNSUPPORTED_DATAREP (`MPI_ERR_UNSUPPORTED_DATAREP`)
* MPI_ERR_UNSUPPORTED_OPERATION (`MPI_ERR_UNSUPPORTED_OPERATION`)
* MPI_ERR_VALUE_TOO_LARGE (`MPI_ERR_VALUE_TOO_LARGE`)
* MPI_ERR_WIN (`MPI_ERR_WIN`)
* MPI_ERR_LASTCODE (`MPI_ERR_LASTCODE`){{% /markdown
%}}</details></td></tr>
</table>
#### Operands:
| Operand | Description |
| :-----: | ----------- |
| `val` | MPI function call return value
#### Results:
| Result | Description |
| :----: | ----------- |
| `res` | 1-bit signless integer
### `mpi.send` (mpi::SendOp)
_Equivalent to `MPI_Send(ptr, size, dtype, dest, tag, MPI_COMM_WORLD)`_
Syntax:
```
operation ::= `mpi.send` `(` $ref `,` $tag `,` $rank `)` attr-dict `:` type($ref) `,` type($tag) `,` type($rank)(`->` type($retval)^)?
```
MPI_Send performs a blocking send of `size` elements of type `dtype` to
rank
`dest`. The `tag` value and communicator enables the library to
determine
the matching of multiple sends and receives between the same ranks.
Communicators other than `MPI_COMM_WORLD` are not supprted for now.
This operation can optionally return an `!mpi.retval` value that can be
used
to check for errors.
#### Operands:
| Operand | Description |
| :-----: | ----------- |
| `ref` | memref of any type values
| `tag` | 32-bit signless integer
| `rank` | 32-bit signless integer
#### Results:
| Result | Description |
| :----: | ----------- |
| `retval` | MPI function call return value
## Attribute definition
### MPI_ErrorClassEnumAttr
MPI error class name
Syntax:
```
#mpi.errclass<
::mlir::mpi::MPI_ErrorClassEnum # value
>
```
Enum cases:
* MPI_SUCCESS (`MPI_SUCCESS`)
* MPI_ERR_ACCESS (`MPI_ERR_ACCESS`)
* MPI_ERR_AMODE (`MPI_ERR_AMODE`)
* ... *all other MPI error codes*
#### Parameters:
| Parameter | C++ type | Description |
| :-------: | :-------: | ----------- |
| value | `::mlir::mpi::MPI_ErrorClassEnum` | an enum of type
MPI_ErrorClassEnum |
## Type definition
### RetvalType
MPI function call return value
Syntax: `!mpi.retval`
This type represents a return value from an MPI function vall.
This value can be MPI_SUCCESS, MPI_ERR_IN_STATUS, or any error code.
This return value can be compared agains the known MPI error classes
represented by `#mpi.errclass` using the `mpi.retval_check` operation.
This adds a new custom CopyPrivateVarList to the single operation.
Each list item is formed by a reference to the variable to be
updated, its type and the function to be used to perform the copy.
It will be translated to LLVM IR using OpenMP builder, that will
use the information in the copyprivate list to call
__kmpc_copyprivate.
This is patch 2 of 4, to add support for COPYPRIVATE in Flang.
Original PR: https://github.com/llvm/llvm-project/pull/73128
We implement a function to compute the generating function corresponding
to a full-dimensional parametric polytope whose tangent cones are all
unimodular.
We fix a bug in unimodGenFunc to check the absolute value of the index.
We also implement Matrix<T>::negateMatrix() and Matrix<T>::scaleRow for
convenience.
This reverts commit b4c7152eb4f7971c111e3e2f60b55892def58d5d.
Downstream regression due to another issue that this PR exposes. We have identified the work-items to fix the new issue here: https://github.com/openxla/iree/issues/16406
Co-authored-by: Han-Chung Wang <hanchung@google.com>
Add a new rewrite class for "operation movements". This rewrite class
can roll back `moveOpBefore` and `moveOpAfter`.
`RewriterBase::moveOpBefore` and `RewriterBase::moveOpAfter` is no
longer virtual. (The dialect conversion can gather all required
information for rollbacks from listener notifications.)
Throughout the rewrite process, the dialect conversion maintains a list
of "block actions" that can be rolled back upon failure. This commit
encapsulates the existing block actions into separate classes, making it
easier to add additional actions in the future.
This commit also renames "block actions" to "IR rewrites". In a
subsequent commit, an "operation rewrite" class that allows rolling back
movements of single operations is added. This is to support
`moveOpBefore` in the dialect conversion.
Rewrites have two methods: `commit()` commits an action. It can no
longer be rolled back afterwards. `rollback()` undoes a rewrite. It can
no longer be committed afterwards.
Context: Conversion patterns provide a `ConversionPatternRewriter` to
modify the IR. `ConversionPatternRewriter` provides the public API. Most
function calls are forwarded/handled by `ConversionPatternRewriterImpl`.
The dialect conversion uses the listener infrastructure to get notified
about op/block insertions.
In the current design, `ConversionPatternRewriter` inherits from both
`PatternRewriter` and `Listener`. The conversion rewriter registers
itself as a listener. This is problematic because listener functions
such as `notifyOperationInserted` are now part of the public API and can
be called from conversion patterns; that would bring the dialect
conversion into an inconsistent state.
With this commit, `ConversionPatternRewriter` no longer inherits from
`Listener`. Instead `ConversionPatternRewriterImpl` inherits from
`Listener`. This removes the problematic public API and also simplifies
the code a bit: block/op insertion notifications were previously
forwarded to the `ConversionPatternRewriterImpl`. This is no longer
needed.
Expose the API for constructing and inspecting StructTypes from the LLVM
dialect. Separate constructor methods are used instead of overloads for
better readability, similarly to IntegerType.
This PR that introduces the `nvvm.barrier` OP to the NVVM dialect.
Currently, NVVM only supports the `nvvm.barrier0`, which synchronizes
all threads using barrier resource 0.
The new `nvvm.barrier` has two essential arguments: the barrier resource
and the number of threads. This added flexibility allows for selective
synchronization of threads within a CTA, aligning with the capabilities
provided by LLVM intrinsics or the PTX model.
I think we can deprecate `nvvm.barrier0` in favor of the more generic
`nvvm.barrier`.
```
// Equivalent to nvvm.barrier0 (or __syncthreads() in CUDA)
nvvm.barrier
// Synchronize all threads using the 3rd barrier resource.
nvvm.barrier id = 3
// Synchronize %numberOfThreads threads using the 3rd barrier resource.
nvvm.barrier id = 3 number_of_threads = %numberOfThreads
```
…ct LevelType from LevelFormat and properties instead.
**Rationale**
We used to explicitly declare every possible combination between
`LevelFormat` and `LevelProperties`, and it now becomes difficult to
scale as more properties/level formats are going to be introduced.
This patch reworks the way that wsloop reduction operations function to
better match the expected semantics from the OpenMP specification,
following the rework of parallel reductions.
The new semantics create a private reduction variable as a block
argument which should be used normally for all operations on that
variable in the region; this private variable is then combined with the
others into the shared variable. This way no special omp.reduction
operations are needed inside the region. These block arguments follow
the loop control block arguments.
---------
Co-authored-by: Kiran Chandramohan <kiran.chandramohan@arm.com>
Common backends (LLVM, SPIR-V) only supports 1D vectors, LLVM conversion
handles ND vectors (N >= 2) as `array<array<... vector>>` and SPIR-V
conversion doesn't handle them at all at the moment. Sometimes it's
preferable to treat multidim vectors as linearized 1D. Add pass to do
this. Only constants and simple elementwise ops are supported for now.
@krzysz00 I've extracted yours result type conversion code from
LegalizeToF32 and moved it to common place.
Also, add ConversionPattern class operating on traits.
This patch adds support for the depend clause in a number of OpenMP
directives/constructs related to offloading. Specifically, it adds the
handling of the depend clause when it is used with the following
constructs
- target
- target enter data
- target update data
- target exit data
Currently, `phaseParity` argument of `nvgpu.mbarrier.try_wait.parity` is
index. This can cause a problem if it's passed any value different than
0 or 1. Because the PTX instruction only accepts even or odd phase. This
PR makes phaseParity argument i1 to avoid misuse.
Here is the information from PTX doc:
```
The .parity variant of the instructions test for the completion of the phase indicated
by the operand phaseParity, which is the integer parity of either the current phase or
the immediately preceding phase of the mbarrier object. An even phase has integer
parity 0 and an odd phase has integer parity of 1. So the valid values of phaseParity
operand are 0 and 1.
```
See for more information:
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-test-wait-mbarrier-try-wait
Allow TMA's last dimension to be non-128B when swizzling mode is not
set.
Test `tma_load_64x8_8x128_noswizzle.mlir` is failing due to the
verifier. This PR will fix that
This PR moves lowering of math dialect later in the pipeline. Because
math dialect is lowered correctly by createConvertGpuOpsToNVVMOps for
GPU target, and it needs to run it first.
Reland #78556
This PR adds a new op to the OpenMP dialect: `PrivateClauseOp`. This op
will be later used to model `[first]private` clauses for differnt OpenMP
directives.
This is part of productizing the "delayed privatization" PoC which can
be found in #79862.
