Conservatively only allow inlining for loads and stores that don't carry
any attributes that require handling while inlining. This can later be
relaxed when proper handling is introduced.
Reviewed By: Dinistro, gysit
Differential Revision: https://reviews.llvm.org/D141115
This change introduces new LLVMIR dialect operations to represent
TBAA root, type descriptor and access tag metadata nodes.
For the purpose of importing TBAA metadata from LLVM IR it only
supports the current version of TBAA format described in
https://llvm.org/docs/LangRef.html#tbaa-metadata (i.e. size-aware
representation introduced in D41501 is not supported).
TBAA attribute support is only added for LLVM::LoadOp and LLVM::StoreOp.
Support for intrinsics operations (e.g. LLVM::MemcpyOp) may be added later.
The TBAA attribute is represented as an array of access tags, though,
LLVM IR supports only single access tag per memory accessing instruction.
I implemented it as an array anticipating similar support in LLVM IR
to combine TBAA graphs with different roots for Flang - one of the options
described in https://docs.google.com/document/d/16kKZVmI585wth01VSaJAqZMZpoX68rcdBmgfj0kNAt0/edit#heading=h.jzzheaz9vqac
It should be easy to restrict MLIR operation to a single access tag,
if we end up using a different approach for Flang.
Differential Revision: https://reviews.llvm.org/D140768
This new option is set to `false` by default. It should be set only in Canonicalizer tests to detect faulty canonicalization patterns. I.e., patterns that prevent the canonicalizer from converging. The canonicalizer should always convergence on such small unit tests that we have in `canonicalize.mlir`.
Two faulty canonicalization patterns were detected and fixed with this change.
Differential Revision: https://reviews.llvm.org/D140873
This allows to inline regions containing pure LLVM ops into their call
sites. (Note that this is not related to inlining of llvm.func but to
any the inlining of any Callable.) For now, we only allow the inlining
of Pure ops to be conservative but other ops may be considered inlinable
in the future.
Testing for purity of ops requires the C++ equivalent of the Pure trait
from SideEffectInterfaces.td, which this patch also provide. Its
implementation calls the C++ equivalents of the two traits that the Pure
trait is based on and, thus, has to be kept with the tablegen trait.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D139187
The revision adds an inbounds attribute to the LLVM dialect
GEP operation. It extends the builders and the import and export
to support the optional inbounds attribute.
As all builders set inbounds to false by default, existing lowerings
from higher-level dialects to LLVM dialect are not affected by the
change. Canonicalization/folding remains untouched since it currently
does not implement any simplifications in case of undefined behavior
(the handling of undefined behavior is deferred to LLVM).
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D139821
The pass was not checking for uninitialized states due to dead code.
This patch also makes LLVMFuncOp correctly return a null body when it is
external.
Fixes#58807
Depends on D139388
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D139389
The revision makes additional integer debug info parameters such as
line or column numbers optional (e.g. the line and column numbers 0 are
elided when printing, which is in line with the LLVM implementation).
Depends on D138405
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D138437
Currently, the import of LLVMIR fails if the program contains debug
intrinsics. The revision adds support to import debug intrinsics that
have no debug expression attached and drops all debug intrinsics with a
non-empty debug expression. It also moves the existing debug intrinsics
into the "intr" namespace by deriving from LLVM_IntrOp.
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D138405
This is generated by running
```
sed --in-place 's/[[:space:]]\+$//' mlir/**/*.td
sed --in-place 's/[[:space:]]\+$//' mlir/**/*.mlir
```
Reviewed By: rriddle, dcaballe
Differential Revision: https://reviews.llvm.org/D138866
The revision makes specific debug information attribute parameters
optional since some of them can be omitted in LLVMIR. The additional
flexibility enables a later revision that will support importing
debug information from LLVMIR. A special case is the types parameter
of the SubroutineTypeAttr. For void functions, its first entry is
null in LLVMIR. This revision splits the type parameter in an optional
resultType parameter and an argumentTypes array to support this corner
case.
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D138206
So far the function argument attributes were only translated
for functions with bodies. This change makes sure that this
happens for functions without bodies (declarations) as well.
This is needed for https://github.com/llvm/llvm-project/issues/58579
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D137047
In D134622 the printed form of a pass manager is changed to include the
name of the op that the pass manager is anchored on. This updates the
`-pass-pipeline` argument format to include the anchor op as well, so
that the printed form of a pipeline can be directly passed to
`-pass-pipeline`. In most cases this requires updating
`-pass-pipeline='pipeline'` to
`-pass-pipeline='builtin.module(pipeline)'`.
This also fixes an outdated assert that prevented running a
`PassManager` anchored on `'any'`.
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D134900
The call_intrinsic op allows us to call LLVM intrinsics from the LLVMDialect without implementing a new op every time.
Reviewed By: lattner, rriddle
Differential Revision: https://reviews.llvm.org/D137187
This is required for D126305 code to propagate fastmath attributes
for Arith operations that are converted to LLVM IR intrinsics
operations.
LLVM IR intrinsic operations are using custom assembly format now
to avoid printing {fastmathFlags = #llvm.fastmath<none>}, which
is too verbose.
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D136225
This adds a subset of the necessary metadata for defining
debug info in the LLVM dialect. It doesn't import everything,
but just enough to start actually generating LLVM debug info
the expected way. Export/Import to LLVMIR will be added in a
followup.
Differential Revision: https://reviews.llvm.org/D136542
This greatly simplifies composing enums in attribute/type printers,
which currently reimplement these functions as needed.
Differential Revision: https://reviews.llvm.org/D136407
This moves the `LLVMArrayType` to a `TypeDef`. The main side-effect of
this change is that the syntax `array<4xi32>` is no longer allowed. It
was previously parsed and then printed as `array<4 x i32>`. Now the
syntax must be the latter.
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D136473
This diff updates the `fastmath` attribute in the LLVMIR dialect to use `tblgen`
classes that were developed after the initial LLVMIR `fastmath` implementation.
Using the `EnumAttr` `tblgen` classes brings the LLVMIR `fastmath` attribute in
line with other dialects, and eliminates some of the custom printing and parsing
code in the LLVMIR dialect.
Subsequent commits will further reduce the custom processing code for the LLVMIR
`fastmath` attribute by unifying printing/parsing functionality between the
LLVMIR and `arith` `fastmath` attributes. (The actual attributes will remain
separate, but the printing and parsing will be made generic, and will be usable
by other dialects/attributes.)
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D135289
It adds the missing `bar.warp.sync` to the nvvm dialect. It is a barrier to synchronize for threads in a warp.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D135253
If any of the operands for FCmpOp is a vector, returns a vector<Nxi1>,
rather than an i1 type result.
Differential Revision: https://reviews.llvm.org/D134449
This patch adds the `llvm.intr.lifetime.start` and `llvm.intr.lifetime.end`
intrinsics which are used to indicate to LLVM the lifetimes of allocated
memory.
These ops have the requirement that the first argument (the size) be an
"immediate argument". I added an OpTrait to check this, but it is
possible that an approach like GEPArg would work too.
Reviewed By: rriddle, dcaballe
Differential Revision: https://reviews.llvm.org/D133867
With the transition to opaque pointers, type information has been
transferred to function parameter attributes. This patch adds correct
parsing for some of those arguments and fixes some tests, that
previously used UnitAttr for those.
Differential Revision: https://reviews.llvm.org/D132366
The attribute is translated into LLVM's function attribute 'readnone'.
There is no explicit verification regarding conflicting 'readnone'
and function attributes from 'passthrough', though, LLVM would assert
if they are incompatible during LLVM IR creation.
Differential Revision: https://reviews.llvm.org/D131457
Default attributes were only handled by ODS accessors generated with the
intention that these behave as if set attributes. This addresses the
long standing TODO to address this inconsistency. Moving the
initialization to construction vs every access. Removing need for
duplicated default attribute population in python bindings.
Switch some of the OpenMP ones to optional attribute with default as the
currently set default values are not legal. May need to dig more there.
Switched LinAlg generated ones to optional attribute with default as its
quite widely used and unclear where it falls on two different
interpretations.
Differential Revision: https://reviews.llvm.org/D130916
This patch "modernizes" the implementation of these operations by
switching them to assembly formats and type inference traits.
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D131971
This patch moves `LLVM::ShuffleVectorOp` to assembly format and in the
process drops the extra type that can be inferred (both operand types
are required to be the same) and switches to a dense integer array.
The syntax change:
```
// Before
%0 = llvm.shufflevector %0, %1 [0 : i32, 0 : i32, 0 : i32, 0 : i32] : vector<4xf32>, vector<4xf32>
// After
%0 = llvm.shufflevector %0, %1 [0, 0, 0, 0] : vector<4xf32>
```
Reviewed By: dcaballe
Differential Revision: https://reviews.llvm.org/D132038
Follow-up to D123774, where the syntax of dense arrays was discussed. It
was included that the syntax should be changed to `array<i32: 1, 2>`.
This patch changes the syntax but importantly preserves the `[1, 2]`
syntax when embedding these attributes in assembly formats through ODS.
Reviewed By: mehdi_amini, jpienaar
Differential Revision: https://reviews.llvm.org/D131738
This patch "modernizes" the LLVM `insertvalue` and `extractvalue`
operations to use DenseI64ArrayAttr, since they only require an array of
indices and previously there was confusion about whether to use i32 or
i64 arrays, and to use assembly format.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D131537
This patch removes the `type` field from `Attribute` along with the
`Attribute::getType` accessor.
Going forward, this means that attributes in MLIR will no longer have
types as a first-class concept. This patch lays the groundwork to
incrementally remove or refactor code that relies on generic attributes
being typed. The immediate impact will be on attributes that rely on
`Attribute` containing a type, such as `IntegerAttr`,
`DenseElementsAttr`, and `ml_program::ExternAttr`, which will now need
to define a type parameter on their storage classes. This will save
memory as all other attribute kinds will no longer contain a type.
Moreover, it will not be possible to generically query the type of an
attribute directly. This patch provides an attribute interface
`TypedAttr` that implements only one method, `getType`, which can be
used to generically query the types of attributes that implement the
interface. This interface can be used to retain the concept of a "typed
attribute". The ODS-generated accessor for a `type` parameter
automatically implements this method.
Next steps will be to refactor the assembly formats of certain operations
that rely on `parseAttribute(type)` and `printAttributeWithoutType` to
remove special handling of type elision until `type` can be removed from
the dialect parsing hook entirely; and incrementally remove uses of
`TypedAttr`.
Reviewed By: lattner, rriddle, jpienaar
Differential Revision: https://reviews.llvm.org/D130092
The implementation and API of GEP Op has gotten a bit convoluted over the time. Issues with it are:
* Misleading naming: `indices` actually only contains the dynamic indices, not all of them. To get the amount of indices you need to query the size of `structIndices`
* Very difficult to iterate over all indices properly: One had to iterate over `structIndices`, check whether it contains the magic constant `kDynamicIndex`, if it does, access the next value in `index` etc.
* Inconvenient to build: One either has create lots of constant ops for every index or have an odd split of passing both a `ValueRange` as well as a `ArrayRef<int32_t>` filled with `kDynamicIndex` at the correct places.
* Implementation doing verification in the build method
and more.
This patch attempts to address all these issues via convenience classes and reworking the way GEP Op works:
* Adds `GEPArg` class which is a sum type of a `int32_t` and `Value` and is used to have a single convenient easy to use `ArrayRef<GEPArg>` in the builders instead of the previous `ValueRange` + `ArrayRef<int32_t>` builders.
* Adds `GEPIndicesAdapter` which is a class used for easy random access and iteration over the indices of a GEP. It is generic and flexible enough to also instead return eg. a corresponding `Attribute` for an operand inside of `fold`.
* Rename `structIndices` to `rawConstantIndices` and `indices` to `dynamicIndices`: `rawConstantIndices` signifies one shouldn't access it directly as it is encoded, and `dynamicIndices` is more accurate and also frees up the `indices` name.
* Add `getIndices` returning a `GEPIndicesAdapter` to easily iterate over the GEP Ops indices.
* Move the verification/asserts out of the build method and into the `verify` method emitting op error messages.
* Add convenient builder methods making use of `GEPArg`.
* Add canonicalizer turning dynamic indices with constant values into constant indices to have a canonical representation.
The only breaking change is for any users building GEPOps that have so far used the old `ValueRange` + `ArrayRef<int32_t>` builder as well as those using the generic syntax.
Another follow up patch then goes through upstream and makes use of the new `ArrayRef<GEPArg>` builder to remove a lot of code building constants for GEP indices.
Differential Revision: https://reviews.llvm.org/D130730
The fold in it's current state only checks whether the amount of dynamic indices is 1. This does however not check for the presence of any struct indices, leading to an incorrect fold.
This patch fixes that issue by checking that struct indices are 1, which in addition to the pre-existing check that dynamic indices are 1, guarantees that the single index is a dynamic one.
Differential Revision: https://reviews.llvm.org/D129374
- Applying CallOpInterface on CallOp and InvokeOp.
- Applying CallableInterface on LLVMFuncOp.
We're testing the changes using CallGraph, which uses both interfaces.
Differential Revision: https://reviews.llvm.org/D129026
This patch adds three new LLVM intrinsic operations: llvm.intr.vastart/copy/end.
And its translation from LLVM IR.
This effectively removes a restriction, imposed by 0126dcf1f0a1, where
non-external functions in LLVM dialect cannot be variadic. At that time
it was not clear how LLVM intrinsics are going to be modeled, which
indirectly affects va_start/copy/end, the core intrinsics used in
variadic functions. But since we have LLVM intrinsics as normal
MLIR operations, it's not a problem anymore.
Differential Revision: https://reviews.llvm.org/D127540
These intrinsics will be needed to convert between fixed-length vectors
and scalable vectors.
This operation will be needed for VLS (vector-length specific)
vectorization, when interfacing with vector functions or intrinsics that
take scalable vectors as operands in a context where the length of our
vectors is known or assumed at compile time, but we still want to
generate scalable vector instructions.
Differential Revision: https://reviews.llvm.org/D127100
In order to support newer hardware, define wrappers around MFMA
intrinsics that have not previously been exposed in the ROCDL dialect.
A `amdgpu.mfma` wrapper around these instructions is in development
and will provide a more user-friendly interface to them.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D128079
The 'emit_c_wrappers' option in the FuncToLLVM conversion requests C interface
wrappers to be emitted for every builtin function in the module. While this has
been useful to bootstrap the interface, it is problematic in the longer term as
it may unintentionally affect the functions that should retain their existing
interface, e.g., libm functions obtained by lowering math operations (see
D126964 for an example). Since D77314, we have a finer-grain control over
interface generation via an attribute that avoids the problem entirely. Remove
the 'emit_c_wrappers' option. Introduce the '-llvm-request-c-wrappers' pass
that can be run in any pipeline that needs blanket emission of functions to
annotate all builtin functions with the attribute before performing the usual
lowering that accounts for the attribute.
Reviewed By: chelini
Differential Revision: https://reviews.llvm.org/D127952
If any of the operands for ICmpOp is a vector, returns a vector<Nxi1>
, rather than an i1 type result.
Differential Revision: https://reviews.llvm.org/D127536
Instead of casting the incoming operand into VectorType to check if it's
scalable or not.
This is the place I missed to fix in f088b99eac74.
Differential Revision: https://reviews.llvm.org/D127535
This is correct for all values, i.e. the same as promoting the division to fp32 in the NVPTX backend. But it is faster (~10% in average, sometimes more) because:
- it performs less Newton iterations
- it avoids the slow path for e.g. denormals
- it allows reuse of the reciprocal for multiple divisions by the same divisor
Test program:
```
#include <stdio.h>
#include "cuda_fp16.h"
// This is a variant of CUDA's own __hdiv which is fast than hdiv_promote below
// and doesn't suffer from the perf cliff of div.rn.fp32 with 'special' values.
__device__ half hdiv_newton(half a, half b) {
float fa = __half2float(a);
float fb = __half2float(b);
float rcp;
asm("{rcp.approx.ftz.f32 %0, %1;\n}" : "=f"(rcp) : "f"(fb));
float result = fa * rcp;
auto exponent = reinterpret_cast<const unsigned&>(result) & 0x7f800000;
if (exponent != 0 && exponent != 0x7f800000) {
float err = __fmaf_rn(-fb, result, fa);
result = __fmaf_rn(rcp, err, result);
}
return __float2half(result);
}
// Surprisingly, this is faster than CUDA's own __hdiv.
__device__ half hdiv_promote(half a, half b) {
return __float2half(__half2float(a) / __half2float(b));
}
// This is an approximation that is accurate up to 1 ulp.
__device__ half hdiv_approx(half a, half b) {
float fa = __half2float(a);
float fb = __half2float(b);
float result;
asm("{div.approx.ftz.f32 %0, %1, %2;\n}" : "=f"(result) : "f"(fa), "f"(fb));
return __float2half(result);
}
__global__ void CheckCorrectness() {
int i = threadIdx.x + blockIdx.x * blockDim.x;
half x = reinterpret_cast<const half&>(i);
for (int j = 0; j < 65536; ++j) {
half y = reinterpret_cast<const half&>(j);
half d1 = hdiv_newton(x, y);
half d2 = hdiv_promote(x, y);
auto s1 = reinterpret_cast<const short&>(d1);
auto s2 = reinterpret_cast<const short&>(d2);
if (s1 != s2) {
printf("%f (%u) / %f (%u), got %f (%hu), expected: %f (%hu)\n",
__half2float(x), i, __half2float(y), j, __half2float(d1), s1,
__half2float(d2), s2);
//__trap();
}
}
}
__device__ half dst;
__global__ void ProfileBuiltin(half x) {
#pragma unroll 1
for (int i = 0; i < 10000000; ++i) {
x = x / x;
}
dst = x;
}
__global__ void ProfilePromote(half x) {
#pragma unroll 1
for (int i = 0; i < 10000000; ++i) {
x = hdiv_promote(x, x);
}
dst = x;
}
__global__ void ProfileNewton(half x) {
#pragma unroll 1
for (int i = 0; i < 10000000; ++i) {
x = hdiv_newton(x, x);
}
dst = x;
}
__global__ void ProfileApprox(half x) {
#pragma unroll 1
for (int i = 0; i < 10000000; ++i) {
x = hdiv_approx(x, x);
}
dst = x;
}
int main() {
CheckCorrectness<<<256, 256>>>();
half one = __float2half(1.0f);
ProfileBuiltin<<<1, 1>>>(one); // 1.001s
ProfilePromote<<<1, 1>>>(one); // 0.560s
ProfileNewton<<<1, 1>>>(one); // 0.508s
ProfileApprox<<<1, 1>>>(one); // 0.304s
auto status = cudaDeviceSynchronize();
printf("%s\n", cudaGetErrorString(status));
}
```
Reviewed By: herhut
Differential Revision: https://reviews.llvm.org/D126158
