This commit relaxes Mem2Reg's type equality requirement for the LLVM
dialect's load and store operations. For now, we only allow loads to be
promoted if the reaching definition can be casted into a value of the
target type.
For stores, the same conversion casting check is applied and we ensure
that their result is properly casted to the type of the memory slot.
This is necessary to satisfy assumptions of the general mem2reg pass, as
it creates block arguments with the types of the memory slot.
This relands https://github.com/llvm/llvm-project/pull/87504
Add `requiresReplacedValues` and `visitReplacedValues` methods to
`PromotableOpInterface`. These methods allow `PromotableOpInterface` ops
to transforms definitions mutated by a `store`.
This change is necessary to correctly handle the promotion of
`LLVM_DbgDeclareOp`.
---------
Co-authored-by: Théo Degioanni <30992420+Moxinilian@users.noreply.github.com>
This reverts commit d6e458219897fad0e460e663833b2190af48c06d as it
violates an assumption of Mem2Reg's block argument creation. Mem2Reg
strongly assumes that all involved values have the same type as the
alloca, which was relaxed by this PR. Therefore, branches got created
that jumped to basic blocks with differently typed block arguments.
This commit relaxes Mem2Reg's type equality requirement for the LLVM
dialect's load and store operations. For now, we only allow loads to be
promoted if the reaching definition can be casted into a value of the
target type.
For stores, all type checks are removed, as a non-volatile store that
does not write out the alloca's pointer can always be deleted.
This commit removes SROA's type consistency constraints from LLVM
dialect's GEPOp. The checks for valid indexing are now purely done by
computing the GEP's offset with the aid of the data layout.
To simplify handling of "nested subslots", we are tricking the SROA by
handing in memory slots that hold byte array types. This ensures that
subsequent accesses only need to check if their access will be
in-bounds. This lifts the requirement of determining the sub-types for
all but the first level of subslots.
This commit resolves a SROA bug caused by not properly checking if a
llvm store operation writes the pointer to memory or not. Now, we do no
longer consider stores that use a slot pointer as a value to store as
fixable.
This commit relaxes the assumption of type consistency for LLVM dialect
load and store operations in SROA. Instead, there is now a check that
loads and stores are in the bounds specified by the sub-slot they
access.
This commit additionally removes the corresponding patterns from the
type consistency pass, as they are no longer necessary.
Note: It will be necessary to extend Mem2Reg with the logic for
differently sized accesses as well. This is non-the-less a strict
upgrade for productive flows, as the type consistency pass can produce
invalid IR for some odd cases.
This commit expends the Mem2Reg and SROA interface methods with passed
in handles to a `DataLayout` structure. This is done to avoid
superfluous retreiving of data layouts during each conversion of
intrinsics.
This change, additionally, enables subsequent changes to make the LLVM
dialect implementation of these interfaces type agnostic.
This commit ensures that SROA does no longer attempt to split allocas
that are indexed into dynamically. Dynamic indices into arrays are
allowed to be negative or out-of-bounds, when the alloca containing the
array has memory backing these produced indices.
This commit renames 4 pattern rewriter API functions:
* `updateRootInPlace` -> `modifyOpInPlace`
* `startRootUpdate` -> `startOpModification`
* `finalizeRootUpdate` -> `finalizeOpModification`
* `cancelRootUpdate` -> `cancelOpModification`
The term "root" is a misnomer. The root is the op that a rewrite pattern
matches against
(https://mlir.llvm.org/docs/PatternRewriter/#root-operation-name-optional).
A rewriter must be notified of all in-place op modifications, not just
in-place modifications of the root
(https://mlir.llvm.org/docs/PatternRewriter/#pattern-rewriter). The old
function names were confusing and have contributed to various broken
rewrite patterns.
Note: The new function names use the term "modify" instead of "update"
for consistency with the `RewriterBase::Listener` terminology
(`notifyOperationModified`).
This commit implements the LLVM IR invariant intrinsics in LLVM dialect.
These intrinsics can be used to mark a program regions in which the
contents of a specific memory object will not change.
The LLVM dialect implementation also implements the
PromotableOpInterface to ensure Mem2Reg & SROA are able to promote
pointers that are marked using the invariant intrinsics.
Add initial support for DIExpression in LLVM dialect.
Similar to LLVM IR, DI Expression is encoded as a list of uint64. The
difference is that LLVM IR has helpers for understanding the expression
(e.g. for verification and pretty printing), whereas the current support
added by this PR treats the expression elements as opaque.
This commit removes the support for typed pointers from the LLVM
dialect. Typed pointers have been deprecated for a while and thus this
removal was announced in a PSA:
https://discourse.llvm.org/t/psa-removal-of-typed-pointers-from-the-llvm-dialect/74502
This change includes:
- Changing the ` LLVMPointerType`
- Removing remaining usages of the builders and the now removed element
type
- Fixing assembly formats that require fully qualified pointer types
- Updating ODS pointer constraints
The goal of this pattern is to eliminate all GEPs that have more than two indices by splitting it into multiple GEPs.
The advantage of this change is that the resulting GEPs only ever index into one aggregate at the time. This enables handling sub-aggregates in other patterns and also creates IR with easier to deduce pointer element types.
As a minor note, `getResultPtrElementType` for `GEPOp` was rewritten since it did not properly handle dynamic indices. The way GEPOp is specified, the resulting pointer element type can *always* be deduced from its base type and indices.
Differential Revision: https://reviews.llvm.org/D154692
Jeremy Morse noted in D154451 that LLVM doesn't drop a `dbg.value` when its value is being removed, but rather sets the operand to `undef`. This preserves the debug info and gives allows the debugger to instead inform the user that the variable has been optimized out, rather than not displaying the variable at all.
This patch fixes that mistake done in the previous revision by mirroring that behaviour in MLIR as well.
Differential Revision: https://reviews.llvm.org/D154476
This has previously been done for `llvm.intr.dbg.declare`, which is a common occurrence when the debug info points to the variable through the pointer, but may also occur when the `alloca` itself is a local variable in debug info.
Not doing so prevents `SROA` and `mem2reg` from promoting e.g. an `alloca`. We simply drop the value completetly, since there is no meaninful debug info that can be constructed instead as the pointer value is removed.
Differential Revision: https://reviews.llvm.org/D154451
This revision introduces new rewrites to improve the type consistency of
a program expressed in the LLVM dialect.
Type consistency means that a given opaque pointer is consistently used
assuming the same pointee type, in a best effort basis. The introduced
rewrites modify the program to improve type consistency while preserving
the same semantics. This is useful for two main reasons:
- Transformation passes in the LLVM dialect like SROA or Mem2Reg can
analyse code better if type information and structure is used in a
consistent manner. Opaque pointers make this difficult to enforce, but
type consistency improvements increase the amount of occurences where
reasonable analysis can pick up on transformable patterns.
- While LLVM IR is not particularly picky about inconsistent type uses,
it may be of interest to lift LLVM IR into higher level dialects.
Having more instances of consistent type information would help
lifting into dialects that do care about consistent types.
In order to detect cases of inconsistent uses, operations returning an
LLVMPointer can implement the GetResultPtrElementType interface, which
allows getting a hint of which type the provided pointer should see its
pointee as, if such hint is available. The provided rewrites will then
use this hint to attempt to modify operations using the pointers so they
use the hinted type when dealing with the pointer.
Two transformations have been implemented in this revision:
- When a load/store uses a struct ptr directly to write to the first
element of the struct, introduce a GEP to the first element so the
type structure is preserved.
- When a GEP statically indexes in a pointer with a base type
inconsistent with the hint, try to find indices using the hint as a
base type that would result in the same offset, and replace the GEP
with this indexing.
More transformations are possible and I hope this is only a beginning
for this simplification effort.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D153973
This revision introduces SROA and mem2reg support for the family of
memcpy-like intrinsics (memcpy, memcpy.inline and memmove).
The mem2reg implementation transforms memcpys of full types into loads
and store. Memcpy between two promotable slots always disappear.
The SROA implementation transforms memcpys of *entire* aggregate types
into memcpys of all of their fields.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D152898
This revision introduces support for memset intrinsics in SROA and
mem2reg for the LLVM dialect. This is achieved for SROA by breaking
memsets of aggregates into multiple memsets of scalars, and for mem2reg
by promoting memsets of single integer slots into the value the memset
operation would yield.
The SROA logic supports breaking memsets of static size operating at the
start of a memory slot. The intended most common case is for memsets
covering the entirety of a struct, most often as a way to initialize it
to 0.
The mem2reg logic supports dynamic values and static sizes as input to
promotable memsets. This is achieved by lowering memsets into
`ceil(log_2(n))` LeftShift operations, `ceil(log_2(n))` Or operations
and up to one ZExt operation (for n the byte width of the integer),
computing in registers the integer value the memset would create. Only
byte-aligned integers are supported, more types could easily be added
afterwards.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D152367
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This patch updates all remaining uses of the deprecated functionality in
mlir/. This was done with clang-tidy as described below and further
modifications to GPUBase.td and OpenMPOpsInterfaces.td.
Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```
Differential Revision: https://reviews.llvm.org/D151542
This revision introduces a generic implementation of Scalar Replacement
Of Aggregates. In contrast to the implementation in LLVM, this focuses
on the core of SROA: destructuring aggregates. By implementing
interfaces on allocators and accessors, memory allocators can be
destructured into smaller allocators, through the MemorySlot
abstraction.
This pass only works on aggregates that are accessed in a "type-safe"
way, that is within the bounds and respecting the type of a given memory
slot. The destructuring pattern and functions only peel off the first
layer of aggregates and can safely be applied repeatedly. For
convenience, the transformation is also available as a pass that will
apply the pattern repeatedly.
Depends on D149958
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D150186
This revision modifies the mem2reg interfaces and algorithm to be more
omfortable to use as a pattern. The motivation behind this is that
currently the pattern needs to be applied to the scope op of the region
in which allocators should be promoted. However, a more natural way to
apply the pattern would be to apply it on the allocator directly. This
is not only clearer but easier to parallelize.
This revision changes the mem2reg pattern to operate this way. This
required restraining the interfaces to only mutate IR using
RewriterBase, as the previously used escape hatch is not granular enough
to match on the region that is modified only. This has the unfortunate
cost of preventing batching allocator promotion and making the block
argument adding logic more complex. Because batching no longer made any
sense, I made the internal analyzer/promoter decoupling private again.
This also adds statistics to the mem2reg infrastructure.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D150432
This patch refactors the Mem2Reg infrastructure. It decouples
analysis from promotion, allowing for more control over the execution of
the logic. It also adjusts the interfaces to be less coupled to mem2reg
and more general. This will be useful for an upcoming revision
introducing generic SROA.
This commit reverts f333977eb20a and relands 91cff8a71872.
The original commit was reverted accidentally due to a misinterpretation
of a bazel build bot failure.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D149825
This patch refactors the Mem2Reg infrastructure. It decouples
analysis from promotion, allowing for more control over the execution of
the logic. It also adjusts the interfaces to be less coupled to mem2reg
and more general. This will be useful for an upcoming revision
introducing generic SROA.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D149825
