Context: Conversion patterns provide a `ConversionPatternRewriter` to
modify the IR. `ConversionPatternRewriter` provides the public API. Most
function calls are forwarded/handled by `ConversionPatternRewriterImpl`.
The dialect conversion uses the listener infrastructure to get notified
about op/block insertions.
In the current design, `ConversionPatternRewriter` inherits from both
`PatternRewriter` and `Listener`. The conversion rewriter registers
itself as a listener. This is problematic because listener functions
such as `notifyOperationInserted` are now part of the public API and can
be called from conversion patterns; that would bring the dialect
conversion into an inconsistent state.
With this commit, `ConversionPatternRewriter` no longer inherits from
`Listener`. Instead `ConversionPatternRewriterImpl` inherits from
`Listener`. This removes the problematic public API and also simplifies
the code a bit: block/op insertion notifications were previously
forwarded to the `ConversionPatternRewriterImpl`. This is no longer
needed.
https://github.com/llvm/llvm-project/pull/80683 revealed that
hlfir.as_expr was propagating the temporary buffer for polymorphic
values as an allocatable while codegen later expects to be working with
fir.box/fir.class but not fir.ref<box/class> when processing the
operations using the hlfir.as_expr result.
Dereference the temporary allocatable as soon as it is created.
This change makes the callback consistent with
`notifyOperationInserted`: both now notify about IR insertion, not IR
creation. See also #78988.
This change also simplifies the dialect conversion: it is no longer
necessary to override the `inlineRegionBefore` method. All information
that is necessary for rollback is provided with the
`notifyBlockInserted` callback.
The pattern rewriter documentation states that "*all* IR mutations [...]
are required to be performed via the `PatternRewriter`." This commit
adds two functions that were missing from the rewriter API:
`moveOpBefore` and `moveOpAfter`.
After an operation was moved, the `notifyOperationInserted` callback is
triggered. This allows listeners such as the greedy pattern rewrite
driver to react to IR changes.
This commit narrows the discrepancy between the kind of IR modification
that can be performed and the kind of IR modifications that can be
listened to.
This commit renames 4 pattern rewriter API functions:
* `updateRootInPlace` -> `modifyOpInPlace`
* `startRootUpdate` -> `startOpModification`
* `finalizeRootUpdate` -> `finalizeOpModification`
* `cancelRootUpdate` -> `cancelOpModification`
The term "root" is a misnomer. The root is the op that a rewrite pattern
matches against
(https://mlir.llvm.org/docs/PatternRewriter/#root-operation-name-optional).
A rewriter must be notified of all in-place op modifications, not just
in-place modifications of the root
(https://mlir.llvm.org/docs/PatternRewriter/#pattern-rewriter). The old
function names were confusing and have contributed to various broken
rewrite patterns.
Note: The new function names use the term "modify" instead of "update"
for consistency with the `RewriterBase::Listener` terminology
(`notifyOperationModified`).
This got "lost" in the HLFIR transformation. This patch applies the old
attribute to the AssociateOp that needs it, and forwards it to the
AllocaOp that is generated when lowering to FIR.
To avoid the overhead of deallocating allocatable components of the
elemental temporary result on every iteration of the elemental operation,
we can use a shallow copy instead of deep-copy assign.
Assumed shape array are using descriptor and must be handled differently
than known shape arrays. This patch adds support to generate the `init`
and `combiner` region for the reduction recipe operation with assumed
shape array by using the descriptor and the HLFIR lowering path.
`createTempFromMold` function is moved from
`flang/lib/Optimizer/HLFIR/Transforms/BufferizeHLFIR.cpp` to
`flang/include/flang/Optimizer/Builder/HLFIRTools.h` to be reused to
create the private copy.
This set of commits resolves some of the issues with elemental calls producing
results that may require finalization, and also some memory leak issues due to
the missing deallocation of allocatable components of the temporary buffers
created by the bufferization pass.
- [flang][runtime] Expose Finalize API for derived types.
- [flang][hlfir] Add 'finalize' attribute for DestroyOp.
- [flang][hlfir] Postpone result finalization for elemental calls.
The results of elemental calls generated inside hlfir.elemental must not
be finalized/destructed before they are copied into the resulting
array. The finalization must be done on the array as a whole
(e.g. there might be different scalar and array finalization routines).
The finalization work is left to the hlfir.destroy corresponding
to this hlfir.elemental.
- [flang][hlfir] Tighten requirements on hlfir.end_associate operand.
If component deallocation might be required for the operand of
hlfir.end_associate, we have to be able to get the variable
shape/params to create a descriptor for calling the runtime.
This commit adds verification that we can do so.
- [flang][hlfir] Lower argument clean-ups using valid hlfir.end_associate.
The operand must be a Fortran entity, when allocatable component
deallocation may be required.
- [flang][hlfir] Properly clean-up temporary buffers in bufferization pass.
This commit combines changes for proper finalization and component
deallocation of the temporary buffers. The finalization part
relates to hlfir.destroy operations with 'finalize' attribute.
The component deallocation might be invoked for both hlfir.destroy
and hlfir.end_associate, if the operand is of a derived type
with allocatable component(s).
The changes are mostly in one function, so I decided not to split them.
- [flang][hlfir] Disable optimizations for hlfir.elemental requiring finalization.
If hlfir.elemental is coupled with hlfir.destroy with 'finalize' attribute,
the temporary array result of hlfir.elemental needs to be created
for the purpose of finalization. We cannot do certain optimizations
on such hlfir.elemental operations.
I was not able to come up with a test for the OptimizedBufferization pass,
but I put the check there as well.
This effectively reverts D154715.
The issue appears as the dialect conversion error because we try to
erase an op that has already been erased. See the added LIT test case
with HLFIR that may appear as a result of CSE.
The `adaptor.getSource()` is an operation producing a tuple,
which does not have users, so `allOtherUsesAreSafeForAssociate`
just looks at the empty list of users. So we get completely wrong
answers from it. This causes problems with the following
`eraseAllUsesInDestroys` that tries to remove the `DestroyOp` twice
during both `hflir.associate` processing.
But we also cannot use `associate.getSource()` *efficiently*, because
the original users may still hang around: one example is the original body
of hlfir.elemental (see D154715), another example is other already converted
AssociateOp's that are pending removal in the rewriter
(that is why we have a temporary created for each hlfir.associate
in the newly added LIT case).
This patch just fixes the correctness issue. I think we have to separate
the buffer reuse analysis from the conversion itself.
I also tried to address the issues with the cloned bodies of `hlfir.elemental`,
but this should not matter since D155778: if `hlfir.associate` is inside
`hlfir.elemental`, it will end up inside a do-loop body region, so the early
exit added in D155778 will prevent the buffer reuse.
Reviewed By: tblah
Differential Revision: https://reviews.llvm.org/D158471
The polymorphic temporary array is created using the provided mold
and the shape of the hlfir.elemental. The array is allocated right away,
because it is going to be initialized element per element.
Depends on D157315
Reviewed By: clementval, tblah
Differential Revision: https://reviews.llvm.org/D157316
hlfir.count lowering was using incorrect default integer kind
by ignoring the kind specified in the ModuleOp.
Reviewed By: tblah
Differential Revision: https://reviews.llvm.org/D156017
If end_associate may execute more times than the expr value producer,
then it cannot take ownership of the expr buffer. Otherwise, it may
result in double-free errors.
Note that the LIT test exposes a different issue with fir.alloca
inside the do-loop produced for hlfir.elemental. This may cause
out-of-stack conditions in valid Fortran programs that are not expected
to run out of stack. I will create an issue for this.
Reviewed By: tblah
Differential Revision: https://reviews.llvm.org/D155778
This patch sets 'polymorphic' attribute of hlfir::ExprType when
the value is created from a polymorphic entity.
Memoization of such ExprType involves creating a mutable descriptor
on the stack, which is initialized (as a null box) and passed to
AllocatableApplyMold with the mold being the entity from which
the ExprType value is being created.
This patch fixes "creating polymorphic temporary" TODO and also
several cases of "'fir.convert' op invalid type conversion" error.
Reviewed By: tblah
Differential Revision: https://reviews.llvm.org/D155541
hlfir.get_length will not modify the buffer and so it is safe for a
hlfir.associate using the same expression buffer not to make its own
copy.
Differential Revision: https://reviews.llvm.org/D154942
Make a copy of the expression and associate that so that this is the
only use.
So far as I know, we don't currently generate code for an associate with
more than one use. This is here just in case.
Depends on D154715
Differential Revision: https://reviews.llvm.org/D154721
The associate operation checks if it is the only use of the hlfir.expr,
and if so it can take ownership of the hlfir.expr instead of copying it
(move semantics).
If this check is done by accessing the associate operation's arguments
directly (not through the AssociateOpAdaptor), the expression uses will
contain some operations which have been deleted. These can include prior
copies of the same associate operation, if that operation was cloned
(e.g. to lower a hlfir.elemental into a fir.do_loop). Accessing the
bufferized expression instead of the old hlfir.expr through the adaptor
avoids this false positive.
Differential Revision: https://reviews.llvm.org/D154715
This fixes the majority of cases where we hit the "hlfir.associate of
hlfir.expr with more than one use" TODO. In particular, this allows cam4
to be built.
hlfir.shape_of is just a way to delay reading shape information until
after intrinsics have been lowered to FIR runtime calls. It gets the
shape information from reading existing SSA values (e.g. fetching the
shape used when hlfir.declare'ing the variable).
Therefore hlfir.shape_of doesn't affect decisions about when to
deallocate the buffer.
Differential Revision: https://reviews.llvm.org/D154521
Lower hlfir.get_length into the char length inquiry of the bufferized
entity. In some cases the codegen will fail with
`hlfir.associate of hlfir.expr with more than one use` - this will be fixed
separately (after D154521).
Depends on D154560
Reviewed By: tblah, jeanPerier
Differential Revision: https://reviews.llvm.org/D154561
When `AssignOp` is used with LHS that is a compiler generated temporary
special care must be taken to initialize the temporary and avoid
finalizations of its components. This change-set adds optional
`temporary_lhs` attribute for `AssignOp` to convey this information
to HLFIR-to-FIR conversion pass. Currently, this results in
calling `AssignTemporary` runtime for doing the assignment.
Reviewed By: jeanPerier, tblah
Differential Revision: https://reviews.llvm.org/D152482
This patch adds an hlfir operation called `char_extremum`, which takes the
lexicographic comparison between a variadic number (minimum of 2 arguments) of
characters.
Discussion for this work can be found in the draft revision found
[here](https://reviews.llvm.org/D143326). The reason I'm not promoting that draft to
a true patch for review was because I needed to separate out the op
definition/codegen and lowering as two separate patches, as preferred by
@jeanPerier.
Differential Revision: https://reviews.llvm.org/D152474
Previously only a constant reference was stored in the FirOpBuilder.
However, a lot of code was merged using
FirOpBuilder builder{rewriter, getKindMapping(mod)};
This is incorrect because the KindMapping returned will go out of scope
as soon as FirOpBuilder's constructor had run. This led to an infinite
loop running some tests using HLFIR (because the stack space containing
the kind mapping was re-used and corrupted).
One solution would have just been to fix the incorrect call sites,
however, as a large number of these had already made it past review, I
decided to instead change FirOpBuilder to store its own copy of the
KindMapping. This is not costly because nearly every time we construct a
KindMapping is exclusively to construct a FirOpBuilder. To make this
common pattern simpler, I added a new constructor to FirOpBuilder which
calls getKindMapping().
Differential Revision: https://reviews.llvm.org/D151881
The codegen tried to fir.convert !fir.box to !fir.ref for this case.
I used BoxAddr under a check for the type mismatch, but I am not sure
if this is the right fix. Maybe it has to be handled in the lowering.
The bufferization pass must create the tuple for these operations, because
the users may require it. For example, in case of ElementalOp inlining
a DestroyOp may be generated for the operand of YieldElementOp, and
the operand may be ApplyOp->NoReassocOp chain.
Differential Revision: https://reviews.llvm.org/D150343
First issue is that the clean-ups were generated after the yield_element
operation that must be the terminator. Second issue is that codegen for
elemenal operation was not working properly with nested elemental ops.
Differential Revision: https://reviews.llvm.org/D149921
If possible the shape is gotten from the bufferization of the expr
argument.
The simple cases should already have been resolved during lowering. This
is mostly intended for cases where shape information is added in between
lowering and the end of bufferization (for example transformational
intrinsics with assumed shape arguments).
Depends on: D146832
Differential Revision: https://reviews.llvm.org/D146833
hlfir.assign must be rewritten in the bufferize pass since its operands,
that can be expressions, may have been updated. This is just
an operand update rewrite. The previous code was replacing the
operation, but it was dropping all hlfir.assign on the floor doing
so. This broke allocatable assignment semantics that use attributes.
Update the operands in place instead to preserve the attributes, if any.
Differential Revision: https://reviews.llvm.org/D148310
The bufferization pass was propagating the raw alloca storage
(which may not allow to later retrieve the length) instead of
the hlfir variable value (which is guaranteed to hold the
character length).
Fix this and makes packageBufferizedExpr "storage" argument and
getBufferizedExprStorage return an hlfir::Entity to avoid similar
error in the future (the caller of packageBufferizedExpr will have
to think a bit when adding the explicit hlfir::Entity{} cast).
Differential Revision: https://reviews.llvm.org/D148307
The current code is wrong: it is doing an alloca with the declared
type instead of the dynamic type, leading to undefined behavior
when the dynamic type and declared type differ and the temporary
is later used.
This also introduces some `fir.alloca none` for unlimited polymorphic that
are not allocating the right thing at all.
Add TODOs for now, the correct thing to do will probably be to use the
runtime (like AssignTemporary), but since this happens in code doing
"mold" temp allocation, I first need to check if there is a need for "mold"
temporary creation not followed by an assign, or if this can be combined
with the assign instead (for HLFIR, it is pretty easy combine this from as_expr
codegen, not sure for the current lowering).
Differential Revision: https://reviews.llvm.org/D147333
This move is useful for a few reasons:
- It is easier to see what the intrinsic lowering is doing when the
operations it creates are not immediately lowered
- When lowering a HLFIR intrinsic generates an operation which needs
to be lowered by another pattern matcher in the same pass, MLIR will
run that other substitution before validating and finalizing the
original changes. This means that the erasure of operations is not
yet visible to subsequent matchers, which hugely complicates
transformations (in this case, hlfir.exprs cannot be rewritten
because they are still used by the now-erased HLFIR intrinsic op.
Reviewed By: jeanPerier
Differential Revision: https://reviews.llvm.org/D145502
When the associated expression came from a moved variable, the
type of the moved variable may not exactly match the hlfir.associate
result and cannot be re-used directly. Insert fir.convert/fir.box_addr
as needed.
Reviewed By: clementval
Differential Revision: https://reviews.llvm.org/D144557
We can't test lowering calls with hlfir.expr arguments yet because this
hits a not yet implemented: "get shape form HLFIR expr without producer
holding the shape".
Differential Revision: https://reviews.llvm.org/D144098
This duplicates some argument lowering and return value processing from
flang/lib/Lower/ConvertCall.cpp. The existing code in CovertCall lowers
directly into the fir.call (without the hlfir transformational intrinsic
operation), and it is too tied to the lowering code to move into
flang/lib/Optimizer to allow for more complete re-use here.
Differential Revision: https://reviews.llvm.org/D143512
Add the operation to mark the end of life of hlfir.expr.
As described in its description this is the easiest solution
to deploy given lowering "knows" where expression value are last
used.
However, inserting these points in lowering will probably make
it harder to do some IR transformation that would move the code
using or creating hlfir.expr (no use should be moved after an
hlfir.destroy).
Once the dust settle with the HLFIR change, it will be worth assessing
the situation and see if an analysis could do a better and safer job at
finding those destruction points.
Depends on D141832
Reviewed By: clementval
Differential Revision: https://reviews.llvm.org/D141839
hlfir.as_expr allows turning an array, character, or derived type
variable into a value when it the usage require an hlfir.expr (e.g,
when returning the element value inside and hlfir.elemental).
The default implementation of this operation in bufferization is to
make a copy of the variable into a temporary buffer.
This adds a time and memory overhead in cases where such copy is not
needed because the variable is already a temporary that was created
in lowering to compute the expression value, and the "as_expr" is
the sole usage of the variable.
This is for instance the case for many transformational intrinsics
that do not have hlfir.expr operation (at least for now, but some may
never benefit from having one) and must be implemented "on memory"
in lowering.
This patch adds a way to "move" the variable storage along its value.
It allows the bufferization to re-use the variable storage for the
hlfir.expr created by hlfir.as_expr, and in exchange, the
responsibility of deallocating the buffer (if the variable was heap
allocated) if passed along to the hlfir.expr, and will need to be
done after the last hlfir.expr usage.
Differential Revision: https://reviews.llvm.org/D141832
Hlfir.designate was made to support substrings but so far substrings
were not yet lowered to it. Implement support for them.
Differential Revision: https://reviews.llvm.org/D140310
This will implement evaluate::SetLength where the length of
a character entity is changed (with trimming and padding).
Differential Revision: https://reviews.llvm.org/D140219
The motivation is to have it accessible in HLFIROps.cpp to
use it in hlfir.set_length builder to build the result length
type as best as possible.
Differential Revision: https://reviews.llvm.org/D140214
Lower procedure ref to user defined elemental procedure when:
- there are no arguments that may be dynamically optional
- for functions, the result has no length parameters
- the reference can be unordered
- there are not character by value arguments
This uses the recently added hlfir.elemental operation and tools.
The "core" of the argument preparation is shared between elemental
and non elemental calls (genUserCalls is code moved without any
functional changes)
Differential Revision: https://reviews.llvm.org/D140118
The defining op of HLFIR variables is expected to be visible
in most cases, but HLFIR codegen won't rely on it from a correctness
point of view.
This patch allows building a fir.shape from an hlfir::Entity does not
have a visible FortranVariabeInterface defining op.
Differential Revision: https://reviews.llvm.org/D140099
Without any optimization or when it cannot be optimized before
bufferization, an hlfir.elemental lowers to an array temporary.
Its codegen consists in:
- allocating a temp given the type, shape, and length parameter arguments.
- generating a loop nest given the elemental shape
- inlining the body of the elemental inside the loops, and replacing the
yield_element by an assignment to an element of the temp.
Differential Revision: https://reviews.llvm.org/D140093
