My usecase is simplifying the control flow generated by LoopVectorize
when vectorising loops whose tripcount is a function of the runtime
vector length. This can be problematic because:
* CSE is a pre-LoopVectorize transform and so it's common for an IR
function to include several calls to llvm.vscale(). (NOTE: Code
generation will typically remove the duplicates)
* Pre-LoopVectorize instcombines will rewrite some multiplies as shifts.
This leads to a mismatch between VL based maths of the scalar loop and
that created for the vector loop, which prevents some obvious
simplifications.
SCEV does not suffer these issues because it effectively does CSE during
construction and shifts are represented as multiplies.
If we know that the initial GEP was inbounds, and we change it to a sequence of
GEPs from the same base pointer where every offset is non-negative, then the
new GEPs are inbounds. So far, the implementation only checked if the extracted
offsets are non-negative. In cases where non-extracted offsets can be negative,
this would cause the inbounds flag to be wrongly preserved.
Fixes an issue in #130617 found by nikic.
This PR extends isSafeToCastConstAddrSpace to treat
ConstantAggregateZero like ConstantPointerNull.
Tests shows an extra addrspacecast instruction is removed and icmp
pointer vector operand's address space is now inferred.
This change is motivated by inspecting the test in commit f7629f5945f6.
Previously undef pointer operand is only supported for select inst,
where undef in generic AS behaves like `take the other side`.
This PR extends the support to other instructions, e.g. phi inst. Defer
joining and inferring constant pointer operand until all other operand
AS states considered.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Always add pointers proved to be uniform via legal/SCEV to worklist.
This extends the existing logic to handle a few more pointers known to
be uniform.
If all operands of the non-schedulable nodes were previously only
copyables, need to clear the dependencies of the original schedule data
for such copyable operands and recalculate them to correctly handle
number of dependecies.
Fixes#159406
After https://github.com/llvm/llvm-project/pull/153643, there may be a
BranchOnCond with constant condition in the entry block.
Simplify those in removeBranchOnConst. This removes a number of
redundant conditional branch from entry blocks.
In some cases, it may also make the original scalar loop unreachable,
because we know it will never execute. In that case, we need to remove
the loop from LoopInfo, because all unreachable blocks may dominate each
other, making LoopInfo invalid. In those cases, we can also completely
remove the loop, for which I'll share a follow-up patch.
Depends on https://github.com/llvm/llvm-project/pull/153643.
PR: https://github.com/llvm/llvm-project/pull/154510
`__NoopCoro_ResumeDestroy` currently has private linkage, which causes
[issues for
Arm64EC](https://github.com/llvm/llvm-project/issues/158341). The
Arm64EC lowering is trying to mangle and add thunks for
`__NoopCoro_ResumeDestroy`, since it sees that it's address is taken
(and, therefore, might be called from x64 code via a function pointer).
MSVC's linker requires that the function be placed in COMDAT (`LNK1361:
non COMDAT symbol '.L#__NoopCoro_ResumeDestroy' in hybrid binary`) which
trips an assert in the verifier (`comdat global value has private
linkage`) and the subsequent linking step fails since the private symbol
isn't in the symbol table.
Since there is no reason to use private linkage for
`__NoopCoro_ResumeDestroy` and other coro related functions have also
been [switched to internal linkage to improve
debugging](https://github.com/llvm/llvm-project/pull/151224), this
change switches to using internal linkage.
Fixes#158341
This adds a new pass for dropping assumes that are unlikely to be useful
for further optimization.
It works by discarding any assumes whose affected values are one-use
(which implies that they are only used by the assume, i.e. ephemeral).
This pass currently runs at the start of the module optimization
pipeline, that is post-inline and post-link. Before that point, it is
more likely for previously "useless" assumes to become useful again,
e.g. because an additional user of the value is introduced after
inlining + CSE.
Something like `call void @llvm.assume(i1 true) ["align"(ptr %p, i64 8)]`
is equivalent to placing an `align 8` attribute on the parameter
and should not be considered as capturing.
We currently infer `captures(none)` for calls that are read-only,
nounwind and willreturn. As pointed out in
https://github.com/llvm/llvm-project/issues/129090, this is not correct
even with this set of pre-conditions, because the function could
conditionally cause UB depending on the address.
As such, change this logic to instead report `captures(address)`. This
also allows dropping the nounwind and willreturn checks, as these can
also only capture the address.
When removing EH Pad blocks, the value defined by them becomes poison. These poison values are then used by `catchret` and `cleanupret`, which is invalid. This commit replaces those unreachable `catchret` and `cleanupret` instructions with `unreachable`.
This is a minor fixup for scalable vectors after f9f62ef4ae555a. It
handles them in the same way as other memory locations that are larger
than errno, preventing the failure on implicit conversion from a
scalable location.
As mentioned in #151067, current design of llvm.coro.end mixes two
functionalities: querying where we are and lowering to some code. This
patch separate these functionalities into independent intrinsics by
introducing a new intrinsic llvm.coro.is_in_ramp.
Ensure alias analyses mask out `errnomem` location, refining the
resulting modref info, when the given access/location does not
alias errno. This may occur either when TBAA proves there is no
alias with errno (e.g., float TBAA for the same root would be
disjoint with the int-only compatible TBAA node for errno); or
if the memory access size is larger than the integer size, or
when the underlying object is a potentially-escaping alloca.
Previous discussion: https://discourse.llvm.org/t/rfc-modelling-errno-memory-effects/82972.
The way that loops strength reduction works is that the target has to
upfront decide whether it wants its addressing to be preindex,
postindex, or neither. This choice affects:
* Which potential solutions we generate
* Whether we consider a pre/post index load/store as costing an AddRec
or not.
None of these choices are a good fit for either AArch64 or ARM, where
both preindex and postindex addressing are typically free:
* If we pick None then we count pre/post index addressing as costing one
addrec more than is correct so we don't pick them when we should.
* If we pick PreIndexed or PostIndexed then we get the correct cost for
that addressing type, but still get it wrong for the other and also
exclude potential solutions using offset addressing that could have less
cost.
This patch adds an "all" addressing mode that causes all potential
solutions to be generated and counts both pre and postindex as having
AddRecCost of zero. Unfortuntely this reveals problems elsewhere in how
we calculate the cost of things that need to be fixed before we can make
use of it.
The motivation for this patch is to close the gap between the
VPlan-based CSE and the legacy CSE, to make it easier to remove the
legacy CSE. Before this patch, stubbing out the legacy CSE leads to 22
test failures, and after this patch, there are only 12 failures, and all
of them seem to have a single root cause:
VPlanTransforms::createInterleaveGroups() and
VPInterleaveGroup::execute(). The improvements from this patch are of
course welcome.
While developing the patch, a miscompile was found when GEP
source-element-types differ, and this has been fixed.
Co-authored-by: Florian Hahn <flo@fhahn.com>
Co-authored-by: Luke Lau <luke@igalia.com>
This has been handy locally for debugging cloning issues. The NodeIds
are assigned sequentially on creation and included in the dumps and the
dot graphs.
No measurable memory increase was found for a large thin link.
I only changed one test
(Transforms/MemProfContextDisambiguation/basic.ll)
to actually check the emitted NodeIds, most ignore them.
Update calculateRegisterUsageForPlan to track live-ness of VPValues
instead of recipes. This gives slightly more accurate results for
recipes that define multiple values (i.e. VPInterleaveRecipe).
When tracking the live-ness of recipes, all VPValues defined by an
VPInterleaveRecipe are considered alive until the last use of any of
them. When tracking the live-ness of individual VPValues, we can
accurately track the individual values until their last use.
Note the changes in large-loop-rdx.ll and pr47437.ll. This patch
restores the original behavior before introducing VPlan-based liveness
tracking.
PR: https://github.com/llvm/llvm-project/pull/155301
If the commutable instruction can be represented as a non-commutable
vector instruction (like add 0, %v can be represented as a part of sub
nodes with operation sub %v, 0), its operands might still be reordered
and this should be accounted when checking for copyables in operands
Fixes#158293
This fixes a bug where zero cost instruction was hoisted to nearest
common dominator but the hoisted instruction's operands didn't dominate
the common dominator causing poison values.
Add extra test coverage for
https://github.com/llvm/llvm-project/pull/149706. The added loop should
be interleaved, after narrowing interleave groups, which requires moving
the transform earlier.
The scalarizeExtExtract transform assumed little-endian lane ordering,
causing miscompiles on big-endian targets such as AIX/PowerPC under -O3
-flto.
This patch updates the shift calculation to handle endianness correctly
for big-endian targets. No functional change
for little-endian targets.
Fixes#158197.
---------
Co-authored-by: Simon Pilgrim <llvm-dev@redking.me.uk>
The copied from constant memory analysis had a special case where
nocapture was not required for read-only calls without (or unused)
return. This is not correct, as the address can still be captured though
means other than memory and the return value, for example using
divergence. This code should not be trying to do its own nocapture
inference.
```llvm
%sub = sub nsw T %x, %y
%cmp = icmp sgt T %x, %y ; or sge
%neg = sub T 0, %sub
%abs = select i1 %cmp, T %sub, T %neg
```
becomes:
```llvm
%sub = sub nsw T %x, %y
%abs = call T @llvm.abs.T(T %sub, i1 false)
```
Alive2: https://alive2.llvm.org/ce/z/ApdJX8https://alive2.llvm.org/ce/z/gRTmZk
vputils::isSingleScalar(A) may return true to recipes that produce only
a single scalar value, but they could still end up as vector
instruction, because the recipe could not be converted to a
single-scalar VPInstruction/VPReplicateRecipe.
For now, only apply the fold for recipes guaranteed to produce a single
value, i.e. single-scalar VPInstructions and VPReplicateRecipes.
Fixes https://github.com/llvm/llvm-project/issues/158319.
