Make a separate routine for GEPs cost calculation and make
the approach uniform across load, store and GEP tree nodes.
Additional issue fixed is GEP cost savings were applied twice
for ScatterVectorize nodes (aka gather load) making them look
unrealistically profitable for vectorization.
Differential Revision: https://reviews.llvm.org/D140789
Use deduction guides instead of helper functions.
The only non-automatic changes have been:
1. ArrayRef(some_uint8_pointer, 0) needs to be changed into ArrayRef(some_uint8_pointer, (size_t)0) to avoid an ambiguous call with ArrayRef((uint8_t*), (uint8_t*))
2. CVSymbol sym(makeArrayRef(symStorage)); needed to be rewritten as CVSymbol sym{ArrayRef(symStorage)}; otherwise the compiler is confused and thinks we have a (bad) function prototype. There was a few similar situation across the codebase.
3. ADL doesn't seem to work the same for deduction-guides and functions, so at some point the llvm namespace must be explicitly stated.
4. The "reference mode" of makeArrayRef(ArrayRef<T> &) that acts as no-op is not supported (a constructor cannot achieve that).
Per reviewers' comment, some useless makeArrayRef have been removed in the process.
This is a follow-up to https://reviews.llvm.org/D140896 that introduced
the deduction guides.
Differential Revision: https://reviews.llvm.org/D140955
The validation of vplans could fail if an inloop reduction was created
with a block-in mask that did not dominate the reduction. This makes
sure that the insert point is set when creating the mask, to ensure it
dominates the reduction.
Differential Revision: https://reviews.llvm.org/D141003
Assuming that collecting frame allocas doesn't modify CFG we can safely
move DominatorTree construction outside loop and avoid expensive computations.
Differential Revision: https://reviews.llvm.org/D140818
Fixes https://github.com/llvm/llvm-project/issues/58565
The previous implementation visits operands in pre-order, but this does
not guarantee an instruction is visited before its uses. This can cause
instructions to be copied in the incorrect order. For example:
```
a = ...
b = add a, 1
c = add a, b
d = add b, a
```
Pre-order visits does not guarantee the order in which `a` and `b` are
visited. LoopUnrollAndJam may incorrectly insert `b` before `a`.
This patch implements post-order visits. By visiting dependencies first,
we guarantee that an instruction's dependencies are visited first.
Differential Revision: https://reviews.llvm.org/D140255
Switch simplification could sometimes fail to notice when an
intermediate case removal caused the switch condition to become
constant. This would cause the switch to be simplified into a
conditional branch rather than a direct branch.
Most of the time this didn't matter, except that occasionally
downstream parts of SimplifyCFG expect tautological branches to
already have been eliminated. The missed handling in switch
simplification would cause an assertion failure in the downstream
code.
Triggering the assertion failure is fairly sensitive to the exact
order of various simplifications.
Fixes https://github.com/llvm/llvm-project/issues/59768
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D140831
This reverts commit aa2414729ebbcb2d8f162e9002a3a6aa768b1f9d.
Previously-valid IR from a tensorflow test case (as shown on the
Diffusion revision for aa2414729ebbcb2d8f162e9002a3a6aa768b1f9d) started
hanging in the loop-vectorize pass. Reverting to keep everyone working.
analysis.
Missed the analysis of the shuffle mask when trying to analyze the
operands of the shuffle instruction during peeking through shuffle
instructions.
We shouldn't penalize instructions that have extra flags.
Drop the poison-generating flags if needed instead of bailing out.
This makes canonicalization/optimization more uniform.
There is a chance that dropping flags will cause some
other transform to not fire, but we added a preliminary
patch to avoid that with:
f0faea571403
See D140665 for more details.
The llvm.lifetime.start intrinsic guarantees that the address for a
given alloca is always the same. So variables with escaped addresses
reaching reaching a lifetime start/end block before and after a suspend
must be placed onto the coroutine frame even if the variable itself
is not alive across the suspend point.
This computes a new `LoopKill` flag in the suspend crossing data flow
anaysis to catch the case where a lifetime marker can reach itself
via suspend-crossing path.
This fixes https://llvm.org/PR52501
Differential Revision: https://reviews.llvm.org/D140231
Once the constraint system grows too large in terms of number of rows,
queries can become very slow. This patch adds a new option to limit the
number of rows tracked.
The python script below can be used to generate worst-case IR with a
chain of conditional branches with N branches.
With this limit, we get the following runtimes:
* python3 generate.py 100: 0.1s
* python3 generate.py 1000: 2s
* python3 generate.py 10000: 4s
Without the limit, the case with 1000 chained conditions takes 20+
seconds.
generate.py:
import sys
N = int(sys.argv[1])
args = []
checks = []
for i in range(0, N):
args.append('i32 %l{}'.format(i))
checks.append("""
bb{0}:
%c{0} = icmp uge i32 %l{0}, 100
br i1 %c{0}, label %bb{1}, label %exit
""".format(i, i+1))
print("""
define i1 @foo({0}) {{
{1}
bb{2}:
%c{2} = icmp uge i32 %l0, 100
ret i1 %c{2}
exit:
ret i1 false
}}
""".format(' ,'.join(args), '\n'.join(checks), N))
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D140926
HandlePassthroughUser may sometimes create a new entry for the OffsetInfo of a
user in the OffsetInfoMap. This can invalidate outstanding references into the
map, including the one which needs to be copied into the new entry. This
produces invalid offset info that can trigger assertions.
Fixed this by not using references at this point. The bug was originally
introduced in commit ID 0dc0a441323d41b4860668f38d290579e0de130c.
Reviewed By: ronlieb
Differential Revision: https://reviews.llvm.org/D140837
Using the function's address space makes no sense. Copied from the
existing test, with more addrspace variation. Could just replace the
existing one with this version if it's redundant.
compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake:ALL_DFSAN_SUPPORTED_ARCH
allows AArch64 but currently the instrumentation will crash.
Port Linux AArch64 memory mappings from msan but use
SizeClassAllocator64 for a slightly more efficient allocator (used by
asan/lsan). Change dfsan/lit.cfg.py to allow Linux aarch64. All tests
should pass.
* dfsan/origin_invalid.c uses x86_64 assembly. Just make it x86_64 specific.
* dfsan/interceptors.c our mallinfo interceptor takes an argument
instead of returning a struct. This does not work on AArch64 which
uses different registers for the two function types. Disable AArch64
as msan/Linux/mallinfo.cpp does.
Reviewed By: #sanitizers, vitalybuka
Differential Revision: https://reviews.llvm.org/D140770
We incorrectly assume intrinsic as a function call and it prevents us from
the opportunity to vectorize. On Aarch64 Cortex-A53 we think that
llvm.fmuladd.f64 is a function call which is wrong.
Differential Revision: https://reviews.llvm.org/D140392
D97409 added injectMetadataGlobals to differentiate the shadow mode.
This feature has been unused and is unneeded after D103745 removed fast16 mode.
Reviewed By: browneee
Differential Revision: https://reviews.llvm.org/D140797
For `srem x, y`, if abs(constant range of x) less than abs(constant
range of y), we can simplify it as:
`srem x, y => x` if y is guaranteed to be positive.
'srem x, y => -x' if y is guaranteed to be negative.
Differential Revision: https://reviews.llvm.org/D140405
~sext(A) & Op1 --> A ? 0 : Op1
With no extra uses, this pattern is already reduced,
but we would miss it in examples such as issue #59773.
https://alive2.llvm.org/ce/z/WGLcSR
EmitGEPOffset() supports vector GEPs nowadays, so we don't need
any further code changes.
compare_gep_with_base_vector1 shows a weakness in folding the
resulting comparison if an index splat has to be performed.
The way these APIs are used, there isn't really a benefit to
deduplicating the ops as part of the API. The only place that
benefits from this is PoisonChecking, and for that particular
use the assertion emission was potentially non-deterministic.
We should populate a vector for deterministic order and then
deduplicate via a separate set.
I think this doesn't make any difference right now, but once
we take into account that branch on undef is UB in
programUndefinedIfUndefOrPoison() the new position of the branch
would imply that the condition can't be poison, which would
defeat the purpose of the freeze insertion here. We need to
perform the check before the branch is moved.
Adjust mergeReplicateRegions to be in line with
mergeBlocksIntoPredecessors added in 36d70a6aea6b by collecting only the
valid candidates first.
Also rename to mergeReplicateRegionsIntoSuccessors and add missing
doc-comment.
This addresses post-commit suggestions by @Ayal.
This reduces the size of VPlan.h and avoids future growth of the file
when the graph traits are extended in future patches.
Reviewed By: Ayal
Differential Revision: https://reviews.llvm.org/D140500
r249754 (2015) added this macro (not set anywhere) for an escape hatch.
This old layout has been non-working for many years (Linux
arch/x86/include/asm/elf.h ELF_ET_DYN_BASE is outside the APP range):
FATAL: Code 0x558547327980 is out of application range. Non-PIE build?
FATAL: MemorySanitizer can not mmap the shadow memory.
FATAL: Make sure to compile with -fPIE and to link with -pie.
FATAL: Disabling ASLR is known to cause this error.
FATAL: If running under GDB, try 'set disable-randomization off'.
Non-pie doesn't work either.
As per the post-commit feedback - that was not the correct precondition
to avoid it here. I think we should generally start changing mentality
about `freeze`, the fact that we have been conditioned to be afraid of it
(or of anything in LLVM in general) is the key problem here.
Even if the the sink candidate is already in the target block, its
operands can be candidates for sinking. Queue them up as well. Also
moves the queuing logic to a helper.
This kind of thing happens really frequently in LLVM's very own
shuffle combining methods, and it is even considered bad practice
to use `%` there, instead of using this expansion directly.
Though, many of the cases there have variable divisors,
so this won't help everything.
Simple case: https://alive2.llvm.org/ce/z/PjvYf-
There's alternative expansion via `umin`:
https://alive2.llvm.org/ce/z/hWCVPb
BUT while we can transform the first expansion
into the `umin` one (e.g. for SCEV):
https://alive2.llvm.org/ce/z/iNxKmJ
... we can't go in the opposite direction.
Also, the non-`umin` expansion seems somewhat more codegen-friendly:
https://godbolt.org/z/qzjx5bqWKhttps://godbolt.org/z/a7bj1axbx
There's second variant of precondition:
https://alive2.llvm.org/ce/z/zE6cbM
but there the numerator must be non-undef / must be frozen.
