The new mask represents the order, not the mask itself. At first, need
to treat as the order, convert to mask and only after that reorder
gathered scalars to build correct clustered order.
Differential Revision: https://reviews.llvm.org/D141161
Prior to this patch, variadic DIExpressions (i.e. ones that contain
DW_OP_LLVM_arg) could only be created by salvaging debug values to create
stack value expressions, resulting in a DBG_VALUE_LIST being created. As of
the previous patch in this patch stack, DBG_INSTR_REF's syntax has been
changed to match DBG_VALUE_LIST in preparation for supporting variadic
expressions. This patch adds some minor changes needed to allow variadic
expressions that aren't stack values to exist, and allows variadic expressions
that are trivially reduceable to non-variadic expressions to be handled
similarly to non-variadic expressions.
Reviewed by: jmorse
Differential Revision: https://reviews.llvm.org/D133926
Non-throwing inline asm infers the nounwind attribute in
instcombine. Thus, it can be handled in the same manner as
non-throwing target functions are generally. Further special casing is
unnecessary complexity.
Need to include the cost of the initial insertelement to the cost of the
broadcasts. Also, need to adjust the cost of the gather/buildvector if
the element is inserted into poison/undef vector.
Differential Revision: https://reviews.llvm.org/D140498
This reverts commit 7f0de9573f758f5f9108795850337a5acbd17eef.
This is missing handling for !isReachableFromEntry() blocks, which
may be relevant for some callers. Revert for now.
This is a recurring pattern: We want to find the nearest common
dominator (instruction) for two instructions, but currently only
provide an API for the nearest common dominator of two basic blocks.
Add an overload that accepts and return instructions.
Similar to D138404, we were not guarding against extra uses of the Mul.
In most cases other checks would catch the issue due to unsupported
instructions in the outer loop, but certain non-canonical loop forms
could still get through.
Fixes#59339
Differential Revision: https://reviews.llvm.org/D141114
When fetching allocation sizes, we almost always want to have the
size in bytes, but we were only providing an InBits API. Also add
the corresponding byte-based conjugate to save some *8 and /8
juggling everywhere.
This patch adds metadata to disable runtime unrolling to the vectorized
loop. If runtime unrolling/interleaving is considered profitable, LV
will interleave the loop directly. There should be no need to perform
runtime unrolling at a later stage.
Note that we already add metadata to disable runtime unrolling to the
scalar loop after vectorization.
The additional unrolling unnecessarily increases code size and compile
time. In addition to that we have several bug reports of unncessary
runtime unrolling for vectorized loops, e.g. PR40961
Compile-time improvements:
NewPM-O3: -1.04%
NewPM-ReleaseThinLTO: -0.59%
NewPM-ReleaseLTO-g: -0.97%
https://llvm-compile-time-tracker.com/compare.php?from=ce1be13a868d0f8afa367975558c1a6175cce33a&to=78bc2e67f22e9e10e61cdb6cdac4bb857d95eb1b&stat=instructions:uFixes#40306.
Reviewed By: lebedev.ri, nikic
Differential Revision: https://reviews.llvm.org/D115261
NFC-ish. There is a functional change but the outputs are semantically
identical. Where we might've before replaced one operand with undef (which
means "this is a kill location marker") the use of `setKillLocation` will
replace all location operands with `undef` (which also means "this is a kill
location marker").
Related to https://discourse.llvm.org/t/auto-undef-debug-uses-of-a-deleted-value
Reviewed By: StephenTozer
Differential Revision: https://reviews.llvm.org/D140904
Make a separate routine for GEPs cost calculation and make
the approach uniform across load, store and GEP tree nodes.
Additional issue fixed is GEP cost savings were applied twice
for ScatterVectorize nodes (aka gather load) making them look
unrealistically profitable for vectorization.
Differential Revision: https://reviews.llvm.org/D140789
Use deduction guides instead of helper functions.
The only non-automatic changes have been:
1. ArrayRef(some_uint8_pointer, 0) needs to be changed into ArrayRef(some_uint8_pointer, (size_t)0) to avoid an ambiguous call with ArrayRef((uint8_t*), (uint8_t*))
2. CVSymbol sym(makeArrayRef(symStorage)); needed to be rewritten as CVSymbol sym{ArrayRef(symStorage)}; otherwise the compiler is confused and thinks we have a (bad) function prototype. There was a few similar situation across the codebase.
3. ADL doesn't seem to work the same for deduction-guides and functions, so at some point the llvm namespace must be explicitly stated.
4. The "reference mode" of makeArrayRef(ArrayRef<T> &) that acts as no-op is not supported (a constructor cannot achieve that).
Per reviewers' comment, some useless makeArrayRef have been removed in the process.
This is a follow-up to https://reviews.llvm.org/D140896 that introduced
the deduction guides.
Differential Revision: https://reviews.llvm.org/D140955
The validation of vplans could fail if an inloop reduction was created
with a block-in mask that did not dominate the reduction. This makes
sure that the insert point is set when creating the mask, to ensure it
dominates the reduction.
Differential Revision: https://reviews.llvm.org/D141003
Assuming that collecting frame allocas doesn't modify CFG we can safely
move DominatorTree construction outside loop and avoid expensive computations.
Differential Revision: https://reviews.llvm.org/D140818
Fixes https://github.com/llvm/llvm-project/issues/58565
The previous implementation visits operands in pre-order, but this does
not guarantee an instruction is visited before its uses. This can cause
instructions to be copied in the incorrect order. For example:
```
a = ...
b = add a, 1
c = add a, b
d = add b, a
```
Pre-order visits does not guarantee the order in which `a` and `b` are
visited. LoopUnrollAndJam may incorrectly insert `b` before `a`.
This patch implements post-order visits. By visiting dependencies first,
we guarantee that an instruction's dependencies are visited first.
Differential Revision: https://reviews.llvm.org/D140255
Switch simplification could sometimes fail to notice when an
intermediate case removal caused the switch condition to become
constant. This would cause the switch to be simplified into a
conditional branch rather than a direct branch.
Most of the time this didn't matter, except that occasionally
downstream parts of SimplifyCFG expect tautological branches to
already have been eliminated. The missed handling in switch
simplification would cause an assertion failure in the downstream
code.
Triggering the assertion failure is fairly sensitive to the exact
order of various simplifications.
Fixes https://github.com/llvm/llvm-project/issues/59768
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D140831
This reverts commit aa2414729ebbcb2d8f162e9002a3a6aa768b1f9d.
Previously-valid IR from a tensorflow test case (as shown on the
Diffusion revision for aa2414729ebbcb2d8f162e9002a3a6aa768b1f9d) started
hanging in the loop-vectorize pass. Reverting to keep everyone working.
analysis.
Missed the analysis of the shuffle mask when trying to analyze the
operands of the shuffle instruction during peeking through shuffle
instructions.
We shouldn't penalize instructions that have extra flags.
Drop the poison-generating flags if needed instead of bailing out.
This makes canonicalization/optimization more uniform.
There is a chance that dropping flags will cause some
other transform to not fire, but we added a preliminary
patch to avoid that with:
f0faea571403
See D140665 for more details.
The llvm.lifetime.start intrinsic guarantees that the address for a
given alloca is always the same. So variables with escaped addresses
reaching reaching a lifetime start/end block before and after a suspend
must be placed onto the coroutine frame even if the variable itself
is not alive across the suspend point.
This computes a new `LoopKill` flag in the suspend crossing data flow
anaysis to catch the case where a lifetime marker can reach itself
via suspend-crossing path.
This fixes https://llvm.org/PR52501
Differential Revision: https://reviews.llvm.org/D140231
Once the constraint system grows too large in terms of number of rows,
queries can become very slow. This patch adds a new option to limit the
number of rows tracked.
The python script below can be used to generate worst-case IR with a
chain of conditional branches with N branches.
With this limit, we get the following runtimes:
* python3 generate.py 100: 0.1s
* python3 generate.py 1000: 2s
* python3 generate.py 10000: 4s
Without the limit, the case with 1000 chained conditions takes 20+
seconds.
generate.py:
import sys
N = int(sys.argv[1])
args = []
checks = []
for i in range(0, N):
args.append('i32 %l{}'.format(i))
checks.append("""
bb{0}:
%c{0} = icmp uge i32 %l{0}, 100
br i1 %c{0}, label %bb{1}, label %exit
""".format(i, i+1))
print("""
define i1 @foo({0}) {{
{1}
bb{2}:
%c{2} = icmp uge i32 %l0, 100
ret i1 %c{2}
exit:
ret i1 false
}}
""".format(' ,'.join(args), '\n'.join(checks), N))
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D140926
HandlePassthroughUser may sometimes create a new entry for the OffsetInfo of a
user in the OffsetInfoMap. This can invalidate outstanding references into the
map, including the one which needs to be copied into the new entry. This
produces invalid offset info that can trigger assertions.
Fixed this by not using references at this point. The bug was originally
introduced in commit ID 0dc0a441323d41b4860668f38d290579e0de130c.
Reviewed By: ronlieb
Differential Revision: https://reviews.llvm.org/D140837
Using the function's address space makes no sense. Copied from the
existing test, with more addrspace variation. Could just replace the
existing one with this version if it's redundant.
compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake:ALL_DFSAN_SUPPORTED_ARCH
allows AArch64 but currently the instrumentation will crash.
Port Linux AArch64 memory mappings from msan but use
SizeClassAllocator64 for a slightly more efficient allocator (used by
asan/lsan). Change dfsan/lit.cfg.py to allow Linux aarch64. All tests
should pass.
* dfsan/origin_invalid.c uses x86_64 assembly. Just make it x86_64 specific.
* dfsan/interceptors.c our mallinfo interceptor takes an argument
instead of returning a struct. This does not work on AArch64 which
uses different registers for the two function types. Disable AArch64
as msan/Linux/mallinfo.cpp does.
Reviewed By: #sanitizers, vitalybuka
Differential Revision: https://reviews.llvm.org/D140770
We incorrectly assume intrinsic as a function call and it prevents us from
the opportunity to vectorize. On Aarch64 Cortex-A53 we think that
llvm.fmuladd.f64 is a function call which is wrong.
Differential Revision: https://reviews.llvm.org/D140392
D97409 added injectMetadataGlobals to differentiate the shadow mode.
This feature has been unused and is unneeded after D103745 removed fast16 mode.
Reviewed By: browneee
Differential Revision: https://reviews.llvm.org/D140797
For `srem x, y`, if abs(constant range of x) less than abs(constant
range of y), we can simplify it as:
`srem x, y => x` if y is guaranteed to be positive.
'srem x, y => -x' if y is guaranteed to be negative.
Differential Revision: https://reviews.llvm.org/D140405
