SLP vectorizer passes the type of the subvector and the mask, which size
determines the size of the resulting vector. TTI should support this
pattern to improve cost estimation of the insert_subvector shuffle
pattern.
At the moment, block and edge masks are created on demand, which means
that they are inserted at the point where they are demanded and then
cached. It is possible that the mask for a block is looked up later at a
point that's not dominated by the point where the mask has been
inserted.
To avoid this, create masks up front on entry to the corresponding basic
block and leave it to VPlan simplification to remove unneeded masks.
Note that we need to create masks for all blocks, if any of the blocks
in the loop needs predication, as computing the mask of a block depends
on the masks of its predecessor.
Needed for #76090.
https://github.com/llvm/llvm-project/pull/76635
It's been more than 3 years since -pgo-instr-old-cfg-hashing was
introduced by:
commit 120e66b3418b37b95fc1dbbb23e296a602a24fa8
Author: Hiroshi Yamauchi <yamauchi@google.com>
Date: Tue Jul 28 10:09:49 2020 -0700
I don't think anyone really cares about the ability to use the old CFG
hashing at this point.
Need to limit the transformation of the VecMask by the corresponding part of the mask of SliceSize size to avoid compiler crash during further cost analysis.
When adding information for induction variables, add both unsigned and
signed constraints, with corresponding signed and unsigned
preconditions.
I believe the logic here is equally valid for signed/unsigned, we just
need to add preconditions of the same type.
Whether runtime registration is needed is not dependent on the OS but
the file format. For ELF, COFF, Mach-O or XCOFF, we can always use the
linker support. This is important for baremetal platforms such as RTOS
and UEFI platforms where there is no OS but we still don't want to use
runtime registration and rely on linker support instead.
As suggested as follow-up in
https://github.com/llvm/llvm-project/pull/72164, manage inbounds via
VPRecipeWithIRFlags.
Note that in some cases we can now preserve inbounds in a few more
cases.
With #70253 landed, selects for reduction results are explicitly used by
ComputeReductionResult and Selects can be marked as not having
side-effects again.
This reverts the revert commit 173032902c960d4d0d67b521d8c149553d8e8ba3.
The use of std::pair makes the values it holds opaque. Using classes
improves this while keeping the POD aspect of a std::pair. As a nice
addition, the "known" functions held inappropriately in the Visitor
classes can now properly reside in the value classes. :-)
This patch folds `switch(zext/sext(X))` into `switch(X)`.
The original motivation of this patch is to optimize a pattern found in
cvc5. For example:
```
%bf.load.i = load i16, ptr %d_kind.i, align 8
%bf.clear.i = and i16 %bf.load.i, 1023
%bf.cast.i = zext nneg i16 %bf.clear.i to i32
switch i32 %bf.cast.i, label %if.else [
i32 335, label %if.then
i32 303, label %if.then
]
if.then: ; preds = %entry, %entry
%d_children.i.i = getelementptr inbounds %"class.cvc5::internal::expr::NodeValue", ptr %0, i64 0, i32 3
%cmp.i.i.i.i.i = icmp eq i16 %bf.clear.i, 1023
%cond.i.i.i.i.i = select i1 %cmp.i.i.i.i.i, i32 -1, i32 %bf.cast.i
```
`%cmp.i.i.i.i.i` always evaluates to false because `%bf.clear.i` can
only be 335 or 303.
Folding `switch i32 %bf.cast.i` to `switch i16 %bf.clear.i` will help
`CVP` to handle this case.
See also
https://github.com/llvm/llvm-project/pull/76928#issuecomment-1877055722.
Compile-time impact:
http://llvm-compile-time-tracker.com/compare.php?from=7954c57124b495fbdc73674d71f2e366e4afe522&to=502b13ed34e561d995ae1f724cf06d20008bd86f&stat=instructions:u
|stage1-O3|stage1-ReleaseThinLTO|stage1-ReleaseLTO-g|stage1-O0-g|stage2-O3|stage2-O0-g|stage2-clang|
|--|--|--|--|--|--|--|
|+0.03%|+0.06%|+0.07%|+0.00%|-0.02%|-0.03%|+0.02%|
The current code for reprocessing the result of fortified libcall
simplifications is not correct, because we might simplify to an
argument of the original call, and if that is again a libcall,
mistakenly think that this is actually the simplification result.
Instead of trying to fix this, simply remove the code entirely,
because InstCombine nowadays correctly handles reprocessing of
SimplifyLibCall results.
Fixes https://github.com/llvm/llvm-project/issues/77064.
shl nsw x, shift can be interpreted as mul nsw x, (1<<shift), except
when shift is bw-1 (https://alive2.llvm.org/ce/z/vDh2xT). Use this when
decomposing shl. The equivalent decomposition for the unsigned case
already exists.
This patch tries to flip the signedness of predicates when folding an
unsigned icmp with a signed min/max. It will enable more optimizations
as we canonicalizes a signed icmp into an unsigned icmp when both
operands are known to have the same sign.
Fixes#76672.
Compile-time impact:
http://llvm-compile-time-tracker.com/compare.php?from=949ec83eaf6fa6dbffb94c2ea9c0a4d5efdbd239&to=2deca1aea8a4e13609bab72c522a97d424f0fc2d&stat=instructions:u
|stage1-O3|stage1-ReleaseThinLTO|stage1-ReleaseLTO-g|stage1-O0-g|stage2-O3|stage2-O0-g|stage2-clang|
|--|--|--|--|--|--|--|
|-0.00%|+0.01%|+0.05%|-0.12%|-0.01%|-0.03%|-0.00%|
NOTE: We can flip the signedness of predicate if both operands are
negative. But I don't see the benefit of handling these cases.
This patch introduces a new ComputeReductionResult opcode to compute the
final reduction result in the middle block. The code from fixReduction
has been moved to ComputeReductionResult, after some earlier cleanup
changes to model parts of fixReduction explicitly elsewhere as needed.
The recipe may be broken down further in the future.
Note that the phi nodes to merge the reduction result from the trip
count check and the middle block, to be used as resume value for the
scalar remainder loop are also generated based on
ComputeReductionResult.
Once we have a VPValue for the reduction result, this can also be
modeled explicitly and moved out of the recipe.
Address sanitizer checks for AMDGPU target in non-recovery mode aren't
quite efficient at the moment which can be illustrated with a program:
```
instr_before;
load ptr1;
instr_in_the_middle;
load ptr2;
instr_after;
```
ASAN generates the following instrumentation:
```
instr_before;
if (sanity_check_passed(ptr1))
load ptr1;
instr_in_the_middle;
if (sanity_check_passed(ptr2))
load ptr2;
instr_after;
else
// ASAN report block 2
__asan_report(ptr2); // wave terminates
unreachable;
else
// ASAN report block 1
__asan_report(ptr1); // wave terminates
unreachable;
```
Each sanitizer check is treated as a non-uniform condition (and this is
true because some lanes may pass the check and some don't). This results
in the program above: basically normal program flow is continued in
_then_ blocks. This way it allows lanes that pass all sanity checks to
complete the program and then the wave terminates at the first reporting
_else_ block. For each _else_ block it has to keep execmask and pointer
value to report error consuming tons (megatons!) of registers which are
live till the program end.
This patch changes the behavior on a failing sanity check: instead of
waiting when passing lanes reach program end report error and terminate
as soon as any lane has violated the sanity check. Sanity check
condition is treated uniform with this approach and the resulting
program looks much like ordinary CPU code:
```
instr_before;
if (any_lane_violated(sanity_check_passed(ptr1)))
// ASAN report block 1
__asan_report(ptr1); // abort the program
unreachable;
load ptr1;
instr_in_the_middle;
if (any_lane_violated(sanity_check_passed(ptr2)))
// ASAN report block 2
__asan_report(ptr2); // abort the program
unreachable;
load ptr2;
instr_after;
```
However it has to use a trick to pass structurizer and some later
passes: ASAN check is generated like in recovery mode but reporting
function aborts, that is standard _unreachable_ instruction isn't used:
```
...
if (any_lane_violated(sanity_check_passed(ptr1)))
// ASAN report block 1
__asan_report(ptr1); // abort the program
// pretend we're going to continue the program
load ptr1;
...
```
This may create some undesirable effects:
1. Register allocator generates a lot of code for save/restore registers
for asan_report call. This may potentially bloat the code since we have
a report block for every accessed pointer.
2. Loop invariant code in report blocks is hoisted into a loop
preheader. I'm not sure but probably this can be solved using block
frequency information, but most likely this isn't a problem at all.
These problems are to be addressed later.
### Flattening address sanitizer check
In order to simplify divergent CFG this patch also changes the
instrumentation code from:
```
uint64_t address = ptr;
sbyte *shadow_address = MemToShadow(address);
sbyte shadow_value = *shadow_address;
if (shadow_value) {
sbyte last_accessed_byte = (address & 7) + kAccessSize - 1;
if (last_accessed_byte >= shadow_value) {
ReportError(address, kAccessSize, kIsWrite);
abort();
}
}
```
to
```
uint64_t address = ptr;
sbyte *shadow_address = MemToShadow(address);
sbyte shadow_value = *shadow_address;
sbyte last_accessed_byte = (address & 7) + kAccessSize - 1;
if (shadow_value && last_accessed_byte >= shadow_value) {
ReportError(address, kAccessSize, kIsWrite);
abort();
}
```
It saves one _if_ which really avoids very few instructions and their
latency can be hidden by the load from shadow memory.
This was found in julia when we changed some operations from explicit
loads + stores to memcpys. While applying it to both the src and the
dest seems weird, thats what we do for normal TBAA.
vectorized.
When trying to reuse the extractelement instruction, emitted for the
insertelement instruction, need to check, if the this insertelement
instruction was vectorized. In this case, need to use vectorized value,
not the original insertelement.
When adding constraints for induction variables, if the step is not one,
we need to make sure that (end-start) is a multiple of step, otherwise
we might step over the end value.
Currently this only supports one specific pattern for pointers, where
the end is a gep of the start with an appropriate offset.
Generalize this by using SCEV to check for multiples, which also makes
this work for integer IVs.
Vectors are always bit-packed and don't respect the elements' alignment
requirements. This is different from arrays. This means offsets of
vector GEPs need to be computed differently than offsets of array GEPs.
This PR fixes many places that rely on an incorrect pattern
that always relies on `DL.getTypeAllocSize(GTI.getIndexedType())`.
We replace these by usages of `GTI.getSequentialElementStride(DL)`,
which is a new helper function added in this PR.
This changes behavior for GEPs into vectors with element types for which
the (bit) size and alloc size is different. This includes two cases:
* Types with a bit size that is not a multiple of a byte, e.g. i1.
GEPs into such vectors are questionable to begin with, as some elements
are not even addressable.
* Overaligned types, e.g. i16 with 32-bit alignment.
Existing tests are unaffected, but a miscompilation of a new test is fixed.
---------
Co-authored-by: Nikita Popov <github@npopov.com>
[LV] Change loops' interleave count computation
A set of microbenchmarks in llvm-test-suite (https://github.com/llvm/llvm-test-suite/pull/56), when tested on a AArch64 platform, demonstrates that loop interleaving is beneficial when the vector loop runs at least twice or when the epilogue loop trip count (TC) is minimal. Therefore, we choose interleaving count (IC) between TC/VF & TC/2*VF (VF = vectorization factor), such that remainder TC for the epilogue loop is minimum while the IC is maximum in case the remainder TC is same for both.
The initial tests for this change were submitted in PRs:
https://github.com/llvm/llvm-project/pull/70272 and https://github.com/llvm/llvm-project/pull/74689.
This patch cleans up the duplicate code for folding commutative binops
over `select/phi/minmax`.
Related commits:
+ select support:
88cc35b27e
+ phi support:
8674a023bc
+ minmax support:
624973806c
vectorized.
If the insertelement instruction is vectorized, and the extractelement
instruction from such insertelement also vectorized as part of the same
tree, need to extract from the corresponding for insertelement vectorized value rather than original insertelement instruction.
When creating a declaration for a vector variant, in order to determine
the argument types we need to consult the VFABI demangler. This will
allow us to add TLI mappings with linear arguments (see #76060).
The current patterns for NUWSub decompositions do not handle negative
constants correctly at the moment (causing #76713).
Replace the incorrect pattern by more general code that recursively
decomposes the operands and then combines the results. This is already
done in most other places that handle operators like add/mul.
This means we fall back to the general constant handling code (fixes the
mis-compile) while also being able to support reasoning about
decomposable expressions in the SUB operands.
Fixes https://github.com/llvm/llvm-project/issues/76713.
llvm/lib/IR/Type.cpp:694:
Assertion `isValidElementType(ElementType) && "Element type of a
VectorType must be an integer, floating point, or pointer type."'
failed.
Stack dump:
llvm::FixedVectorType::get(llvm::Type*, unsigned int)
llvm::VPWidenCallRecipe::execute(llvm::VPTransformState&)
llvm::VPBasicBlock::execute(llvm::VPTransformState*)
llvm::VPRegionBlock::execute(llvm::VPTransformState*)
llvm::VPlan::execute(llvm::VPTransformState*)
...
Happens with function calls of void return type.
Make the hoisted dbg.declare inherent the DILocation scope from the new
storage.
After hoisting, the dbg.declare is moved into the block that defines the
new storage. This could create an inconsistency in the debug location
scope hierarchy where the scope of hoisted dbg.declare (i.e.
DILexicalBlock) is enclosed with the scope of the block (i.e.
DISubprogram). This confuses LiveDebugValues pass to think that the
hoisted dbg.declare is killed in that block and does not generate
DBG_VALUE in other blocks. Debugger won't be able to track its value
anymore.
We do this for unoptimized binary only.
