insertelement instructions.
If the original vector has undef, not poison values, which are not
rewritten by later insertelement instructions, need to transform shuffle
with the undef vector, not a poison vector, and actual indices, not
PoisonMaskElem, otherwise the transformation may produce more poisons
output than the input.
`@llvm.ptrmask` is basically just `and` with a `ptr` operand. This is
a trivial combine to do with `and` (many others could also be added).
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D154006
This avoids the need to regenerate the PGO raw profile on version
changes. Modify the update script to autogenerate new PGO proftext
inputs.
Differential Revision: https://reviews.llvm.org/D156460
As noted on #63980 rotate by immediate amounts is much cheaper than variable amounts.
This still needs to be expanded to vector rotate cases, and we need to add reasonable funnel-shift costs as well (very tricky as there's a huge range in CPU behaviour for these).
This allows us to handle dead blocks with multiple incoming edges,
where we can determine that all of those edges are dead (or cycles).
This allows InstCombine to handle certain dead code patterns that
can be produced by LoopVectorize in a single iteration.
This is in preparation for D154579.
The debug output of replaceDominatedUsesWith() prints incorrect
information, and the user is left confused about what exactly was
replaced. Fix this.
Differential Revision: https://reviews.llvm.org/D156318
Split off min-max in-loop reduction tests into separate file and extend
them by adding tests with
* min & max intrinsics
* fmuladd with permuted operands
* min & max select tests with permuted operands.
Adds extra test coverage as suggested in D155845.
This reverts commit 673a4671f3e8b7158d990f6456428175a6eac38c.
Depends on reverted commit 0cab8d20417c0e2ccc1ffc5505e080126f5de8e6.
That commit was reverted due to an LTO crash. I've put a reduced
test case here: https://github.com/llvm/llvm-project/issues/64114
This patch introduces per kernel environment. Previously, flags such as execution mode are set through global variables with name like `__kernel_name_exec_mode`. They are accessible on the host by reading the corresponding global variable, but not from the device. Besides, some assumptions, such as no nested parallelism, are not per kernel basis, preventing us applying per kernel optimization in the device runtime.
This is a combination and refinement of patch series D116908, D116909, and D116910.
Depend on D155886.
Reviewed By: jdoerfert
Differential Revision: https://reviews.llvm.org/D142569
The most straightforward extension to D150851 would involve a loop with
decreasing induction variable, with a constant start value.
iv-select-cmp.ll only contains a negative test for the decreasing
induction variable case when the start value is variable, namely
not_vectorized_select_decreasing_induction_icmp. Hence, add a test for
the most straightforward extension to D150851, in preparation to
vectorize:
long rdx = 331;
for (long i = 19999; i >= 0; i--) {
if (a[i] > 3)
rdx = i;
}
return rdx;
Differential Revision: https://reviews.llvm.org/D156152
We can improve our deduction if we stop at PHI and select instructions
and also iterate the returned values explicitly. The latter helps with
isImpliedByIR deductions.
This is a complete fix for CompleteLoadGroups introduced in
D154309. We need to check for dependency between A and every member of
the load Group of B.
This patch also fixes another miscompile seen when we incorrectly sink stores
below a depending load (see testcase in
interleaved-accesses-sink-store-across-load.ll). This is fixed by
releasing store groups correctly.
Differential Revision: https://reviews.llvm.org/D155520
Guard FoldBranchToCommonDest in SimplifyCFG with the SpeculateBlocks
flag as it can also speculate instructions.
This was split out of D155997.
Differential Revision: https://reviews.llvm.org/D156194
This reapplies the change for and, but also marks or as undesirable
at the same time. Only handling one of them can cause infinite
combine loops due to the asymmetric handling.
-----
In preparation for removing support for and/or expressions, mark
them as undesirable. As such, we will no longer implicitly create
such expressions, but they still exist.
This reverts commit eea9258648ce73507f6f85c395de978af659d498.
That commit triggered crashes in the following testcase:
$ cat reduced.c
typedef struct {
int a[8]
} b;
typedef struct {
b *c;
short d
} e;
void f() {
int g;
char *h;
e *i = f;
short j = i->d;
int a = i->c->a[0];
for (;;)
for (; g < a; g++) {
*h = j * i->d >> 8;
h++;
}
}
$ clang -target aarch64-linux-gnu -w -c -O2 reduced.c
On AIX, a libatomic supporting inline quadword atomic operations has been released, so that compatibility is not an issue now, we can enable quadword atomics by default.
Reviewed By: #powerpc, nemanjai
Differential Revision: https://reviews.llvm.org/D151312
This makes it possible to use canonicalize to perform a dynamic check
for whether denormal flushing is enabled, which will fold out when the
denormal mode is known. Previously it would only fold if denormal
flushing were known enabled.
https://reviews.llvm.org/D156107
Need to check the scalars if they can be vectorized before trying to
schedule them. It may save compile time and improve vectorization on
large functions/basic blocks.
Differential Revision: https://reviews.llvm.org/D154891
This extends foldCastedBitwiseLogic to handle the similar cases.
I have recently submitted a patch to implement a single fold like:
(A > 0) | (A < 0) -> zext (A != 0)
But it is not general enough, and some problems like
a < b & a >= b - 1 happen again.
So I generalize this fold by matching the pattern
bitwise(A >> C - 1, zext(icmp)), and replace A >> C - 1 with
zext(A < 0) here. (C is the scalar size bits of the type of A.)
Then we get bitwise(zext(A < 0), zext(icmp)), this will be folded
by original code in foldCastedBitwiseLogic, into
zext(bitwise(A < 0, icmp)). And finally, any related icmp fold will
be automatically implemented because bitwise(icmp,icmp) had been
implemented.
The proof of the correctness is obvious, because the folds below
were previously proved and implemented.
A >> C - 1 -> zext(A < 0)
bitwise(zext(A), zext(B)) -> zext(bitwise(A, B))
And the fold of this patch is the combination of folds above.
Fixes https://github.com/llvm/llvm-project/issues/63751.
Differential Revision: https://reviews.llvm.org/D154791
