Change getPointersDiff to return an std::optional<int64_t>, and fill
this value with using APInt::trySExtValue. This simple change requires
changes to other functions in LAA, and major changes in SLPVectorizer
changing types from 32-bit to 64-bit.
Fixes#139202.
Thread-local globals live, by default, in the default globals address
space, which may not be 0, so we need to overload @llvm.thread.pointer
to support other address spaces, and use the default globals address
space in Clang.
When we enable EVL-based loop vectorization w/ predicated tail-folding,
each vectorized loop has effectively two induction variables: one
calculates the step using (VF x vscale) and the other one increases the
IV by values returned from experiment.get.vector.length. The former,
also known as canonical IV, is more favorable for analyses as it's
"countable" in the sense of SCEV; the latter (EVL-based IV), however, is
more favorable to codegen, at least for those that support scalable
vectors like AArch64 SVE and RISC-V.
The idea is that we use canonical IV all the way until the end of all
vectorizers, where we replace it with EVL-based IV using EVLIVSimplify
introduced here. Such that we can have the best from both worlds.
This Pass is enabled by default in RISC-V. However, since we haven't
really vectorize loops with predicate tail-folding by default, this Pass
is no-op at this moment.
Don't use the order of incoming values of IR phis when creating
VPBlendRecipes. Instead, simply use the incoming operands and
blocks from the VPWidenPHIRecipe.
Note that this changes the order of the incoming operands/masks for some
blends.
PR: https://github.com/llvm/llvm-project/pull/139475
This teaches ObjCARCContract to transform attachedcall bundles
referencing objc_retainAutoreleasedReturnValue to instead reference
objc_claimAutoreleasedReturnValue.
The only distinction between the two is that the latter is required to
be guaranteed to immediately follow the call it's attached to, and, by
construction, the bundles always achieve that by:
- not being separable from the call through IR and the backend
- not getting the marker emitted when claimARV is the attachedcall.
This is enabled only for arm64, arm64e, and arm64_32 on macOS13/iOS16
and related operating systems.
Co-authored-by: Ahmed Bougacha <ahmed@bougacha.org>
We can narrow `trunc(lshr(i32)) to i8` to `trunc(lshr(i16)) to i8` even
when the bits that we are shifting in are not zero, in the cases where
the MSBs of the shifted value don't actually matter and actually end up
being truncated away.
This kind of narrowing does not remove the trunc but can help the
vectorizer generate better code in a smaller type.
Motivation: libyuv, functions like ARGBToUV444Row_C().
Proof: https://alive2.llvm.org/ce/z/9Ao2aJ
This is a patch with precommitted tests to make
https://github.com/llvm/llvm-project/pull/136630 easier to review. The
`virtual-const-prop-small-alignment-*` tests check the output when the
loaded int alignment is less than the vtable alignment.
This also changes some constants to make it easier to differentiate
between propagated values in vtables.
This teaches ObjCARCContract to transform attachedcall bundles
referencing objc_retainAutoreleasedReturnValue to instead reference
objc_claimAutoreleasedReturnValue.
The only distinction between the two is that the latter is required to
be guaranteed to immediately follow the call it's attached to, and, by
construction, the bundles always achieve that by:
- not being separable from the call through IR and the backend
- not getting the marker emitted when claimARV is the attachedcall.
This is enabled only for arm64, arm64e, and arm64_32 on macOS13/iOS16
and related operating systems.
Co-authored-by: Ahmed Bougacha <ahmed@bougacha.org>
When proving the legality of exchanging two loops, it doesn't need to
check the elements of the direction vectors associated with the loops
outside of the two target loops. Before this patch, the legality check
looked at all elements of a direction vector to calculate the
lexicographically order of the vector, which may reject some legal
exchanges. For example, if a direction vector is `[* < =]`, it is safe
to swap the last two loops because the corresponding subsequence of the
vector (`[< =]`) is lexicographically positive for both before and after
the exchange. However, the its order is unknown if we don't drop the
prefix since the first element is `*`. This patch improves the logic of
legality check to ignore such unrelated prefixes of direction vectors.
When a read(first)lane is used on a binary operator and the intrinsic is
the only user of the operator, we can move the read(first)lane into the
operand if the other operand is uniform.
Unfortunately IC doesn't let us access UniformityAnalysis and thus we
can't truly check uniformity, we have to do with a basic uniformity
check which only allows constants or trivially uniform intrinsics calls.
We can also do the same for unary and cast operators.
If we have `icmp eq or(a, shl(b)), 0` then the shift can be removed so
long as it is nuw or nsw. It is still comparing that some bits are
non-zero.
https://alive2.llvm.org/ce/z/nhrBVX.
This is also true of ne, and true for longer or chains.
Match icmps of binops where both operands are select with constant arms,
i.e., `icmp pred (select A ? C1 : C2) binop (select B ? C3 : C4), C5`.
Fold such patterns by creating a truth table of the possible four
constant variants, and materialize back the optimal logic from it via
`createLogicFromTable` helper. This also generalizes an existing fold,
which has therefore been dropped.
Proofs: https://alive2.llvm.org/ce/z/NS7Vzu.
Fixes: https://github.com/llvm/llvm-project/issues/138212.
Given a binary op on splatted vector and a splatted constant,
InstCombine will normally pull the shuffle out in
`InstCombinerImpl::foldVectorBinop`:
```llvm
define <4 x i32> @f(i32 %x) {
%x.insert = insertelement <4 x i32> poison, i32 %x, i64 0
%x.splat = shufflevector <4 x i32> %x.insert, <4 x i32> poison, <4 x i32> zeroinitializer
%res = add <4 x i32> %x.splat, splat (i32 42)
ret <4 x i32> %res
}
```
```llvm
define <4 x i32> @f(i32 %x) {
%x.insert = insertelement <4 x i32> poison, i32 %x, i64 0
%1 = add <4 x i32> %x.insert, <i32 42, i32 poison, i32 poison, i32 poison>
%res = shufflevector <4 x i32> %1, <4 x i32> poison, <4 x i32> zeroinitializer
ret <4 x i32> %res
}
```
However, this currently only operates on fixed length vectors. Splats of
scalable vectors don't currently have their shuffle pulled out, e.g:
```llvm
define <vscale x 4 x i32> @f(i32 %x) {
%x.insert = insertelement <vscale x 4 x i32> poison, i32 %x, i64 0
%x.splat = shufflevector <vscale x 4 x i32> %x.insert, <vscale x 4 x i32> poison, <vscale x 4 x i32> zeroinitializer
%res = add <vscale x 4 x i32> %x.splat, splat (i32 42)
ret <vscale x 4 x i32> %res
}
```
Having this canonical form with the shuffle pulled out is important as
VectorCombine relies on it in order to scalarize binary ops in
`scalarizeBinopOrCmp`, which would prevent the need for #137786. This
also brings it in line for scalable binary ops with two non-constant
operands: https://godbolt.org/z/M9f7ebzca
This adds a combine just after the fixed-length version, but restricted
to splats at index 0 so that it also handles the scalable case:
So the whilst the existing combine looks like: `Op(shuffle(V1, Mask), C)
-> shuffle(Op(V1, NewC), Mask)`
This patch adds: `Op(shuffle(V1, 0), (splat C)) -> shuffle(Op(V1, (splat
C)), 0)`
I think this could be generalized to other splat indexes that aren't
zero, but I think it would be dead code since only fixed-length vectors
can have non-zero shuffle indices, which would be covered by the
existing combine.
If the interleaved loads require reordering, better to avoid generate
load + shuffle sequence, which in this case cannot be recognized as
interleaved load. Also, it fixes the issue with the incorrect codegen.
Fixes#138923
By the pseudocode in the ISA manual, if any input is a nan it acts
like min3, which will fold to min2 of the other operands. The other
cases fold to min, I'm not sure how this one was wrong.
Currently when we version a loop all loads and stores have the noalias
metadata added to them. If there were some pointers that could not be
analysed, and thus we could not generate runtime aliasing checks for,
then we should not mark loads and stores using these pointers as
noalias.
This is done by getting rid of setNoAliasToLoop and instead using
annotateLoopWithNoAlias, as that already correctly handles partial alias
information. This does result in slightly different aliasing metadata
being generated, but it looks like it's more precise.
Currently this doesn't result in any change to the transforms that
LoopVersioningLICM does, as LoopAccessAnalysis discards all results if
it couldn't analyse every pointer leading to no loop versioning
happening, but an upcoming patch will change that and we need this first
otherwise we incorrectly mark some pointers as noalias even when they
aren't.
In most cases, the type information attached to load and store
instructions is meaningless and inconsistently applied. We can usually
use ".b" loads and avoid the complexity of trying to assign the correct
type. The one expectation is sign-extending load, which will continue to
use ".s" to ensure the sign extension into a larger register is done
correctly.
Adds a new option -memprof-callsite-cold-threshold that allows
specifying a percent that will cause non-cold contexts to be discarded
if the percent cold bytes at a callsite including that context exceeds
the given threshold. Default is 100% (no discarding).
This reduces the amount of cloning needed to expose cold allocation
contexts when parts of the context are dominantly cold.
This motivated the change in PR138792, since discarding a context might
require a different decision about which not-cold contexts must be kept
to expose cloning requirements, so we need to determine that on the fly.
Additionally, this required a change to include the context size
information in the alloc trie in more cases, so we now guard the
inclusion of this information in the generated metadata on the option
values.
Adds support for MSVC's undocumented `/funcoverride` flag, which marks
functions as being replaceable by the Windows kernel loader. This is
used to allow functions to be upgraded depending on the capabilities of
the current processor (e.g., the kernel can be built with the naive
implementation of a function, but that function can be replaced at boot
with one that uses SIMD instructions if the processor supports them).
For each marked function we need to generate:
* An undefined symbol named `<name>_$fo$`.
* A defined symbol `<name>_$fo_default$` that points to the `.data`
section (anywhere in the data section, it is assumed to be zero sized).
* An `/ALTERNATENAME` linker directive that points from `<name>_$fo$` to
`<name>_$fo_default$`.
This is used by the MSVC linker to generate the appropriate metadata in
the Dynamic Value Relocation Table.
Marked function must never be inlined (otherwise those inline sites
can't be replaced).
Note that I've chosen to implement this in AsmPrinter as there was no
way to create a `GlobalVariable` for `<name>_$fo$` that would result in
a symbol being emitted (as nothing consumes it and it has no
initializer). I tried to have `llvm.used` and `llvm.compiler.used` point
to it, but this didn't help.
Within LLVM I referred to this feature as "loader replaceable" as
"function override" already has a different meaning to C++ developers...
I also took the opportunity to extract the feature symbol generation
code used by both AArch64 and X86 into a common function in AsmPrinter.
The intent of this code is to split larger vectors into smaller shuffles, but
it currently triggering on some small vector types. Limit it to vectors of size
>128bit.
Flow blocks are generated code that don't really correspond to any
location in the source, so principally they should have empty DebugLocs.
Practically, setting these debug locs leads to redundant is_stmts being
generated after #108251, causing stepping test failures in the ROCm GDB
test suite.
Fixes SWDEV-502134
