This adds the analogous metadata to the nofpclass attribute
to assert values are not a certain set of floating-point classes.
This allows the same information to be expressed if a function
argument is passed indirectly. This matches the bitmask encoding
of nofpclass.
I also think this should be allowed for stores to symmetrically handle
sret, but leave that for later.
Alternatively we could add a more expressive !fprange metadata,
but that would be much more complex. It's useful to match the attribute,
and more annotations can always be added.
Fixes#133560
Previously, `canReplacePointersIfEqual` unconditionally returned
`true` for vectors of pointers (e.g., `<2 x ptr>`) because it only
checked for scalar pointer types.
This resulted in a failure to perform appropriate verification for
these types. This patch fixes the logic to ensure they are properly
validated.
Fixes https://github.com/llvm/llvm-project/issues/174045
This changes LangRef to specify that pointer icmp only compares the
address bits of the pointers. That is, `icmp pred %a, %b` is equivalent
to `icmp pred ptrtoaddr(%a), ptrtoaddr(%b)`.
Similarly, it specifies that the `nonnull` attribute requires that the
address bits are non-zero.
There are a couple of motivations for this:
* For inequality comparisons, this is really the only sensible
semantics. Relational comparison of address and metadata bits as a
single integer is generally meaningless (unless the metadata bits are
equal).
* This matches (as far as I understand) the behavior of existing CHERI
implementations.
* LLVM can only reason about the address bits. These semantics allow
pointers with non-address bits to receive essentially the same
comparison optimization support as ordinary pointers.
In terms of implementation, this PR adjusts:
* The AMDGPULowerBufferFatPointers pass.
* An InstCombine fold that may replace pointers with different
non-address bits.
* The fold that replaces pointers based on dominating pointer equality.
It does not adjust:
* ISel, because we don't have in-tree targets where we can show a
difference.
* Various icmp+ptrtoint transforms, because we'll have to change this
code for ptrtoaddr optimization support anyway, and these changes are
tightly related.
Related discussion starting from:
https://discourse.llvm.org/t/clarifiying-the-semantics-of-ptrtoint/83987/60?u=nikic
This patch provides an approximation of the memory locations touched by
`llvm.matrix.column.major.load` and `llvm.matrix.column.major.store`,
enabling dead store elimination and GVN to remove redundant loads and
dead stores.
PR: https://github.com/llvm/llvm-project/pull/163368
This patch adds several tests identifying potential opportunities for
eliminating dead stores and redundant loads when using the
`llvm.matrix.column.major.store.*` and `llvm.matrix.column.major.load.*`
intrinsics.
PR: https://github.com/llvm/llvm-project/pull/163573
The `masked.load`, `masked.store`, `masked.gather` and `masked.scatter`
intrinsics currently accept a separate alignment immarg. Replace this
with an `align` attribute on the pointer / vector of pointers argument.
This is the standard representation for alignment information on
intrinsics, and is already used by all other memory intrinsics. This
means the signatures now match llvm.expandload, llvm.vp.load, etc.
(Things like llvm.memcpy used to have a separate alignment argument as
well, but were already migrated a long time ago.)
It's worth noting that the masked.gather and masked.scatter intrinsics
previously accepted a zero alignment to indicate the ABI type alignment
of the element type. This special case is gone now: If the align
attribute is omitted, the implied alignment is 1, as usual. If ABI
alignment is desired, it needs to be explicitly emitted (which the
IRBuilder API already requires anyway).
fixes#161754
When the GVN pass calls `PerformLoadPRE` or `processNonLocalLoad` it can
invoke the `SSAUpdater` which adds a phi node for our tokenLike type. If
we check for if the load is on a token like type at the `processLoad` we
can cover both cases. This is because if we don't GVN will use the
SSAUpdater to insert a phi node to reduce duplicate resource.getpointer
calls.
Because in an earlier commit:
01c0a8409a
we made the verifier error with `PHI nodes cannot have token type!`
This test case will fail today if we try to perform this load
optimization
https://godbolt.org/z/xM69fY8zM
This will impact clang aswell because `isTokenLikeTy` also checks for
`isTokenTy` Clang is likely also failing validation with token types but
just doesn't have a test case because the validator would error if it
were in a phi node.
GVN currently has two different implementation of equality propagation.
One of them is used for branch conditions (dominating an edge), which
performs replacements across multiple blocks. This is also used for
assumes to handle uses outside the current block.
However, uses inside the block are handled using a completely separate
implementation, which involves populating a replacement map and then
checking it for individual instructions during normal GVN. While this
approach generally makes sense, it is kind of pointless if we already do
a use walk to handle the cross-block case anyway.
This PR generalizes propagateEquality() to accept either a
BasicBlockEdge or an Instruction* and replace dominated users. This
removes the need for special handling of uses in the same block for
assumes, as they're covered by instruction dominance, and ensures that
both implementations do not go out of sync.
Tests have been regenerated to favour UTC-emitted lines. While
this may not be strictly necessary – as the final migration to
MemorySSA should boil down to a simple flip –, adopting UTC-based
lines should overall make tests more robust as well as easier to
diff (at least on the majority of the tests instances where the new
lines have been added). Crash tests now feature check lines as well
(notably, pr17852.ll, pr24397.ll and unreachable_block_infinite_loop.ll).
Pre-2010 tests leveraging llvm-dis have not been updated.
This patch teaches GVN how to eliminate redundant masked loads and
forward previous loads or instructions with a select. This is possible
when the same mask is used for masked stores/loads that write to the
same memory location
fixes#154407
In HLSL the GVNPass was adding a phi node on
a target extention type.
https://hlsl.godbolt.org/z/sc14YenEe
This is something we cleaned up in a past PR
(https://github.com/llvm/llvm-project/pull/154620) by introducing
`isTokenLikeTy`. In the case of the GVN pass the target extention type
was still making its way through. This change makes it so if we see this
type we don't do PRE.
Now that #149310 has restricted lifetime intrinsics to only work on
allocas, we can also drop the explicit size argument. Instead, the size
is implied by the alloca.
This removes the ability to only mark a prefix of an alloca alive/dead.
We never used that capability, so we should remove the need to handle
that possibility everywhere (though many key places, including stack
coloring, did not actually respect this).
If we have a known `p == p2` equality, we cannot replace `p2` with `p`
unless they are known to have the same provenance. GVN handles this when
propagating equalities from conditions, but not for assumes, as these go
through a different code path for uses in the same block.
Call canReplacePointersInUseIfEqual() before performing the replacement.
This is subject to the usual approximations (e.g. that we always allow
replacement with a dereferenceable constant and null).
This restriction does not appear to have any impact in practice.
Look through `not` when propagating equalities in GVN. Usually these
will be canonicalized away, but they may be retained due to multi-use or
involvement in logical expressions.
Fixes https://github.com/llvm/llvm-project/issues/143529.
There is a larger problem here in that we should not be performing
arbitrary pointer replacements for assumes. This is handled for
branches, but assume goes through a different code path.
Fixes https://github.com/llvm/llvm-project/issues/151785.
lifetime.start and lifetime.end are primarily intended for use on
allocas, to enable stack coloring and other liveness optimizations. This
is necessary because all (static) allocas are hoisted into the entry
block, so lifetime markers are the only way to convey the actual
lifetimes.
However, lifetime.start and lifetime.end are currently *allowed* to be
used on non-alloca pointers. We don't actually do this in practice, but
just the mere fact that this is possible breaks the core purpose of the
lifetime markers, which is stack coloring of allocas. Stack coloring can
only work correctly if all lifetime markers for an alloca are
analyzable.
* If a lifetime marker may operate on multiple allocas via a select/phi,
we don't know which lifetime actually starts/ends and handle it
incorrectly (https://github.com/llvm/llvm-project/issues/104776).
* Stack coloring operates on the assumption that all lifetime markers
are visible, and not, for example, hidden behind a function call or
escaped pointer. It's not possible to change this, as part of the
purpose of lifetime markers is that they work even in the presence of
escaped pointers, where simple use analysis is insufficient.
I don't think there is any way to have coherent semantics for lifetime
markers on allocas, while also permitting them on arbitrary pointer
values.
This PR restricts lifetimes to operate on allocas only. As a followup, I
will also drop the size argument, which is superfluous if we always
operate on an alloca. (This change also renders various code handling
lifetime markers on non-alloca dead. I plan to clean up that kind of
code after dropping the size argument as well.)
In practice, I've only found a few places that currently produce
lifetimes on non-allocas:
* CoroEarly replaces the promise alloca with the result of an intrinsic,
which will later be replaced back with an alloca. I think this is the
only place where there is some legitimate loss of functionality, but I
don't think this is particularly important (I don't think we'd expect
the promise in a coroutine to admit useful lifetime optimization.)
* SafeStack moves unsafe allocas onto a separate frame. We can safely
drop lifetimes here, as SafeStack performs its own stack coloring.
* Similar for AddressSanitizer, it also moves allocas into separate
memory.
* LSR sometimes replaces the lifetime argument with a GEP chain of the
alloca (where the offsets ultimately cancel out). This is just
unnecessary. (Fixed separately in
https://github.com/llvm/llvm-project/pull/149492.)
* InferAddrSpaces sometimes makes lifetimes operate on an addrspacecast
of an alloca. I don't think this is necessary.
`combineMetadata` helper currently drops `!nosanitize` metadata when
merging two instructions, even if both originally carried `!nosanitize`.
This is problematic because `!nosanitize` is a key mechanism used by
sanitizer (e.g., ASan) to suppress instrumentation. Removing it can lead
to unintended sanitizer behavior.
This patch adds `nosanitize` to the whitelist in combineMetadata,
preserving it only if both instructions carry `!nosanitize`; otherwise,
it is dropped. This patch also adds corresponding tests in a test file
and regenerates it.
---
### Details
**Example (see [Godbolt](https://godbolt.org/z/83P5eWczx) for
details**):
```llvm
%v1 = load i32, ptr %p, !nosanitize
%v2 = load i32, ptr %p, !nosanitize
```
When merged via `combineMetadata(%v1, %v2, ...)`, the resulting
instruction loses its `!nosanitize` metadata.
Tools such as UBSan and AFL rely on `nosanitize` to prevent unwanted
transformations or checks. However, the current implementation of
combineMetadata mistakenly drops !nosanitize. This may lead to
unintended behavior during optimization.
For example, under `-fsanitize=address,undefined -O2`, IR emitted by
UBSan may lose its `!nosanitize` metadata due to the incorrect metadata
merging in optimization. As a result, ASan could unexpectedly instrument
those instructions.
> Note: due to the current UBSan handlers having relatively
coarse-grained attributes, this specific case is difficult to reproduce
end-to-end from source code—UBSan currently inhibits such optimizations
(refer to #135135 for details).
Still, I believe it's necessary to fix this now, to support future
versions of UBSan that might allow such optimizations, and to support
third-party tools (such as AFL-based fuzzers) that rely on the presence
of !nosanitize.
Update the AA CaptureAnalysis providers to return CaptureComponents, so
we can distinguish between full provenance and read-only provenance
captures.
Use this to restrict "other" memory effects on call from ModRef to Ref.
Ideally we would also apply the same reasoning for escape sources, but
the current API cannot actually convey the necessary information (we can
only say NoAlias or MayAlias, not MayAlias but only via Ref).
This patch adds to GVN's `propagateEquality()` to reason about equality
constraints through `trunc nuw iN to i1`.
Given:
%tr = trunc nuw iN %v to i1
We can deduce that if `%tr == true`, then `%v == 1`, and if `%tr ==
false`, then `%v == 0`. This is valid because `nuw` guarantees that
truncation didn't lose unsigned bits, so `%v` must have been either 0 or
1.
The patch adds logic to propagate this information via the GVN worklist.
This enables further simplification opportunities downstream, such as
folding redundant stores or conditionals that depend on `%v`.
Includes a test case in `GVN/trunc-nuw-equality.ll`.
Resolves#142744
Preserve branch weight metadata when merging instructions if one of the
instructions is missing metadata. This is similar in behaviour to what
we do today for other types of metadata such as mmra, memprof and
callsite metadata.
Also add a legality check when merging prof metadata based on
instruction type. Without this check GVN PRE optimizations result in
prof metadata on phi nodes which break the module verifier.
Build failure caught by
https://lab.llvm.org/buildbot/#/builders/113/builds/6621
```
!9185 = !{!"branch_weights", i32 3912, i32 802}
Wrong number of operands
!9185 = !{!"branch_weights", i32 3912, i32 802}
fatal error: error in backend: Broken module found, compilation aborted!
```
Reverts #134200 with additional changes.
