This patch adds support in Clang for the PRFM IR instruction, by adding
the following builtin:
void __pldir(void const *addr);
This builtin is described in the following ACLE proposal:
https://github.com/ARM-software/acle/pull/406
This lowers the splitting as:
```
any_active(hi_mask)
? (find_last_active(hi_mask) + lo_mask.getVectorElementCount())
: find_last_active(lo_mask)
```
And trivially lowers `<1 x i1>` scalarization to returning zero. Which
is a natural result of the splitting (and the lack of a sentinel
"none-active" result value).
The lowerings likely can be improved. This patch is for completeness.
Should fix:
https://github.com/llvm/llvm-project/pull/178862#issuecomment-3862310334Fixes#180212
The target function to be checked by the Control Flow Guard Check
function is stored in `X15` on AArch64. This register is guaranteed to
be preserved by that function (on success), thus after it returns `X15`
can be used to branch to the target function instead of having to load
it from another register or the stack.
If 32-bit (or less) "v0" registers coming from inline asm are treated as
vector ones, codegen might produce incorrect vector<->scalar
conversions. This causes types mismatch assertion failures later during
compile-time. The fix treats 32-bit or less v0-v31 AArch64 registers as
scalar, along with 64-bit ones.
Fixes#153442
Adresses one of the subtasks of #150515.
The code is ported from `SelectionDAG::computeKnownBits` and tests are
loosely based on `AArch64/GlobalISel/knownbits-shl.mir`.
Profile data for instructions (e.g., branch weights) is automatically
preserved via `splice()` which moves the basic blocks along with their
instruction metadata. However, entry count is stored as function
metadata, which was dropped when creating merged function and thunks.
The fix is to explicitly set entry count for both merged function (.Tgm)
and thunks via `setEntryCount()`.
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
cttz.elts is usually lowered (for SVE) to a brkb followed by a cntp. If
we then want a mask based on that (say, for early exit masking) then we
would use a whilelo from 0 to the result of cntp. But that just gives us
the same mask as the initial brkb, so we can just remove the cntp and
the whilelo.
Brka matches the extra +1 in the pattern.
When computing the viable cycles for scheduling an instruction,
`computeStart` used to include special-case logic to handle loop-carried
dependencies. This special handling was necessary because loop-carried
dependencies were represented by reversed forward-direction edges in the
DAG. Now that we have the DDG, which explicitly models loop-carried
dependencies, this special handling is no longer required. As a first
step towards completely removing `isLoopCarriedDep`, this patch
eliminates the special-case logic from `computeStart` and some related
functions.
Split off from https://github.com/llvm/llvm-project/pull/135148
As with loads and stores, instructions that may trigger floating‑point
exceptions must not be reordered across a barrier instruction. This
patch adds the missing loop‑carried dependencies between such
instructions and the barrier, preventing reordering that could
previously occur. Same as #174391, the implementation is based on that
of `ScheduleDAGInstrs::buildSchedGraph`.
Split off from #135148
The loads/stores must not be reordered across barrier instructions.
However, in MachinePipeliner, it potentially could happen since
loop-carried dependencies from loads/stores to a barrier instruction
were not considered. The same problem exists for barrier-to-barrier
dependencies. This patch adds the handling for those cases. The
implementation is based on that of `ScheduleDAGInstrs::buildSchedGraph`.
Split off from https://github.com/llvm/llvm-project/pull/135148
This patch fixes https://github.com/llvm/llvm-project/issues/150737.
The original computed CSRCost is too small, so the optimization of
spilling instead of using CSR is rarely triggered.
Also the original cost model is too difficult to be understood and too
hard to be tuned by backend developers and users.
So this patch changes the CSRCost to be
CSRCost = TRI->getCSRFirstUseCost() * EntryFreq * Scale
TRI->getCSRFirstUseCost() is the raw cost of save/restore a CSR. Usually
we don't need to tune this number.
EntryFreq is the BlockFrequency of the entry block.
Scale is used to scale down the CSRCost, because we usually prefer a CSR
register instead of spilling if we have similar CSRCost and spill cost,
so it should be less than 100%. We usually tune this number.
Another problem is the original function RAGreedy::calcSpillCost()
actually computes a cost for block split, so this patch also implements
a correct RAGreedy::calcSpillCost() function.
This new behavior is not enabled by default. This optimization is used
by 3 targets (AArch64 / AMDGPU / RISCV), I will change them one by one
in following patches.
ptrauth intrinsic to safely construct relative ptr for swift coroutines.
A ptrauth intrinsic for swift co-routine support that allows creation of
signed pointer
from offset stored at address relative to the pointer.
Following C-like pseudo code (ignoring keys,discriminators) explains its
operation:
let rawptr = PACauth(inputptr);
return PACsign( rawptr + signextend64( *(int32*)(rawptr+addend) ))
What: Authenticate a signed pointer, load a 32bit value at offset
'addend' from pointer,
add this value to pointer, sign this new pointer.
builtin: __builtin_ptrauth_auth_load_relative_and_sign
intrinsic: ptrauth_auth_resign_load_relative
These were added recently with a fairly complex propagation step,
however, these optimizations can cause regressions in some cases.
This patch limits the cross-block optimizations to the simple case
picking a state that matches all incoming blocks. If any block doesn't
match, we fallback to using "ACTIVE", the default state.
When lowering cttz.elts for fixed-length vectors when SVE is available,
we use scalable container types for the predicate types since NEON
doesn't have dedicated predicate registers. Unfortunately, this also
discards the actual length of the vector to look at if it's shorter than a
full vector.
Example codegen, for a llvm.experimental.cttz.elts.i64.v4i1
shl v0.4h, v0.4h, 15
ptrue p0.h, vl4
ptrue p1.h
cmpne p0.h, p0/z, z0.h, #0
brkb p0.b, p1/z, p0.b
cntp x8, p0, p0.h
The 'ptrue p1.h' is where we went wrong -- if p0 is empty, we should
only set 4 lanes active at most, but since brkb's pg operand is all
active, it sets all available lanes (e.g. 8 .h lanes on a 128b SVE
implementation).
We need to use the 'vl4' mask for brkb's pg operand instead. So I've
added an extra parameter to AArch64ISD::CTTZ_ELTS to carry the
appropriate pattern from where it's lowered to ISel.
Fixes#178644
[AArch64][GlobalISel] Constrain G_CONSTANT_FOLD_BARRIER operand
Instruction selection is lowering:
bb.1:
%6:gpr(s64) = G_CONSTANT i64 457873110
...
bb.2:
%12:gpr(s64) = G_CONSTANT_FOLD_BARRIER %6
%24:gpr(s64) = G_CONSTANT i64 0
%13:gpr(s64) = G_AND %24, %12
...
to:
%13:gpr64 = ANDXrr %24:gpr64, %6:gpr64sp'
which is causing the verifier to fail with:
Expected a GPR64 register, but got a GPR64sp register
the reason for this is there are multiple G_CONSTANT_FOLD_BARRIER ops
with the same input register:
Select: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr
Erasing: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
--
Select: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
Erasing: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64
--
Select: %7:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64
Erasing: %7:gpr64sp(s64) = G
and the middle one used by the AND gets overridden by last and becomes
invalid.
this patch fixes this by constraining the register class such that the
middle one gets constrained to the subset of gpr64sp and gpr64, i.e.
gpr64common:
Select: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr
Erasing: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
--
Select: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
Erasing: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64common
--
Select: %7:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64common
Erasing: %7:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64common
canReplaceReg doesnt handle when both regs already have classes so the
assert is removed.
Fixes#166563.
Currently performBuildShuffleExtendCombine only supports ANY_EXTEND
operands for BUILD_VECTOR inputs, and will bail if it encounters a
VECTOR_SHUFFLE with ANY_EXTEND operands. Update the logic so that we
support shuffles with ANY_EXTEND operands, which brings the code in line
with the comment.
We can, when attempting to lower to tbz, skip a zext that is then not
accounted for elsewhere. The attached test ends up with a tbz from an
extract that then does not properly zext the value extracted from the
vector. This patch fixes that by only looking through a G_ZEXT if the
bit checked is in the low part of the value, lining up the code with the
comment.
Fixes#173895
Preserve, PDT, BPI, LazyBPI and LazyBFI. These are all IR analysis that
are not invalidated by machine passes.
This partially mitigates the compile-time regression from
https://github.com/llvm/llvm-project/pull/174746.
This patch converts AArch64 CLS intrinsics (aarch64_neon_cls) to use the
generic ISD::CTLS node.
- aarch64_neon_cls: Lowered to ISD::CTLS, pattern-matched to CLS
instruction
- Set ISD::CTLS as Legal for NEON vector types (v8i8, v16i8, v4i16,
v8i16, v2i32, v4i32)
Also adds generic CTLS expansion support:
- ExpandIntRes_CTLS in LegalizeIntegerTypes for i64->i32 type expansion
- expandCTLS in TargetLowering for targets without native CLS
instruction
Part of: https://github.com/llvm/llvm-project/issues/174337
This patch extends `BuildVectorSDNode::isConstantSequence` to recognize
constant sequences that contain undef elements at any position.
The new implementation finds the first two non-undef constant elements,
computes the stride from their difference, then verifies all other
defined elements match the sequence. This enables SVE's INDEX
instruction to be used in more cases.
This change particularly benefits ZIP1/ZIP2 patterns where one operand
is a constant sequence. When a smaller constant vector like `<0, 1, 2,
3>` is used in a ZIP1 shuffle producing a wider result, it gets expanded
with trailing undefs. Similarly, for ZIP2 patterns, the DAG combiner
transforms the constant to have leading undefs since ZIP2 only uses the
upper half of its operands.
In particular, these patterns arise naturally from `VectorCombine`'s
`compactShuffleOperands` optimization (see #176074) that I am suggesting
as a fix for #137447.
This is a reland of #172523.
The original patch caused an assertion failure on RISC-V because it
attempted to create a bitcast from an illegal type (i32 on RV64) during
the post-type-legalization DAGCombine stage.
Added a `TLI.isTypeLegal(Val.getValueType())` check to ensure we only
proceed with the bitcast STLF optimization when the source value's type
is legal for the target.
This patch introduces support for Store-to-Load Forwarding (STLF) in
`DAGCombiner::ForwardStoreValueToDirectLoad` when the store and load
have **different types but equal memory size** (e.g., storing an `i32`
then loading a `float` from the same location).
### What this patch does:
**Enables Optimization:** It allows for the safe forwarding of the
stored value as a Bitcast when the value is:
* A **Constant** (`ConstantSDNode`, `ConstantFPSDNode`,
`ConstantPoolSDNode`).
* **Undef**.
* And the memory sizes (`LdMemSize` == `StMemSize`) match.
### Scope and Next Steps:
This patch **only implements forwarding for constant and undef values
that has the same memory size** so far.
**I am submitting this initial patch to get early review feedback on the
core logic and fix the immediate crashes before tackling the more
complex scenarios.**
For the simple case:
```llvm
; Case Handled by this PR so far (e.g., zeroinitializer is a constant)
define float @test_stlf_integer(ptr %p, float %v) {
store i32 0, ptr %p, align 4
%f = load float, ptr %p, align 4
; ...
}
```
Fixes: #151683
This can be seen as a special-case of the existing fptoi(fmul(a, 2.0))
combine that will be moved to tablegen, which can generate a FCVT_shift
instruction.
This change improves memset code generation for non-zero values on
AArch64 by using NEON's DUP instruction instead of
the less efficient multiplication with 0x01010101 pattern.
For small sizes, the value is extracted from a larger DUP. For
non-power-of-two sizes, overlapping stores are used in some cases.
TargetLowering::findOptimalMemOpLowering is modified to allow explicitly
specifying the size of the constant in cases where the constant is
larger than the store operations.
Fixes#165949
This PR adds the combine rule `fold (add x, shl(0 - y, n)) -> sub(x,
shl(y, n))` to GlobalISel, corresponding to an existing SelectionDAG
combine in
[DAGCombiner::visitADDLikeCommutative](fcba304010/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (L3367)).
Co-authored-by: Sarah Kuhn <sarahlinhkuhn@gmail.com>
---------
Co-authored-by: Sarah Kuhn <sarahlinhkuhn@gmail.com>
Co-authored-by: Luisa Cicolini <48860705+luisacicolini@users.noreply.github.com>
This PR converts two DAGCombiner fold patterns to GlobalISel MIR
patterns:
- `((A + (B - C)) - B) -> A - C`
- `((A - (B - C)) - C) -> A - B`
Original patterns:
5b4811eddb/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (L4359)
The patterns do not include hasOneUse checks since the instruction count
remains the same even when both intermediate results have other uses (3
ops → 3 ops).
Currently, when SDAG is run on AArch64 and an `optnone` function is
encountered, the selector is chosen as FastISel. AArch64 makes use of
GlobalISel at O0 and this patch aims to align `optnone` with this
functionality.
A flag is exposed to enable this functionality for a given backend but,
as AArch64 is currently the only backend I could find using GlobalISel
at O0 this is the only one with it implemented. This flag is set when
the target supports GlobalISel & GlobalISel hasn't been forced by the
user, the target machine or by being at an optimisation level lower than
`EnableGlobalISelAtO`.
If this happens, the GlobalISel passes are included as shown in
`llvm/test/CodeGen/AArch64/O3-pipeline.ll` and skipped by IRTranslator
for functions not marked as `optnone`.
In updating the tests based on this functionality, I found some unused
check lines or run lines that mixed SDAG with GlobalISel pass names
which have been fixed.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Since #145322 relaxed the SME predicate for the multi-register while
instructions, these instructions are allowed in non-streaming mode
when SME2 is available.
This patch removes the isStreaming() restriction from both
performActiveLaneMaskCombine & ReplaceGetActiveLaneMaskResults,
allowing the whilelo.x2 intrinsic to be used if SVE or streaming
SVE is available.
Following on from https://github.com/llvm/llvm-project/pull/172484 I
have added support to tryToFoldExtOfLoad for looking through freezes, in
order to catch more cases of extending loads. This type of code is
sometimes seen being generated by the loop vectoriser. For now I've
limited this to cases where the load is only used by the freeze, since
otherwise it leads to worse code in some X86 tests.
Darwin reserves slot 231 for storing a pointer to the history ring
buffer. It also uses bits 60-62 to store the size of the ring buffer
rdar://168176496
Add infrastructure to identify conditional branches on values loaded from
memory. Such branches are likely to be harder to predict accurately since
branch history (probably) provides little useful information.
This analysis walks the def-use chain from the branch condition to find
loads that feed into it. Several cases are excluded from consideration:
- Loads from constant pools (predictable values)
- Dereferenceable invariant loads (loop-invariant)
- Branches with biased probability (null checks, etc.)
- Loads not "close in program time" to the branch (must be in the same
basic block with no intervening calls)
The analysis is disabled by default behind -enable-early-ifcvt-data-dependent.
