This patch prioritizes lowering to `stnp` over `st2` store instructions
marked !nontemporal.
From performance perspective, we should conservatively prioritize STNP
lowering for non-temporal stores, because currently NT stores requires
explicit usage of `__builtin_nontemporal_store()` intrinsic, so I think
its reasonable to assume the developer explicitly intends to optimize
D-cache usage of some hot non-temporal execution. He can rollback if it
doesnt help.
The cost here is it adds a few instructions for code size (thus we
predicate when not optimizing for code size), few extra fast
instructions to execute, few extra short dep chains - should be commonly
handled by OOO execution, I-cache alignment effects, few extra
registers. In the future we can may be able to approximate a cost model
to select by.
The patch implements an AArch64 specific static function to model what
NT stores are directly legal on the backend currently, and
`AArch64TargetLowering::lowerInterleavedStore` to conditionally skip st2
lowering.
Rotations that implement reverse operations, for example:
```c
uint64x2_t foo(uint64x2_t r) {
return (r >> 32) | (r << 32);
}
```
Are currently lowered as XAR (when available):
```gas
foo:
movi v1.2d, #0000000000000000
xar v0.2d, v0.2d, v1.2d, #32
ret
```
This is suboptimal as REV* instructions typically have higher throughput
than XAR and do not require the zero operand.
This patch combines half-rotations to Neon or SVE REVs so that they're
no longer selected as XAR.
This change turns the `"nooutline"` attribute into an enum attribute
called `nooutline`, and adds an auto-upgrader for bitcode to make the
same change to existing IR.
This IR attribute disables both the Machine Outliner (enabled at Oz for
some targets), and the IR Outliner (disabled by default).
This patch adds support in Clang for the PRFM IR instruction, by adding
the following builtin:
void __pldir(void const *addr);
This builtin is described in the following ACLE proposal:
https://github.com/ARM-software/acle/pull/406
This lowers the splitting as:
```
any_active(hi_mask)
? (find_last_active(hi_mask) + lo_mask.getVectorElementCount())
: find_last_active(lo_mask)
```
And trivially lowers `<1 x i1>` scalarization to returning zero. Which
is a natural result of the splitting (and the lack of a sentinel
"none-active" result value).
The lowerings likely can be improved. This patch is for completeness.
Should fix:
https://github.com/llvm/llvm-project/pull/178862#issuecomment-3862310334Fixes#180212
The target function to be checked by the Control Flow Guard Check
function is stored in `X15` on AArch64. This register is guaranteed to
be preserved by that function (on success), thus after it returns `X15`
can be used to branch to the target function instead of having to load
it from another register or the stack.
If 32-bit (or less) "v0" registers coming from inline asm are treated as
vector ones, codegen might produce incorrect vector<->scalar
conversions. This causes types mismatch assertion failures later during
compile-time. The fix treats 32-bit or less v0-v31 AArch64 registers as
scalar, along with 64-bit ones.
Fixes#153442
Adresses one of the subtasks of #150515.
The code is ported from `SelectionDAG::computeKnownBits` and tests are
loosely based on `AArch64/GlobalISel/knownbits-shl.mir`.
Profile data for instructions (e.g., branch weights) is automatically
preserved via `splice()` which moves the basic blocks along with their
instruction metadata. However, entry count is stored as function
metadata, which was dropped when creating merged function and thunks.
The fix is to explicitly set entry count for both merged function (.Tgm)
and thunks via `setEntryCount()`.
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
cttz.elts is usually lowered (for SVE) to a brkb followed by a cntp. If
we then want a mask based on that (say, for early exit masking) then we
would use a whilelo from 0 to the result of cntp. But that just gives us
the same mask as the initial brkb, so we can just remove the cntp and
the whilelo.
Brka matches the extra +1 in the pattern.
When computing the viable cycles for scheduling an instruction,
`computeStart` used to include special-case logic to handle loop-carried
dependencies. This special handling was necessary because loop-carried
dependencies were represented by reversed forward-direction edges in the
DAG. Now that we have the DDG, which explicitly models loop-carried
dependencies, this special handling is no longer required. As a first
step towards completely removing `isLoopCarriedDep`, this patch
eliminates the special-case logic from `computeStart` and some related
functions.
Split off from https://github.com/llvm/llvm-project/pull/135148
As with loads and stores, instructions that may trigger floating‑point
exceptions must not be reordered across a barrier instruction. This
patch adds the missing loop‑carried dependencies between such
instructions and the barrier, preventing reordering that could
previously occur. Same as #174391, the implementation is based on that
of `ScheduleDAGInstrs::buildSchedGraph`.
Split off from #135148
The loads/stores must not be reordered across barrier instructions.
However, in MachinePipeliner, it potentially could happen since
loop-carried dependencies from loads/stores to a barrier instruction
were not considered. The same problem exists for barrier-to-barrier
dependencies. This patch adds the handling for those cases. The
implementation is based on that of `ScheduleDAGInstrs::buildSchedGraph`.
Split off from https://github.com/llvm/llvm-project/pull/135148
This patch fixes https://github.com/llvm/llvm-project/issues/150737.
The original computed CSRCost is too small, so the optimization of
spilling instead of using CSR is rarely triggered.
Also the original cost model is too difficult to be understood and too
hard to be tuned by backend developers and users.
So this patch changes the CSRCost to be
CSRCost = TRI->getCSRFirstUseCost() * EntryFreq * Scale
TRI->getCSRFirstUseCost() is the raw cost of save/restore a CSR. Usually
we don't need to tune this number.
EntryFreq is the BlockFrequency of the entry block.
Scale is used to scale down the CSRCost, because we usually prefer a CSR
register instead of spilling if we have similar CSRCost and spill cost,
so it should be less than 100%. We usually tune this number.
Another problem is the original function RAGreedy::calcSpillCost()
actually computes a cost for block split, so this patch also implements
a correct RAGreedy::calcSpillCost() function.
This new behavior is not enabled by default. This optimization is used
by 3 targets (AArch64 / AMDGPU / RISCV), I will change them one by one
in following patches.
ptrauth intrinsic to safely construct relative ptr for swift coroutines.
A ptrauth intrinsic for swift co-routine support that allows creation of
signed pointer
from offset stored at address relative to the pointer.
Following C-like pseudo code (ignoring keys,discriminators) explains its
operation:
let rawptr = PACauth(inputptr);
return PACsign( rawptr + signextend64( *(int32*)(rawptr+addend) ))
What: Authenticate a signed pointer, load a 32bit value at offset
'addend' from pointer,
add this value to pointer, sign this new pointer.
builtin: __builtin_ptrauth_auth_load_relative_and_sign
intrinsic: ptrauth_auth_resign_load_relative
These were added recently with a fairly complex propagation step,
however, these optimizations can cause regressions in some cases.
This patch limits the cross-block optimizations to the simple case
picking a state that matches all incoming blocks. If any block doesn't
match, we fallback to using "ACTIVE", the default state.
When lowering cttz.elts for fixed-length vectors when SVE is available,
we use scalable container types for the predicate types since NEON
doesn't have dedicated predicate registers. Unfortunately, this also
discards the actual length of the vector to look at if it's shorter than a
full vector.
Example codegen, for a llvm.experimental.cttz.elts.i64.v4i1
shl v0.4h, v0.4h, 15
ptrue p0.h, vl4
ptrue p1.h
cmpne p0.h, p0/z, z0.h, #0
brkb p0.b, p1/z, p0.b
cntp x8, p0, p0.h
The 'ptrue p1.h' is where we went wrong -- if p0 is empty, we should
only set 4 lanes active at most, but since brkb's pg operand is all
active, it sets all available lanes (e.g. 8 .h lanes on a 128b SVE
implementation).
We need to use the 'vl4' mask for brkb's pg operand instead. So I've
added an extra parameter to AArch64ISD::CTTZ_ELTS to carry the
appropriate pattern from where it's lowered to ISel.
Fixes#178644
[AArch64][GlobalISel] Constrain G_CONSTANT_FOLD_BARRIER operand
Instruction selection is lowering:
bb.1:
%6:gpr(s64) = G_CONSTANT i64 457873110
...
bb.2:
%12:gpr(s64) = G_CONSTANT_FOLD_BARRIER %6
%24:gpr(s64) = G_CONSTANT i64 0
%13:gpr(s64) = G_AND %24, %12
...
to:
%13:gpr64 = ANDXrr %24:gpr64, %6:gpr64sp'
which is causing the verifier to fail with:
Expected a GPR64 register, but got a GPR64sp register
the reason for this is there are multiple G_CONSTANT_FOLD_BARRIER ops
with the same input register:
Select: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr
Erasing: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
--
Select: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
Erasing: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64
--
Select: %7:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64
Erasing: %7:gpr64sp(s64) = G
and the middle one used by the AND gets overridden by last and becomes
invalid.
this patch fixes this by constraining the register class such that the
middle one gets constrained to the subset of gpr64sp and gpr64, i.e.
gpr64common:
Select: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr
Erasing: %14:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
--
Select: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64sp
Erasing: %12:gpr64(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64common
--
Select: %7:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64common
Erasing: %7:gpr64sp(s64) = G_CONSTANT_FOLD_BARRIER %6:gpr64common
canReplaceReg doesnt handle when both regs already have classes so the
assert is removed.
Fixes#166563.
Currently performBuildShuffleExtendCombine only supports ANY_EXTEND
operands for BUILD_VECTOR inputs, and will bail if it encounters a
VECTOR_SHUFFLE with ANY_EXTEND operands. Update the logic so that we
support shuffles with ANY_EXTEND operands, which brings the code in line
with the comment.
We can, when attempting to lower to tbz, skip a zext that is then not
accounted for elsewhere. The attached test ends up with a tbz from an
extract that then does not properly zext the value extracted from the
vector. This patch fixes that by only looking through a G_ZEXT if the
bit checked is in the low part of the value, lining up the code with the
comment.
Fixes#173895
Preserve, PDT, BPI, LazyBPI and LazyBFI. These are all IR analysis that
are not invalidated by machine passes.
This partially mitigates the compile-time regression from
https://github.com/llvm/llvm-project/pull/174746.
This patch converts AArch64 CLS intrinsics (aarch64_neon_cls) to use the
generic ISD::CTLS node.
- aarch64_neon_cls: Lowered to ISD::CTLS, pattern-matched to CLS
instruction
- Set ISD::CTLS as Legal for NEON vector types (v8i8, v16i8, v4i16,
v8i16, v2i32, v4i32)
Also adds generic CTLS expansion support:
- ExpandIntRes_CTLS in LegalizeIntegerTypes for i64->i32 type expansion
- expandCTLS in TargetLowering for targets without native CLS
instruction
Part of: https://github.com/llvm/llvm-project/issues/174337
This patch extends `BuildVectorSDNode::isConstantSequence` to recognize
constant sequences that contain undef elements at any position.
The new implementation finds the first two non-undef constant elements,
computes the stride from their difference, then verifies all other
defined elements match the sequence. This enables SVE's INDEX
instruction to be used in more cases.
This change particularly benefits ZIP1/ZIP2 patterns where one operand
is a constant sequence. When a smaller constant vector like `<0, 1, 2,
3>` is used in a ZIP1 shuffle producing a wider result, it gets expanded
with trailing undefs. Similarly, for ZIP2 patterns, the DAG combiner
transforms the constant to have leading undefs since ZIP2 only uses the
upper half of its operands.
In particular, these patterns arise naturally from `VectorCombine`'s
`compactShuffleOperands` optimization (see #176074) that I am suggesting
as a fix for #137447.
This is a reland of #172523.
The original patch caused an assertion failure on RISC-V because it
attempted to create a bitcast from an illegal type (i32 on RV64) during
the post-type-legalization DAGCombine stage.
Added a `TLI.isTypeLegal(Val.getValueType())` check to ensure we only
proceed with the bitcast STLF optimization when the source value's type
is legal for the target.
This patch introduces support for Store-to-Load Forwarding (STLF) in
`DAGCombiner::ForwardStoreValueToDirectLoad` when the store and load
have **different types but equal memory size** (e.g., storing an `i32`
then loading a `float` from the same location).
### What this patch does:
**Enables Optimization:** It allows for the safe forwarding of the
stored value as a Bitcast when the value is:
* A **Constant** (`ConstantSDNode`, `ConstantFPSDNode`,
`ConstantPoolSDNode`).
* **Undef**.
* And the memory sizes (`LdMemSize` == `StMemSize`) match.
### Scope and Next Steps:
This patch **only implements forwarding for constant and undef values
that has the same memory size** so far.
**I am submitting this initial patch to get early review feedback on the
core logic and fix the immediate crashes before tackling the more
complex scenarios.**
For the simple case:
```llvm
; Case Handled by this PR so far (e.g., zeroinitializer is a constant)
define float @test_stlf_integer(ptr %p, float %v) {
store i32 0, ptr %p, align 4
%f = load float, ptr %p, align 4
; ...
}
```
Fixes: #151683
This can be seen as a special-case of the existing fptoi(fmul(a, 2.0))
combine that will be moved to tablegen, which can generate a FCVT_shift
instruction.
This change improves memset code generation for non-zero values on
AArch64 by using NEON's DUP instruction instead of
the less efficient multiplication with 0x01010101 pattern.
For small sizes, the value is extracted from a larger DUP. For
non-power-of-two sizes, overlapping stores are used in some cases.
TargetLowering::findOptimalMemOpLowering is modified to allow explicitly
specifying the size of the constant in cases where the constant is
larger than the store operations.
Fixes#165949
This PR adds the combine rule `fold (add x, shl(0 - y, n)) -> sub(x,
shl(y, n))` to GlobalISel, corresponding to an existing SelectionDAG
combine in
[DAGCombiner::visitADDLikeCommutative](fcba304010/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (L3367)).
Co-authored-by: Sarah Kuhn <sarahlinhkuhn@gmail.com>
---------
Co-authored-by: Sarah Kuhn <sarahlinhkuhn@gmail.com>
Co-authored-by: Luisa Cicolini <48860705+luisacicolini@users.noreply.github.com>
This PR converts two DAGCombiner fold patterns to GlobalISel MIR
patterns:
- `((A + (B - C)) - B) -> A - C`
- `((A - (B - C)) - C) -> A - B`
Original patterns:
5b4811eddb/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (L4359)
The patterns do not include hasOneUse checks since the instruction count
remains the same even when both intermediate results have other uses (3
ops → 3 ops).
