When the first element of a trn mask is undef, the `isTRNMask` function
assumes `WhichResult = 1`. That has a 50% chance of being wrong, so we
fail to match some valid trn1/trn2.
This patch introduces a more precise test to determine the correct value
of `WhichResult`, based on corresponding code in the `isZIPMask` and
`isUZPMask` functions.
- This change is based on #89578. I'd like to follow it up with a
further change along the lines of #167235.
This aims to fix the crash in #168495, my combine rule was
missing a check that the source vector was in fact a vector. This then
caused the legality check to fail in this example as the concat was
trying to concat a non vector.
I have also gated the bitcast of the concat to only work on non-scalable
vectors as the mutation calls `getNumElements` which crashes when called
on a scalable vector.
Fixes#168495
The AArch64 backend converts trees formed by conjunctions/disjunctions
of comparisons into sequences of `CCMP` instructions. The implementation
before this change checks whether a sub-tree must be processed first. If
not, it processes the operations in the order they occur in the DAG.
This may not be optimal if there is a corresponding `SUB` node for one
of the comparisons. In this case, we should process this comparison
first because we can then use the same instruction for the `SUB` node
and the comparison.
To achieve this, this commit comprises the following changes:
- Extend `canEmitConjunction` with a new output parameter `PreferFirst`,
which reports to the caller whether the sub-tree should preferably be
processed first.
- Set `PreferFirst` to `true` if we can find a corresponding `SUB` node
in the DAG.
- If we can process a sub-tree with `PreferFirst = true` first (i.e., we
do not violate any `MustBeFirst` constraint by doing so), we swap the
sub-trees.
- The already existing code for performing the common subexpression
elimination takes care to use only a single instruction for the
comparison and the `SUB` node if possible.
Closes#149685.
Updates the demanded elements before recursing through copies in case
the type of the source register changes from a non-vector register to a
vector register.
Fixes#167842.
This adds handling for f16 and f128 lround/llround under LP64 targets,
promoting the f16 where needed and using a libcall for f128. This
codegen is now identical to the selection dag version.
Arm64EC indirect calls use a function __os_arm64x_check_icall... this
has one obvious return value, x11, which is the function to call.
However, it actually returns one other important value: x9, which is the
final destination for the emulator after the call. If the call is
calling x64 code, x9 is used by the thunk.
Previously, we didn't model this, and it mostly worked because the
compiler usually doesn't modify x9 in the narrow window between the
check, and the call. That said, it can happen in some cases; one
reliable way is to do an indirect tail-call with stack protectors
enabled. (You can also just get unlucky with register allocation, but
it's harder to write a testcase for that.)
This patch uses the cfguardtarget bundle to simplify the calling
convention handling, for similar reasons that x64 uses it: modifying
arbitrary calls is difficult without a separate marking.
Fixes#167430.
The motivation is to allow passes such as MachineLICM to hoist trivial
FMOV instructions out of loops, where previously it didn't do so even
when the RHS is a constant.
On most architectures, these expensive move instructions have a latency
of 2-6 cycles, and certainly not cheap as a 0-1 cycle move.
This PR adds a new combine to the `post-legalizer-combiner` pass. The
new combine checks for vectors being unmerged and subsequently padded
with `G_IMPLICIT_DEF` values by building a new vector. If such a case is
found, the vector being unmerged is instead just concatenated with a
`G_IMPLICIT_DEF` that is as wide as the vector being unmerged.
This removes unnecessary `mov` instructions in a few places.
We generate a ADDLV node that incorporates a vecreduce(zext) from
elements of half the size. This means that we need the input type to be
at least twice the size of the input.
I updated some variable names whilst I was here.
Fixes#167935
In the Dhrystone benchmark, I find some adjacent global not be merged,
on the contrary the GCC's anchor optimize is work. Use
global-merge-max-offset to set the max offset can yield similar results
(still slightly different, at least we can control the offset).
As pointed out in post-commit discussion on #167336
<https://github.com/llvm/llvm-project/pull/167336#issuecomment-3523794206>,
although the test case succeeds in showing a codegen difference now the
faulty MachineCopyPropagation logic was removed, the example was reduced
so much that it actually would have been legal to remove the seemingly
redundant mov.
This is a re-reduction of that test case which should now demonstrate a
mov that can't safely be removed (mov w9, w9) because the upper bits no
longer being zeroed may alter the program logic.
TargetConstant nodes don't match TableGen ImmLeaf patterns during
instruction selection. When this zero constant flows into the AArch64
CCMP formation code, the machine verifier hits an assertion in expensive
checks.
Fixes: #168227
This PR improves the lowering of vectors of fp16 when using fpext.
Previously vectors of fp16 were scalarized leading to lots of extra
instructions. Now, vectors of fp16 will be lowered when extended to fp64
via the preexisting lowering logic for extends. To make use of the
existing logic, we need to add elements until we reach the next power of
2.
This patch adds codegen for CBB and CBH, CB variants operating on bytes
and half-words, allowing to fold sign- and zero-extensions.
Since if-conversion needs to be able to undo conditional branches, we
remember possibly folded zero- and sign-extensions, as well as
potentially folded assertzext and assertsext as additional
arguments of the CBBAssertExt and CBHAssertExt pseudos during codegen.
The Transactional Memory Extension (TME) was introduced as part of
Armv9-A but has not been adopted by the ecosystem. This mirrors what
Arm has observed with similar extensions in other architectures.
Therefore, remove FEAT_TME assembly and ACLE code from llvm, because
support for TME has now been officially withdrawn, as noted here:
```
FEAT_TME is withdrawn from all future versions of Arm®
Architecture Reference Manual for A-profile architecture.
```
referenced in Known Issue D24093, documented here:
https://developer.arm.com/documentation/102105/lb-05/
This patch ensures we switch out of streaming mode before TLS-descriptor
calls. ZA state will also be preserved when using the new SME ABI
lowering (`-aarch64-new-sme-abi`).
Fixes#152165
InlineAsmLowering rejected inline assembly with memory reference inputs
if the values passed to the inline asm weren't pointers. The DAG
lowering however handled them just fine.
This patch updates InlineAsmLowering to store such values on the stack,
and then use the stack pointer as the "indirect" version of the operand.
As new operations are added (for example uinc_wrap, udec_wrap,
usub_cond, usub_sat), they will not automatically be supported by
outline atomics and so should be expanded by the pre-isel pass. Make the
list of supported outline atomics explicit to make sure we only mark the
expected intrinsics as outline atomics.
Fixes#167728
We only seem to use the SVE fdot for fixed-length vector types when they
are larger than 128bits, whereas we can also use them for 128bits
vectors if SVE2p1/SME2 is available.
We're likely to get better code from custom legalisation, where we can
remove unpack instructions (plus SVE2p1 has BFMLSLB/T), but we get much
of benefit with these two small changes.
NOTE: LLVM has no support for FEAT_AFP in terms of feature detection or
ACLE builtins, so the compiler works under the assumption the feature is
not enabled.
Patch is also more aggressive when enabling bfloat fma construction
because it removes unnecessary rounding which is generally preferable
regardless of whether BFMLALB is used or not.
This patch enables `aarch64-split-sve-objects` to handle hazard padding
in functions that use the SVE CC even when there are no predicate
spills/locals.
This improves the codegen over the base hazard padding implementation,
as rather than placing the padding in the callee-save area, it is placed
at the start of the ZPR area.
E.g., Current lowering:
```
sub sp, sp, #1040
str x29, [sp, #1024] // 8-byte Folded Spill
addvl sp, sp, #-1
str z8, [sp] // 16-byte Folded Spill
sub sp, sp, #1040
```
New lowering:
```
str x29, [sp, #-16]! // 8-byte Folded Spill
sub sp, sp, #1024
addvl sp, sp, #-1
str z8, [sp] // 16-byte Folded Spill
sub sp, sp, #1040
```
This also re-enables paired stores for GPRs (as the offsets no longer
include the hazard padding).
This is a follow-up from #166926 that ensures the hints are only added
once, and ensures that hints inserted by the register allocator take
priority over hints to reduce movprfx.
Adjust the frame setup code for Windows ARM64 to attempt to align
pair-wise spills to 16-byte boundaries. This enables us to properly emit
the spills for custom clang calling convensions such as preserve most
which spills r9-r15 which are normally nonvolatile registers. Even when
using the ARM64EC opcodes for the unwinding, we cannot represent the
spill if it is unaligned.
As reported in <https://github.com/llvm/llvm-project/issues/166870>,
some copies with src==reg are not no-ops, e.g. when self-assigning a
w-reg on AArch64 which will zero-extend the corresponding x register.
Revert in order to fix the issue. We may revisit whether the
optimisation can be made safe at a later point.
Reverts dffbc030e75b758e7512518abd499a0f680addbf.
Fixes#166870.
This fixes verifier errors in this test after earlier passes start
introducing more subregister uses. This probably isn't adequately
tested but I know nothing about this pass.
This should be the peephole's job. Because and sets V flag to 0, this is
why signed comparisons with 0 are okay to replace with tst. Note this is
only for AArch64, because ANDS on ARM leaves the V flag the same.
Fixes: https://github.com/llvm/llvm-project/issues/154387
On AArch64, ADRP and its user instructions (LDR, ADD, etc.), that are
referencing a GOT symbol, when separated into different functions by
machine outliner exposes a correctness issue in the linker ICF. In such
cases, user instructions can end up pointing to a folded section (with
its canonical folded symbol), while ADRP instruction point to a GOT
entry corresponding to the original symbol. This leads to loading from
incorrect memory address after ICF. #129122 explains how this can happen
in detail.
This addresses #131660 which should fix two things:
1. Hide the correctness issue described above in the LLVM linker.
2. Allows optimizations that could relax GOT addressing to PC-relative
addressing.
sincospi/sincospif/sincospil does not appear to exist on common
targets. Darwin targets have __sincospi and __sincospif, so define
and use those implementations. I have no idea what version added
those calls, so I'm just guessing it's the same conditions as
__sincos_stret.
Most of this patch is working to preserve codegen when a vector
library is explicitly enabled. This only covers sleef and armpl,
as those are the only cases tested.
The multiple result libcalls have an aberrant process where the
legalizer looks for the scalar type's libcall in RuntimeLibcalls,
and then cross references TargetLibraryInfo to find a matching
vector call. This was unworkable in the sincospi case, since the
common case is there is no scalar call available. To preserve
codegen if the call is available, first try to match a libcall
with the vector type before falling back on the old scalar search.
Eventually all of this logic should be contained in RuntimeLibcalls,
without the link to TargetLibraryInfo. In principle we should perform
the same legalization logic as for an ordinary operation, trying
to find a matching subvector type with a libcall.
For predicated SVE instructions where we know that the inactive lanes
are undef, it is better to pick a destination register that is not
unique. This avoids introducing a movprfx to copy a unique register to
the destination operand, which would be needed to comply with the tied
operand constraints.
For example:
```
%src1 = COPY $z1
%src2 = COPY $z2
%dst = SDIV_ZPZZ_S_UNDEF %p, %src1, %src2
```
Here it is beneficial to pick $z1 or $z2 as the destination register,
because if it would have chosen a unique register (e.g. $z0) then the
pseudo expand pass would need to insert a MOVPRFX to expand the
operation into:
```
$z0 = SDIV_ZPZZ_S_UNDEF $p0, $z1, $z2
->
$z0 = MOVPRFX $z1
$z0 = SDIV_ZPmZ_S $p0, $z0, $z2
```
By picking $z1 directly, we'd get:
```
$z1 = SDIV_ZPmZ_S, $p0 $z1, $z2
```
