This is an initial attempt at preserving debug information in the pseudo
instruction expansion of the AArch backend. In particular, we preserve
the instruction number required by the InstrRef implementation of live
debug values.
There are many other expansions that need to be considered, but the ones
addressed in this commit should be extremely common, as they handle most
arithmetic and logical instructions.
Differential Revision: https://reviews.llvm.org/D145943
Without the single use restriction we may replace the and with a
more costly duplicated compare.
Differential Revision: https://reviews.llvm.org/D145755
These attributes have been accepted in ELF for a while, and are generated by
Clang in some places, so it makes sense to support them on MachO too.
https://reviews.llvm.org/D143173
This patch resolves suboptimal code generation reported by https://github.com/llvm/llvm-project/issues/60571 .
DAGCombiner currently converts `(x or/xor const) + y` to `(x + y) + const` if this is valid.
However, if `.. + const` is broken down into a sequences of adds with carries, the benefit is not clear, introducing two more add(-with-carry) ops (total 6) in the case of the reported issue whereas the optimal sequence must only have 4 add(-with-carry)s.
This patch resolves this issue by allowing this conversion only when (1) `.. + const` is legal or promotable, or (2) `const` is a sign bit because it does not introduce more adds.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D144116
The merged store touches memory for other underlying objects, so mapping
the merged store to the first underlying object is not correct. For example
in https://github.com/llvm/llvm-project/issues/60744, the merged store is
not correctly analyzed as dependent with memory operations which are also
part of the merged store.
Fixes#60744
Reviewed By: foad
Differential Revision: https://reviews.llvm.org/D144711
There is a fold to create LD1DUPpost from dup(load) that can be postinc. If the
dup is used by a "by element" operation such as fmul or fma then it can be
slightly better to fold the dup into the fmul instead, which produces slightly
fast code.
ld1r { v1.4s }, [x0], #4
fmul v0.4s, v1.4s, v0.4s
vs
ldr s1, [x0], #4
fmul v0.4s, v0.4s, v1.s[0]
This could also be done with integer operations such as smull/umull too, so
long as the load/dup gets correctly combined into the mul operation. Currently
this just operates on foating point types.
Differential Revision: https://reviews.llvm.org/D145184
Since AArch64 has sqrt instructions, we want to use those instead of
calls to vector math routines for llvm sqrt intrinsics (since those
don't imply some of the constraints that libm calls might have) so
we just remove the mappings.
Code originally written by mgabka
Reviewed By: danielkiss, paulwalker-arm
Differential Revision: https://reviews.llvm.org/D145392
This extends the patterns for loading into the zeroth lane of a zero vector
from D144086 to SVE, which work in the same way as the existing patterns. Only
full length vectors are added here, not the narrower floating point vector
types.
STG, STZG, ST2G, STZ2G are the exceptions to append 'Offset' to name the
offset format of load/store instructions. All other load/store
instructions use 'i' as the appendix. If there is no special reason to
do so, we should make the naming consistent.
Differential Revision: https://reviews.llvm.org/D141819
These should be identical to the fp16 SVE inserts but were missing from the
tests and tablegen patterns. They are always legal (not requiring +bf16),
although there are some other issues around bf16 legalization currently.
To make legalization easier, the operands and outputs have the same size for
these ISD Nodes. When legalizing the results in PromoteIntegerResult the operands
are legalized to the same size as the outputs.
The ISD Node has two output/results, therefore the legalizing functions update
both results/outputs.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D144846
This is guarding a check for isTypeLegal so it should check is
LegalTypes.
Fixes PR61111.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D145139
This patch adds AArch64 CodeGen support such that the type can be passed
and returned to/from functions, and also adds support to use this type in
load/store operations and PHI nodes.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D136862
A LDR will implicitly zero the rest of the vector, so vector_insert(zeros,
load, 0) can use a single load. This adds tablegen patterns for both scaled and
unscaled loads, detecting where we are inserting a load into the lower element
of a zero vector.
Differential Revision: https://reviews.llvm.org/D144086
We are missing patterns to generate vector splats using LD1R. A shuffle vector
with all 0s is a vector splat if the operands are a load and undef for which
we can generate a LD1R.
Differential Revision: https://reviews.llvm.org/D145004
The AArch64 backend, during lowering, will convert an 64bit vector insert to a
128bit vector:
vector_insert %dreg, %v, %idx
=>
%qreg = insert_subvector undef, %dreg, 0
%ins = vector_insert %qreg, %v, %idx
EXTRACT_SUBREG %ins, dsub
This creates a bit of mess in the DAG, and the EXTRACT_SUBREG being a machine
nodes makes it difficult to simplify. This patch removes that, treating the
64bit vector insert as legal and handling them with extra tablegen patterns.
The end result is a simpler DAG that is easier to write tablegen patterns for.
Differential Revision: https://reviews.llvm.org/D144550
To make legalization easier, the operands and outputs have the same size for
these ISD Nodes. When legalizing the results in SplitVectorResult the operands
are legalized to the same size as the outputs.
The ISD Node has two output/results, therefore the legalizing functions update
both results/outputs.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D144744
This patch adds the LLVM IR intrinsics for the following:
* uunpk (2 and 4 vectors)
* sunpk (2 and 4 vectors)
I have named the tests sve2p1-intrinsics-* because although
the instructions are added as part of the SME2 feature they
only operate on SVE vectors.
NOTE: These intrinsics are still in development and are subject to future changes.
Differential Revision: https://reviews.llvm.org/D142964
In post-isel code, there are cases where there were redundant copies from a source FPR to an intermediate GPR in order to copy to a destination FPR. In this patch, we identify these patterns in post-isel peephole optimization and replace them with a direct FPR-to-FPR copy.
One example for this will be the insertion of the scalar result of 'uaddlv' neon intrinsic function into a destination vector. During instruction selection phase, 'uaddlv' result is copied to a GPR, & a vector insert instruction is matched separately to copy the previous result to a destination SIMD&FP register.
Reviewed By: dmgreen
Differential Revision: https://reviews.llvm.org/D142594
This change reorders the stack up-adjustment and return value copying phases of
machine-ir generation on Aarch64. Doing so prevents a bug observed for fastcc
calls with >8 arguments, where the up-adjustment required from making that call
is placed in the wrong place relative to spill and reloading code.
See: https://github.com/llvm/llvm-project/issues/60972 for full issue
reproduction and context.
Patch contributed by Bruce Collie
Differential Revision: https://reviews.llvm.org/D144791
If we have sext_inreg(vector_extract(x)) but the top bits are not used, DAG
will try to remove the sext_inreg, using vector_extract(x) directly. This can
lead to multiple uses of both sext_inreg(vector_extract(x)) and
vector_extract(x), leading to the generation of both umov and smov extracts.
This adds a target hook to prevent that under AArch64 where the sext_inreg can
be considered free if there are multiple uses of the sext and no uses of the
vector_extract. This helps fix a small regression from D144550.
Differential Revision: https://reviews.llvm.org/D144850
Adds intrinsics for the following SME2 instructions (1, 2 & 4 vector):
- smlall
- umlall
- smlsll
- umlsll
- sumlall
- usmlall
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D143278
This patch adds new intrinsics for uniform dsp operations and changes the lowering for the following builtins to emit calls to the new aarch64.sve.###.u intrinsics.
svsqsub_x
svsqsub_n_x
svuqsub_x
svuqsub_n_x
svsqsubr_x
svsqsubr_n_x
svuqsubr_x
svuqsubr_n_x
Reviewed By: Paul Walker
Differential Revision: https://reviews.llvm.org/D144704
Inserting scalar result of 'uaddlv' neon intrinsic function to a destination vector currently makes use of the integer unit. Subsequent patches will eliminate the redundant use of the integer registers in a more generic way that will include this special case. This is an initial set of tests for this functionality.
Differential Revision: https://reviews.llvm.org/D143038
We use this combine in the AArch64 postlegalizer combiner, which causes this
function to query the legalizer rules for the action for an invalid opcode/type
combination (G_AND and p0). Moving the legalizer query until after the validity
check in matchHoistLogicOpWithSameOpcodeHands() fixes this.
Adds intrinsics for the following SME2 instructions:
- srshl (single, 2 & 4 vector)
- srshl (multi, 2 & 4 vector)
- urshl (single, 2 & 4 vector)
- urshl (multi, 2 & 4 vector)
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D144118
This partially reverts a regression introduced in 8f25e382c5b1 for
AArch64 targets. In particular, we restore the logic of `(abs (sub nsw
x, y)) -> abds(x, y)` for all targets except X86, which keeps the logic
introduced in 8f25e382c5b1. See also https://reviews.llvm.org/D142288.
Differential Revision: https://reviews.llvm.org/D144379
