7dc20ab introduced an extra COPY when spilling and filling a PNR
register, which can't be elided as the input (PNR predicate) and output
(PPR predicate) register classes differ. The patch adds a new register
class that covers both PPR and PNR so that STR_PXI and LDR_PXI can
take either of them, removing the need for the copy.
We can add implicit defs/uses of the 'VG' register to the instructions
to prevent the register allocator from rematerializing values in between
streaming-mode changes, as the def/use of VG will further nail down the
ordering that comes out of ISel. This avoids the heavy-handed approach
to prevent any kind of rematerialization.
While we could add 'VG' as a Use to all SVE instructions, we only really
need to do this for instructions that are rematerializable, as the
smstart/smstop instructions and pseudos act as scheduling barriers which
is sufficient to prevent other instructions from being scheduled in
between the streaming-mode-changing call sequence. However, we may
revisit this in the future.
When we lower calls, the sequence of argument copy-to-reg nodes are
glued to the smstart. In the InstrEmitter, these glued copies are turned
into implicit defs, since the actual call instruction uses those
physregs, resulting in the register allocator adding unnecessary copies
of regs that are preserved anyway.
The svldr_vnum and svstr_vnum builtins always modify the base register
and tile slice and provide immediate offsets of zero, even when the
offset provided to the builtin is an immediate. This patch optimises the
output of the builtins when the offset is an immediate, to pass it
directly to the instruction and to not need the base register and tile
slice updates.
This patch adds the feature flags of SME_F8F16 and SME_F8F32,
and the assembly/disassembly for the following instructions of SME2:
* SME:
- FMLAL, FMLALL
- FVDOT, FVDOTT
- FVDOTB
- FMOPA
That is according to this documentation:
https://developer.arm.com/documentation/ddi0602/2023-09
Co-authored-by: Caroline Concatto <caroline.concatto@arm.com>
This patch adds the feature flags of LUT and SME_LUTv2, and the
assembly/disassembly
for the following instructions of NEON, SVE2 and SME2:
* NEON:
- LUT2
- LUT4
* SVE2:
- LUTI2_ZZZI
- LUTI4_ZZZI
- LUTI4_Z2ZZI
* SME:
- MOVT
- LUTI4_4ZZT2Z
- LUTI4_S_4ZZT2Z
That is according to this documentation:
https://developer.arm.com/documentation/ddi0602/2023-09
…r the assembly disassembly
This NFC patch refactors the assembly/disassembly class and multiclass
in the AArch64 backend to receive a new 2023/09 AArch64[1] ISA release.
The encoding for the 2023 instructions re-uses encoding blocks from
previous assembly/disassembly instructions.
The refactoring makes the class and multiclass for assembly/disassembly
generic so it can be used to describe the instructions for the new ISA.
[1]https://developer.arm.com/documentation/ddi0602/2023-09
This patch separates PNR registers into their own register class instead
of sharing a register class with PPR registers. This primarily allows us
to return more accurate register classes when applying assembly
constraints, but also more protection from supplying an incorrect
predicate type to an invalid register operand.
This patch adds error diagnostics to Clang when code uses the AArch64 SME
attributes without specifying 'sme' as available target attribute.
* Function definitions marked as '__arm_streaming', '__arm_locally_streaming',
'__arm_shared_za' or '__arm_new_za' will by definition use or require SME
instructions.
* Calls from non-streaming functions to streaming-functions require
the compiler to enable/disable streaming-SVE mode around the call-site.
In some cases we can accept the SME attributes without having 'sme' enabled:
* Function declaration can have the SME attributes.
* Definitions can be __arm_streaming_compatible since the generated
code should execute on processing elements without SME.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D157269
All the bits of the first operand are copied to the destination register,
if the tested bit (in the second source operand) is active. This means we
copy over all vscale x 16 x i1's of the first operand. There is no need to
overload that type.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D150958
Since the same encoded offset is used for both the vector
select offset and the address offset we have to spot two
patterns simulatenously in the ldr/str intrinsic inputs, i.e.
vector select = base + off
address = base + (off * VL)
whereas currently we only look for the address pattern. I
don't think this is possible in tablegen, so I suspect we'll
have to do this manually as part of lowering or as a target
DAG combine. For now, I've removed these tablegen patterns
so that we at least do the correct thing even if the code
quality isn't great.
I've also changed some of the ldr/str tests to pass in the
same vector select pattern (base + off) as the address
pattern.
Differential Revision: https://reviews.llvm.org/D147433
Adds intrinsics for the following SME2 instructions (1, 2 & 4 vector):
- smlall
- umlall
- smlsll
- umlsll
- sumlall
- usmlall
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D143278
Adds intrinsics for the following SME2 instructions (2 & 4 vectors):
- smlall
- smlsll
- umlall
- umlsll
- usmlall
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D143277
Adds intrinsics for the following:
- smopa / smops
- umopa / umops
- bmopa / bmops
Tests for existing SME mopa/mops intrinsics have also been updated
to use the maximum allowed ZA tile number.
NOTE: These intrinsics are still in development and are subject
to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D141849
Adds IR intrinsics for the following SME2 instructions:
- sqrshr (2 and 4 vector)
- uqrshr (2 and 4 vector)
- sqrshru (2 and 4 vector)
- sqrshrn (4 vector)
- uqrshrn (4 vector)
- sqrshrun (4 vector)
Also adds intrinsics for the following SVE2p1 instructions:
- sqrshrn (2 vector)
- uqrshrn (2 vector)
- sqrshrun (2 vector)
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: CarolineConcatto
Differential Revision: https://reviews.llvm.org/D142466
Adds intrinsics for the following:
- mova: array to vector / vector to array
- mova: tile to vector / vector to tile
Tablegen patterns have been added to match the ZA write intrinsics. As the
read intrinsics return a multi-vector, a function called SelectMultiVectorMove
has been added to AArch64ISelDAGToDAG to select the correct instruction. The
SelectSMETile function has also been added to check that the tile number
passed to read intrinsics is valid for the base register.
This patch also cleans up the sme_vector_to_tile_patterns multiclass to remove
the pattern for an offset of 0, which is handled by tileslice.
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D142031
Add the following intrinsic:
ADD
SUB
FADD
FSUB
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D142210
Add the following intrinsic:
SQCVTN
SQCVTUN
UQCVTN
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D142089
Add the following intrinsic:
SQCVT
SQCVTU
UQCVT
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D142035
Add the following intrinsic:
FCVT
BFCVT
FCVTZS
FCVTZU
SCVTF
UCVTF
This patch also adds SelectCVTIntrinsic to handle the cases when the
intrinsic returns multiple (two or four) outputs
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D142032
The specification says:
For programmer convenience, an assembler must also accept
predicate-as-counter register names for the destination predicate
register and the first source predicate register
Reviewed By: CarolineConcatto, MattDevereau
Differential Revision: https://reviews.llvm.org/D141603
Add the following intrinsic:
FCVTN
BFCVTN
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D142025
Adds (single, multi & indexed) intrinsics for the following:
- bfmlal/bfmlsl
- fmlal/fmlsl
- smlal/smlsl
- umlal/umlsl
This patch also extends SelectSMETileSlice to handle scaled vector select offsets.
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: CarolineConcatto
Differential Revision: https://reviews.llvm.org/D142004
Adds intrinsics for the following:
- fmla (single, multi & indexed)
- fmls (single, multi & indexed)
NOTE: These intrinsics are still in development and are subject
to future changes.
Reviewed By: CarolineConcatto
Differential Revision: https://reviews.llvm.org/D141946
Adds an instruction mapping to SMEInstrFormats which matches SME
pseudos with the real instructions they are transformed to.
A new flag is also added to AArch64Inst (SMEMatrixType), which is
used to indicate the base register required when emitting many
of the SME instructions.
This reduces the number of pseudos handled by the switch statement
in EmitInstrWithCustomInserter.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D136856
The zip/uzp (2-vector) instruction classes have the incorrect
register constraints and mark the destination as also being an
input. However, the instructions are fully destructive so I've
restructured the classes.
Differential Revision: https://reviews.llvm.org/D138288
This patch adds a new feature flag:
sme-f16f16 to represent FEAT_SME-F16F16
This patch add the following instructions:
SME2.1 stand alone instructions:
MOVAZ (array to vector, four registers): Move and zero four ZA single-vector groups to vector registers.
(array to vector, two registers): Move and zero two ZA single-vector groups to vector registers.
(tile to vector, four registers): Move and zero four ZA tile slices to vector registers.
(tile to vector, single): Move and zero ZA tile slice to vector register.
(tile to vector, two registers): Move and zero two ZA tile slices to vector registers.
LUTI2 (Strided four registers): Lookup table read with 2-bit indexes.
(Strided two registers): Lookup table read with 2-bit indexes.
LUTI4 (Strided four registers): Lookup table read with 4-bit indexes.
(Strided two registers): Lookup table read with 4-bit indexes.
ZERO (double-vector): Zero ZA double-vector groups.
(quad-vector): Zero ZA quad-vector groups.
(single-vector): Zero ZA single-vector groups.
SME2p1 and SME-F16F16:
All instructions are half precision elements:
FADD: Floating-point add multi-vector to ZA array vector accumulators.
FSUB: Floating-point subtract multi-vector from ZA array vector accumulators.
FMLA (multiple and indexed vector): Multi-vector floating-point fused multiply-add by indexed element.
(multiple and single vector): Multi-vector floating-point fused multiply-add by vector.
(multiple vectors): Multi-vector floating-point fused multiply-add.
FMLS (multiple and indexed vector): Multi-vector floating-point fused multiply-subtract by indexed element.
(multiple and single vector): Multi-vector floating-point fused multiply-subtract by vector.
(multiple vectors): Multi-vector floating-point fused multiply-subtract.
FCVT (widening): Multi-vector floating-point convert from half-precision to single-precision (in-order).
FCVTL: Multi-vector floating-point convert from half-precision to deinterleaved single-precision.
FMOPA (non-widening): Floating-point outer product and accumulate.
FMOPS (non-widening): Floating-point outer product and subtract.
SME2p1 and B16B16:
BFADD: BFloat16 floating-point add multi-vector to ZA array vector accumulators.
BFSUB: BFloat16 floating-point subtract multi-vector from ZA array vector accumulators.
BFCLAMP: Multi-vector BFloat16 floating-point clamp to minimum/maximum number.
BFMLA (multiple and indexed vector): Multi-vector BFloat16 floating-point fused multiply-add by indexed element.
(multiple and single vector): Multi-vector BFloat16 floating-point fused multiply-add by vector.
(multiple vectors): Multi-vector BFloat16 floating-point fused multiply-add.
BFMLS (multiple and indexed vector): Multi-vector BFloat16 floating-point fused multiply-subtract by indexed element.
(multiple and single vector): Multi-vector BFloat16 floating-point fused multiply-subtract by vector.
(multiple vectors): Multi-vector BFloat16 floating-point fused multiply-subtract.
BFMAX (multiple and single vector): Multi-vector BFloat16 floating-point maximum by vector.
(multiple vectors): Multi-vector BFloat16 floating-point maximum.
BFMAXNM (multiple and single vector): Multi-vector BFloat16 floating-point maximum number by vector.
(multiple vectors): Multi-vector BFloat16 floating-point maximum number.
BFMIN (multiple and single vector): Multi-vector BFloat16 floating-point minimum by vector.
(multiple vectors): Multi-vector BFloat16 floating-point minimum.
BFMINNM (multiple and single vector): Multi-vector BFloat16 floating-point minimum number by vector.
(multiple vectors): Multi-vector BFloat16 floating-point minimum number.
BFMOPA (non-widening): BFloat16 floating-point outer product and accumulate.
BFMOPS (non-widening): BFloat16 floating-point outer product and subtract.
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
Differential Revision: https://reviews.llvm.org/D137571
This patch adds the assembly/disassembly for the following instruction:
SEL: Multi-vector conditionally select elements from two vectors
for 2 and 4 registers
Non-constiguous load with stride resgisters:
LD1B (scalar + immediate): Contiguous load of bytes to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load of bytes to multiple strided vectors (scalar index).
LD1D (scalar + immediate): Contiguous load of doublewords to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load of doublewords to multiple strided vectors (scalar index).
LD1H (scalar + immediate): Contiguous load of halfwords to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load of halfwords to multiple strided vectors (scalar index).
LD1W (scalar + immediate): Contiguous load of words to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load of words to multiple strided vectors (scalar index).
LDNT1B (scalar + immediate): Contiguous load non-temporal of bytes to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load non-temporal of bytes to multiple strided vectors (scalar index).
LDNT1D (scalar + immediate): Contiguous load non-temporal of doublewords to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load non-temporal of doublewords to multiple strided vectors (scalar index).
LDNT1H (scalar + immediate): Contiguous load non-temporal of halfwords to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load non-temporal of halfwords to multiple strided vectors (scalar index).
LDNT1W (scalar + immediate): Contiguous load non-temporal of words to multiple strided vectors (immediate index).
(scalar + scalar): Contiguous load non-temporal of words to multiple strided vectors (scalar index).
Non-constiguous store with stride resgisters:
ST1B (scalar + immediate): Contiguous store of bytes from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store of bytes from multiple strided vectors (scalar index).
ST1D (scalar + immediate): Contiguous store of doublewords from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store of doublewords from multiple strided vectors (scalar index).
ST1H (scalar + immediate): Contiguous store of halfwords from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store of halfwords from multiple strided vectors (scalar index).
ST1W (scalar + immediate): Contiguous store of words from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store of words from multiple strided vectors (scalar index).
STNT1B (scalar + immediate): Contiguous store non-temporal of bytes from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store non-temporal of bytes from multiple strided vectors (scalar index).
STNT1D (scalar + immediate): Contiguous store non-temporal of doublewords from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store non-temporal of doublewords from multiple strided vectors (scalar index).
STNT1H (scalar + immediate): Contiguous store non-temporal of halfwords from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store non-temporal of halfwords from multiple strided vectors (scalar index).
STNT1W (scalar + immediate): Contiguous store non-temporal of words from multiple strided vectors (immediate index).
(scalar + scalar): Contiguous store non-temporal of words from multiple strided vectors (scalar index).
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
This patch also adds a new SVE vector list to represent the stride loads/stores
ZPRVectorListStrided and the sets of 2 and 4 ZA registers:
ZZ_[b|h|w|d]_strided and ZZZZ_[b|h|w|d]_strided
Differential Revision: https://reviews.llvm.org/D136172
This patch adds the assembly/disassembly for the following instructions:
SQRSHR (four registers): Multi-vector signed saturating rounding shift right narrow by immediate.
(two registers): Multi-vector signed saturating rounding shift right narrow by immediate.
SQRSHRN: Multi-vector signed saturating rounding shift right narrow by immediate and interleave.
SQRSHRU (four registers): Multi-vector signed saturating rounding shift right unsigned narrow by immediate.
(two registers): Multi-vector signed saturating rounding shift right unsigned narrow by immediate.
SQRSHRUN: Multi-vector signed saturating rounding shift right unsigned narrow by immediate and interleave.
UQRSHR (four registers): Multi-vector unsigned saturating rounding shift right narrow by immediate
(two registers): Multi-vector unsigned saturating rounding shift right narrow by immediate.
UQRSHRN: Multi-vector unsigned saturating rounding shift right narrow by immediate and interleave.
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D136150
This patch adds the assembly/disassembly for the following instructions:
MOVA (array to vector, four registers): Move four ZA single-vector groups to four vector registers.
(array to vector, two registers): Move two ZA single-vector groups to two vector registers.
(tile to vector, four registers): Move four ZA tile slices to four vector registers.
(tile to vector, single): Move ZA tile slice to vector register.
(tile to vector, two registers): Move two ZA tile slices to two vector registers.
(vector to array, four registers): Move four vector registers to four ZA single-vector groups.
(vector to array, two registers): Move two vector registers to two ZA single-vector groups.
(vector to tile, four registers): Move four vector registers to four ZA tile slices.
(vector to tile, single): Move vector register to ZA tile slice.
(vector to tile, two registers): Move two vector registers to two ZA tile slices.
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
It add more sizes for Matrix Operand:
MatrixOp8 and MatrixOp16
two implicit operands uimm0s2range and uimm0s4range.
and uimm1s2range that are immediates
Differential Revision: https://reviews.llvm.org/D136142
This patch adds the assembly/disassembly for the following instructions:
SUNPK: Unpack and sign-extend multi-vector elements.
UUNPK: Unpack and zero-extend multi-vector elements.
ZIP (four registers): Interleave elements from four vectors.
ZIP (two registers): Interleave elements from two vectors.
FRINTA: Multi-vector floating-point round to integral value, to nearest with ties away from zero.
FRINTM: Multi-vector floating-point round to integral value, toward minus Infinity.
FRINTN: Multi-vector floating-point round to integral value, to nearest with ties to even.
FRINTP: Multi-vector floating-point round to integral value, toward plus Infinity.
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
Differential Revision: https://reviews.llvm.org/D136091
This patch adds the assembly/disassembly for the following instructions:
ZERO (ZT0): Zero ZT0.
LDR (ZT0): Load ZT0 register.
STR (ZT0): Store ZT0 register.
MOVT (scalar to ZT0): Move 8 bytes from general-purpose register to ZT0.
(ZT0 to scalar): Move 8 bytes from ZT0 to general-purpose register.
Consecutive:
LUTI2 (single): Lookup table read with 2-bit indexes.
(two registers): Lookup table read with 2-bit indexes.
(four registers): Lookup table read with 2-bit indexes.
LUTI4 (single): Lookup table read with 4-bit indexes.
(two registers): Lookup table read with 4-bit indexes.
(four registers): Lookup table read with 4-bit indexes.
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
This patch also adds a new register class and operand for zt0
and a another index operand uimm3s8
Differential Revision: https://reviews.llvm.org/D136088
This patch adds the assembly/disassembly for the following instructions:
BMOPA: Bitwise exclusive NOR population count outer product and accumulate.
BMOPS: Bitwise exclusive NOR population count outer product and subtract.
SMOPA (2-way): Signed integer sum of outer products and accumulate.
SMOPS (2-way): Signed integer sum of outer products and subtract.
UMOPA (2-way): Unsigned integer sum of outer products and accumulate.
UMOPS (2-way): Signed integer sum of outer products and accumulate.
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
Differential Revision: https://reviews.llvm.org/D136077
This aligns with what was done for SVE, which consistently uses 32bit
immediates at the LLVM IR level.
Additionally, this change forces the intrinsic operands to be immediates
using ImmArg<>, which subsequently requires the codegenerator to match
TargetConstants instead of Constants.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D136933
This patch adds the assembly/disassembly for the following instructions:
BMOPA: Bitwise exclusive NOR population count outer product and accumulate.
BMOPS: Bitwise exclusive NOR population count outer product and subtract.
SMOPA (2-way): Signed integer sum of outer products and accumulate.
SMOPS (2-way): Signed integer sum of outer products and subtract.
UMOPA (2-way): Unsigned integer sum of outer products and accumulate.
UMOPS (2-way): Signed integer sum of outer products and accumulate.
The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09
Differential Revision: https://reviews.llvm.org/D136077
