SDNode::use_iterator now returns an SDUse& when dereferenced.
SDNode::user_iterator returns SDNode*. SDNode::use_begin/use_end/uses
work on use_iterator. SDNode::user_begin/user_end/users work on
user_iterator.
We can now write range based for loops using SDUse& and SDNode::uses().
I've converted many of these in this patch. I didn't update loops that
have additional variables updated in their for statement.
Some loops use SDNode::use_iterator::getOperandNo() which also prevents
using range based for loops. I plan to move this into SDUse in a follow
up patch.
In streaming[-compatible] functions, use SVE for scalar FP conversions
to/from integer types. This can help avoid moves between FPRs and GRPs,
which could be costly.
This patch also updates definitions of SCVTF_ZPmZ_StoD and
UCVTF_ZPmZ_StoD to disallow lowering to them from ISD nodes, as doing so
requires creating a [U|S]INT_TO_FP_MERGE_PASSTHRU node with inconsistent
types.
Follow up to #112213.
Note: This PR does not include support for f64 <-> i32 conversions (like
#112564), which needs a bit more work to support.
Following on from https://github.com/llvm/llvm-project/pull/94499, this
patch adds support to the Loop Vectorizer to emit the partial reduction
intrinsics where they may be beneficial for the target.
---------
Co-authored-by: Samuel Tebbs <samuel.tebbs@arm.com>
== We were previously returning an invalid cost when truncating
anything to <vscale x 2 x i1>, which is incorrect since we can
generate perfectly good code for this.
== The costs for truncating legal or unpacked types to predicates
seemed overly optimistic. For example, when truncating
<vscale x 8 x i16> to <vscale x 8 x i1> we typically do
something like
and z0.h, z0.h, #0x1
cmpne p0.h, p0/z, z0.h, #0
I guess it might depend upon whether the input value is
generated in the same block or not and if we can avoid the
inreg zero-extend. However, it feels safe to take the more
conservative cost here.
== The costs for some truncates such as
trunc <vscale x 2 x i32> %a to <vscale x 2 x i16>
were 1, whereas in actual fact they are free and no instructions
are required.
== Also, for this
trunc <vscale x 8 x i32> %a to <vscale x 8 x i16>
it's just a single uzp1 instruction so I reduced the cost to 1.
In general, I've added costs for all cases where the destination
type is legal or unpacked. One unfortunate side effect of this
is the costs for some fixed-width truncates when using SVE now
look too optimistic.
Support the following relocations and assembly operators:
- `R_AARCH64_AUTH_TLSDESC_ADR_PAGE21` (`:tlsdesc_auth:` for `adrp`)
- `R_AARCH64_AUTH_TLSDESC_LD64_LO12` (`:tlsdesc_auth_lo12:` for `ldr`)
- `R_AARCH64_AUTH_TLSDESC_ADD_LO12` (`:tlsdesc_auth_lo12:` for `add`)
ZPR2StridedOrContiguous loads used by a FORM_TRANSPOSED_REG_TUPLE
pseudo should attempt to assign a strided register to avoid unnecessary
copies, even though this may overlap with the list of SVE callee-saved registers.
Most of these are just places that want the first user and aren't
iterating over the whole list.
While there I changed some use_size() == 1 to hasOneUse() which
is more efficient.
This is part of an effort to rename use_iterator to user_iterator
and provide a use_iterator that dereferences to SDUse&. This patch
helps reduce the diff on later patches.
This function is most often used in range based loops or algorithms
where the iterator is implicitly dereferenced. The dereference returns
an SDNode * of the user rather than SDUse * so users() is a better name.
I've long beeen annoyed that we can't write a range based loop over
SDUse when we need getOperandNo. I plan to rename use_iterator to
user_iterator and add a use_iterator that returns SDUse& on dereference.
This will make it more like IR.
The Complex Deinterleaving pass assumes that all values emitted will
result in complex numbers, this patch aims to remove that assumption and
adds support for emitting just the real or imaginary components, not
both.
This makes sure no optimizations are applied that assume the
bigger alignment or size, which could be incorrect if we link
together with non-instrumented code.
Handle ANY_EXTEND when combining a buildvector/shuffle of extended
operands, as we can safely ignore ANY_EXTENDS when checking if all signs
of the other extends are matching.
These opcodes are currently in the "strictfp" section. They should
either be in "memory", or moved into the generic ocodes.
Note that isTargetMemoryOpcode/FIRST_TARGET_MEMORY_OPCODE doesn't seem
to be used for anything at the moment.
Instead, allow any alignment >= the element size (in bytes). This is all
that is needed for (predicated) vector loads even if unaligned accesses
are disabled.
See:
https://developer.arm.com/documentation/ddi0602/2024-09/Shared-Pseudocode/aarch64-functions-memory?lang=en#impl-aarch64.Mem.read.3
Specifically:
```
// Check alignment on size of element accessed, not overall access size.
constant integer alignment = if accdesc.ispair then size DIV 2 else size;
```
The `size` passed to `Mem` by SVE load/store instructions is the element
size.
Re-apply #113148 after revert in #119331
If function pointer signing is enabled, sign personality function
pointer stored in `.DW.ref.__gxx_personality_v0` section with IA key,
0x7EAD = `ptrauth_string_discriminator("personality")` constant
discriminator and address diversity enabled.
This, like other operations, scalarizes shuffle vector operations with
types larger than 64bits. ImplicitDef and Freeze are also handled the
same way, to allow them to legalize. The legalization of
fewerElementsVectorShuffle is adjusted to handled scalarization.
With the introduction of CmpPredicate in 51a895a (IR: introduce struct
with CmpInst::Predicate and samesign), PatternMatch is one of the first
key pieces of infrastructure that must be updated to match a CmpInst
respecting samesign information. Implement this change to Cmp-matchers.
This is a preparatory step in migrating the codebase over to
CmpPredicate. Since we no functional changes are desired at this stage,
we have chosen not to migrate CmpPredicate::operator==(CmpPredicate)
calls to use CmpPredicate::getMatching(), as that would have visible
impact on tests that are not yet written: instead, we call
CmpPredicate::operator==(Predicate), preserving the old behavior, while
also inserting a few FIXME comments for follow-ups.
This reverts commit 9c319d5bb40785c969d2af76535ca62448dfafa7.
Some issues were discovered with the bootstrap builds, which
seem like they were caused by this commit. I'm reverting to investigate.
The FORM_TRANSPOSED_REG_TUPLE pseudos have been created to
improve register allocation for intrinsics which use strided and
contiguous multi-vector registers, avoiding unnecessary copies.
If the operands of the pseudo are copies where the source register is in
the StridedOrContiguous class, the pseudo is used by
getRegAllocationHints
to suggest a contigious multi-vector register which matches the
subregister
sequence used by the operands.
If the operands do not match this pattern, the pseudos are expanded
to a REG_SEQUENCE.
Patch contains changes by Matthew Devereau.
These properties are only valid on ComplexPatterns. Having them as flags
is more convenient because one can now use "let = ... in" syntax to set
these flags on several patterns at a time. This is also less error-prone
as it makes it impossible to specify these properties on records derived
from SDPatternOperator.
Pull Request: https://github.com/llvm/llvm-project/pull/119599
This patch adds the following instrinsics:
* Half-precision and BFloat16 convert, narrow, and interleave to 8-bit
floating-point.
// Variant is also available for: _bf16_x2
svmfloat8_t svcvtn_mf8[_f16_x2]_fpm(svfloat16x2_t zn, fpm_t fpm);
* Single-precision convert, narrow, and interleave to 8-bit
floating-point (top and bottom).
svmfloat8_t svcvtnt_mf8[_f32_x2]_fpm(svmfloat8_t zd, svfloat32x2_t zn,
fpm_t fpm);
svmfloat8_t svcvtnb_mf8[_f32_x2]_fpm(svfloat32x2_t zn, fpm_t fpm);
This patch implements the following intrinsics:
Convert to packed 8-bit floating-point format.
``` c
// Variants are also available for: _mf8[_bf16_x2] and _mf8[_f32_x4]
svmfloat8_t svcvt_mf8[_f16_x2]_fpm(svfloat16x2_t zn, fpm_t fpm) __arm_streaming;
```
Convert to interleaved 8-bit floating-point format.
``` c
svmfloat8_t svcvtn_mf8[_f32_x4]_fpm(svfloat32x4_t zn, fpm_t fpm) __arm_streaming;
```
In accordance with https://github.com/ARM-software/acle/pull/323.
Co-authored-by: Marin Lukac marian.lukac@arm.com
Co-authored-by: Caroline Concatto caroline.concatto@arm.com
The base cost approximates the expansion code in SelectionDAGBuilder.
For the AArch64 cases that don't need generic expansion, fixed-length
search vectors have a higher cost than scalable vectors due to the extra
instructions to convert the boolean mask.
This adds some basic costs for fpext and fpround, many of which were
already handled by the generic costing routines but this does make some
adjustments for larger vector types that can use fcvtn+fcvtn2, as
opposed to fcvtn+fcvtn+concat.
These should now more closely match the codegen from
https://godbolt.org/z/r3P9Mf8ez, for example.
* Avoid unnecessary truncation of comparison results in vecreduce_xor
* Optimize generated code for vecreduce_and and vecreduce_or by
comparing against 0.0 to check if all/any of the values are set
Alive2 proof of vecreduce_and and vecreduce_or transformation:
https://alive2.llvm.org/ce/z/SRfPtw
This patch adds the following intrinsics:
* 8-bit floating-point convert to half-precision and BFloat16.
// Variants are also available for: _bf16
svfloat16_t svcvt1_f16[_mf8]_fpm(svmfloat8_t zn, fpm_t fpm);
svfloat16_t svcvt2_f16[_mf8]_fpm(svmfloat8_t zn, fpm_t fpm);
* 8-bit floating-point convert to half-precision and BFloat16 (top).
// Variants are also available for: _bf16
svfloat16_t svcvtlt1_f16[_mf8]_fpm(svmfloat8_t zn, fpm_t fpm);
svfloat16_t svcvtlt2_f16[_mf8]_fpm(svmfloat8_t zn, fpm_t fpm);
Support the following relocations and assembly operators:
- `R_AARCH64_AUTH_GOT_ADR_PREL_LO21` (`:got_auth:` for `adr`)
- `R_AARCH64_AUTH_GOT_LD_PREL19` (`:got_auth:` for `ldr`)
`LOADgotAUTH` pseudo-instruction is expanded to actual instruction
sequence like the following.
```
adr x16, :got_auth:sym
ldr x0, [x16]
autia x0, x16
```
Both SelectionDAG and GlobalISel are suppported. For FastISel, we fall
back to SelectionDAG.
Tests starting with 'ptrauth-' have corresponding variants w/o this
prefix.
If function pointer signing is enabled, sign personality function
pointer stored in `.DW.ref.__gxx_personality_v0` section with IA key,
0x7EAD = `ptrauth_string_discriminator("personality")` constant
discriminator and address diversity enabled.
Some of the scalable tests have been split off to make the tests more
managable. AArch64TTIImpl::getCastInstrCost is also formatted to avoid the need
to fight against CI.
Add initial heuristics to selectively enable runtime unrolling for loops
where doing so is expected to be highly beneficial on Apple Silicon
CPUs.
To start with, we try to runtime-unroll small, single block loops, if
they have load/store dependencies, to expose more parallel memory
access streams [1] and to improve instruction delivery [2].
We also explicitly avoid runtime-unrolling for loop structures that may
limit the expected gains from runtime unrolling. Such loops include
loops with complex control flow (aren't innermost loops, have multiple
exits, have a large number of blocks), trip count expansion is
expensive and are expected to execute a small number of iterations.
Note that the heuristics here may be overly conservative and we err on
the side of avoiding runtime unrolling rather than unroll excessively.
They are all subject to further refinement.
Across a large set of workloads, this increase the total number of
unrolled loops by 2.9%.
[1] 4.6.10 in Apple Silicon CPU Optimization Guide
[2] 4.4.4 in Apple Silicon CPU Optimization Guide
Depends on https://github.com/llvm/llvm-project/pull/118316 for TTI
changes.
PR: https://github.com/llvm/llvm-project/pull/118317
