On the CPUs listed below, we want to avoid LDAPUR for performance
reasons. Add a tuning feature to disable them when using:
-mcpu=neoverse-v2
-mcpu=neoverse-v3
-mcpu=cortex-x3
-mcpu=cortex-x4
-mcpu=cortex-x925
This allows us to support i128 G_ICMP on RV32. I'm not sure how to test
the "left over" part of this as RISC-V always widens to a power of 2
before narrowing.
This reverts commit 9c319d5bb40785c969d2af76535ca62448dfafa7.
Some issues were discovered with the bootstrap builds, which
seem like they were caused by this commit. I'm reverting to investigate.
This change updates the AArch64DeadRegisterDefinition pass to ensure it
does not replace the destination register of a SWP instruction with the
zero register when its value is unused. This is necessary to ensure that
the ordering of such instructions in relation to DMB.LD barries adheres
to the definitions of the AArch64 Memory Model.
The memory model states the following (ARMARM version DDI 0487K.a
§B2.3.7):
```
Barrier-ordered-before
An effect E1 is Barrier-ordered-before an effect E2 if one of the following applies:
[...]
* All of the following apply:
- E1 is a Memory Read effect.
- E1 is generated by an instruction whose destination register is not WZR or XZR.
- E1 appears in program order before E3.
- E3 is either a DMB LD effect or a DSB LD effect.
- E3 appears in program order before E2.
```
Prior to this change, by replacing the destination register of such SWP
instruction with WZR/XZR, the ordering relation described above was
incorrectly removed from the generated code.
The new behaviour is ensured in this patch by adding the relevant
`SWP[L](B|H|W|X)` instructions to list in the `atomicReadDroppedOnZero`
predicate, which already covered the `LD<Op>` instructions that are
subject to the same effect.
Fixes#68428.
The SME dot instructions in these tests operate on contiguous register
tuples which use one subregister from each of the loads. When using the
strided register form for all loads, enabling subreg liveness tracking
will allow us to recognise that there is no overlap between the register
tuples used by each of the dot instructions.
This is the first in a series of patches to improve the allocation of
strided and contiguous registers for SME.
PR #66334 tried to renumber slot indexes before register allocation, but
the numbering was still affected by list entries for instructions which
had been erased. Fix this to make the register allocator's live range
length heuristics even less dependent on the history of how instructions
have been added to and removed from SlotIndexes's maps.
This fixes up the generation of 128bit atomic, volatile and non-temporal
loads/stores, under the assumption that they should usually be the same as
standard versions.
https://godbolt.org/z/xxc89eMKEFixes#64580Closes#67413
The legalizer currently generates lots of G_AND artifacts.
For example between boolean uses and defs there is always a G_AND with a mask of 1, but when the target uses ZeroOrOneBooleanContents, this is unnecessary.
Currently these artifacts have to be removed using post-legalize combines.
Omitting these artifacts at their source in the artifact combiner has a few advantages:
- We know that the emitted G_AND is very likely to be useless, so our KnownBits call is likely worth it.
- The G_AND and G_CONSTANT can interrupt e.g. G_UADDE/... sequences generated during legalization of wide adds which makes it harder to detect these sequences in the instruction selector (e.g. useful to prevent unnecessary reloading of AArch64 NZCV register).
- This cleans up a lot of legalizer output and even improves compilation-times.
AArch64 CTMark geomean: `O0` -5.6% size..text; `O0` and `O3` ~-0.9% compilation-time (instruction count).
Since this introduces KnownBits into code-paths used by `O0`, I reduced the default recursion depth.
This doesn't seem to make a difference in CTMark, but should prevent excessive recursive calls in the worst case.
Reviewed By: aemerson
Differential Revision: https://reviews.llvm.org/D159140
Before elimination of mostly empty block it makes sense to remove dead PHI nodes.
It open more opportunity for elimination plus eliminates dead code itself.
It appeared that change results in failing many unit tests and some of
them I've updated and for another one I disable this optimization.
The pattern I observed in the tests is that there is a infinite loop
without side effects. As a result after elimination of dead phi node all other
related instruction are also removed and tests stops to check what it is expected.
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D158503
Refresh of the generic scheduling model to use A510 instead of A55.
Main benefits are to the little core, and introducing SVE scheduling information.
Changes tested on various OoO cores, no performance degradation is seen.
Differential Revision: https://reviews.llvm.org/D156799
This implements the remaining overflow generating instructions in the AArch64
GlobalISel selector. Now wide add/sub operations do not fallback to SelectionDAG
anymore. We make use of PostSelectOptimize to cleanup the hereby generated
flag-setting operations when the carry-out is unused. Since we do not fallback
anymore when selecting add/sub atomics on O0 some test changes were required
there.
Fixes: https://github.com/llvm/llvm-project/issues/59407
Differential Revision: https://reviews.llvm.org/D153164
D142712 added tests for when both lse2 and lse128 are available, but
in practice there is no way to enable LSE128 without LSE2 from clang:
LSE128 is a v9 only feature and LSE2 has been mandatory since v8.4,
and +/-lse2 can not be specified on the clang command line.
Therefore it makes more sense that lse2+lse128 should emit lse128
instructions, otherwise they will not be emitted at all.
It also makes sense to remove the lse128-only backend tests if that set
of attributes is never set by the frontend.
Differential Revision: https://reviews.llvm.org/D143506
FEAT_LSE128 implies FEAT_LSE but not FEAT_LSE2, so add tests showing
what happens when you have both.
Differential Revision: https://reviews.llvm.org/D142712
Implements support for the following 128-bit atomic operations with +rcpc3:
- 128-bit store-release -> STILP
- 128-bit load-acquire -> LDIAPP
D126250 and D137590 added support for emitting LDAPR (Load-Acquire RCPc) rather
than LDAP (Load-Acquire) when +rcpc is available. This patch allows emitting
the 128-bit RCPc instructions added in FEAT_LRCPC3 (LDIAPP/STILP). The
implementation is different from LDAPR, because there are no non-RCPc
equivalents for these new instructions.
Support for the offset variants will be added in D141431.
Reviewed By: lenary
Differential Revision: https://reviews.llvm.org/D141429
There are a lot of variants of atomic operations, and AArch64 has several
distinct options to codegen them, and does different things depending on
available features, architecture version and optimisation level. The current
testing for atomic operations has been added gradually over time and does not
give full coverate. Given how complex the codegen for atomic operations is, it
is valuable to cover the entire parameter space, i.e. test them all. The
resulting set of tests serve also as a reference for how each is codegened.
In order to keep the test files readable and avoid constant updating for
unrelated codegen changes, the test outputs are filtered to only include the
relevant instructions. This shows for each operation and feature which codegen
approach is taken (e.g. ll/sc loop, atomic instruction, library call).
The following parameter space is tested:
- feature: +lse, +rcpc, etc
- optimisation level: O0, O1 (covers GISel and SelectionDAG)
- atomic instruction: load, store, cmpxchg, atomirmw*
- size: i8, i16, i32, i64, i128
- aligned/unaligned accesses
- endianness: big, little
- atomic ordering: release, acquire, etc
- load atomic only: const/non-const
- cmpxchg only: weak/strong
- atomicrmw: update operation (add, sub, etc)
Notably not covered:
- volatility: there should be no difference between volatile/non-volatile
- atomicrmw fadd/fsub
The files are split by triple, LLVM instruction, and feature. This makes it
easy to diff between features and endianness for a given instruction.
The file that generates the tests is included.
There are 70 test files with an average of 2800 lines each.
Differential Revision: https://reviews.llvm.org/D141064
