Expand-Atomic pass emits the CAS loop for FP operations
which limits the optimizations offered by atomic optimizer.
Moving atomic optimizer before expand-atomics allows
better codegen.
Reviewed By: arsenm, #amdgpu
Differential Revision: https://reviews.llvm.org/D157265
Reduction and Scan are implemented using `Iterative`
and `DPP` strategy for `float` type.
Reviewed By: arsenm, #amdgpu
Differential Revision: https://reviews.llvm.org/D156301
From the discussion in https://reviews.llvm.org/D158853, moving the truncate
into the splat helps more splatted scalar operands get selected on RISC-V, and
also avoids the need for splat_vector_parts on RV32.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D159147
FMADD, FMSUB instructions perform better or the same compared to indexed
FMLA, FMLS.
For example, the Arm Cortex-A55 Software Optimization Guide lists "FP
multiply accumulate" FMADD, FMSUB instructions with a throughput of 2
IPC, whereas it lists "ASIMD FP multiply accumulate, by element" FMLA,
FMLS with a throughput of 1 IPC.
The Arm Cortex-A77 Software Optimization Guide, however, does not
separately list "by element" variants of the "ASIMD FP multiply
accumulate" instructions, which are listed with the same throughput of 2
IPC as "FP multiply accumulate" instructions.
Reviewed By: samtebbs, dzhidzhoev
Differential Revision: https://reviews.llvm.org/D158008
A rotate of 8 bits of an e16 vector in either direction is equivalent to a
byteswap, i.e. vrev8. There is a generic combine on ISD::ROT{L,R} to
canonicalize these rotations to byteswaps, but on fixed vectors they are
legalized before they have the chance to be combined. This patch teaches the
rotate vector_shuffle lowering to emit these rotations as byteswaps to match
the scalable vector behaviour.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D158195
Given a shuffle mask like <3, 0, 1, 2, 7, 4, 5, 6> for v8i8, we can
reinterpret it as a shuffle of v2i32 where the two i32s are bit rotated, and
lower it as a vror.vi (if legal with zvbb enabled).
We also need to make sure that the larger element type is a valid SEW, hence
the tests for zve32x.
X86 already did this, so I've extracted the logic for it and put it inside
ShuffleVectorSDNode so it could be reused by RISC-V. I originally tried to add
this as a generic combine in DAGCombiner.cpp, but it ended up causing worse
codegen on X86 and PPC.
Reviewed By: reames, pengfei
Differential Revision: https://reviews.llvm.org/D157417
The legalizer could keep an original mask type of masked load combined with
sign/zero extend, but we have to extend the mask to a type similar to our
combined load otherwise instruction selection could not lower the load.
Differential Revision: https://reviews.llvm.org/D158386
As @efriedma's comment, the largest number of bits that can actually be set
for a v8i8 is 11 (the number of bits set in 8*255) so we can change the bound.
Additionally, v16i8 type is supported as v8i8.
Differential Revision: https://reviews.llvm.org/D158613
Before accessing "getOpcode" thorugh machine instruction, check if the iterator
has reached the end of Machine basic block otherwise we will crash at the
assertion `!NodePtr->isKnownSentinel()`.
The above assertion is hit in "Prologue/Epilogue Insertion & Frame Finalization
pass".
Reviewed By: craig.topper, wangpc
Differential Revision: https://reviews.llvm.org/D158256
Machine function splitting + branch relaxation currently don't properly
handle inline asm goto blocks that conditional branch to cold goto
labels. While such inline asm is technically invalid, machine
function splitting is the only thing that exposes it as such.
Since machine function splitting doesn't help too much in these
circumstances anyway, disable it for asm goto blocks and their targets.
Differential Revision: https://reviews.llvm.org/D158647
If doubling the VL will fit in a vsetivli, use it. It will be cheap
to change and cheap to change back.
This improves codegen from D158896.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D158896
That matches how such a SPLAT_VECTOR would have been type legalized
so assume it is ok to use for creating constants after type legalization.
Still need some improvements to SPLAT_VECTOR lowering.
This overlaps with some of what D158742 was trying to fix.
Reviewed By: luke
Differential Revision: https://reviews.llvm.org/D158870
If we're splatting the original subvector width then just use the original X86ISD::SUBV_BROADCAST_LOAD node - similar to what we're already doing with X86ISD::VBROADCAST/VBROADCAST_LOAD
`vmadc` and vector mask-register logical instructions are commutable
and there is no merge operand for their pseudos.
We add `isCommutable=1` for these pseudos to gain more optimization
opportunities.
This patch fixes part of https://github.com/llvm/llvm-project/issues/64422.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D158976
Before elimination of mostly empty block it makes sense to remove dead PHI nodes.
It open more opportunity for elimination plus eliminates dead code itself.
It appeared that change results in failing many unit tests and some of
them I've updated and for another one I disable this optimization.
The pattern I observed in the tests is that there is a infinite loop
without side effects. As a result after elimination of dead phi node all other
related instruction are also removed and tests stops to check what it is expected.
Reviewed By: efriedma
Differential Revision: https://reviews.llvm.org/D158503
Jump tables on AArch64 are label-relative rather than table-relative, so
having jump table destinations that are in different sections causes
problems with relocation. Jump table lookups have a max range of 1MB, so
all destinations must be in the same section as the lookup code. Both of
these restrictions can be mitigated with some careful and complex logic,
but doing so doesn't gain a huge performance benefit.
Efficiently ensuring jump tables are correct and can be compressed on
AArch64 is a TODO item. In the meantime, don't split blocks that can
cause problems.
Differential Revision: https://reviews.llvm.org/D157124
This disables the transform if the branch does not have the kill
flag set for the AND we want to delete.
Ideally we'd be able to share the AND with the AND we create in
the expansion, but that's a more complex transform. So this starts
with the simple approach to fix miscompile.
This should be backported to LLVM 17.
Fixes PR65025.ll
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D158962
Some instruction selection patterns required for ALU GPR instructions have already been automatically imported from existing TableGen descriptions - this patch simply adds testing for them. Logic for selecting constants and copies has been added, along with the first of the GIComplexPatternEquiv definitions required to select the shiftMaskXLen ComplexPattern. New patterns have been added to directly select RV64 W instructions from gMIR rather than using custom gMIR operations earlier in the pipeline. In future this could also support the checks present in the DAGToDAGISel for finding ops whose users only require the lower 32 bits.
Differential Revision: https://reviews.llvm.org/D76445
We can work out the known bits for a given lane by concatenating the known bits of each scalar operand.
In the description of ISD::SPLAT_VECTOR_PARTS in ISDOpcodes.h it says that the
total size of the scalar operands must cover the output element size, but I've
added a stricter assertion here that the total width of the scalar operands
must be exactly equal to the element size. It doesn't seem to trigger, and I'm
not sure if there any targets that use SPLAT_VECTOR_PARTS for anything other
than v4i32 -> v2i64 splats.
We also need to include it in isTargetCanonicalConstantNode, otherwise
returning the known bits introduces an infinite combine loop.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D158852
This improves some cases where a splat_vector uses a build_pair that can be
simplified, e.g:
(rotl x:i64, splat_vector (build_pair x1:i32, x2:i32))
rotl only demands the bottom 6 bits, so this patch allows it to simplify it to:
(rotl x:i64, splat_vector (build_pair x1:i32, undef:i32))
Which in turn improves some cases where a splat_vector_parts is lowered on
RV32.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D158839
Summary: This patch handles the SectionKind of ReadOnlyWithRel on AIX. The failure was discovered during sanitizer enablement and occured with `-fsanitize-coverage` option.
Reviewed By: hubert.reinterpretcast
Differential Revision: https://reviews.llvm.org/D157483
This reverts commit 8d0c3db388143f4e058b5f513a70fd5d089d51c3.
Causes crashes, see comments in https://reviews.llvm.org/D149367.
Some follow-up fixes are also reverted:
This reverts commit 636269f4fca44693bfd787b0a37bb0328ffcc085.
This reverts commit 5966079cf4d4de0285004eef051784d0d9f7a3a6.
This reverts commit e7294dbc85d24a08c716d9babbe7f68390cf219b.
