This started out as trying to combine bf16 fpround to BFCVT2
instructions, but ended up removing the aarch64.neon.nfcvt intrinsics in
favour of generating fpround instructions directly. This simplifies the
patterns and can lead to other optimizations. The BFCVT2 instruction is
adjusted to makes sure the types are valid, and a bfcvt2 is now
generated in more place. The old intrinsics are auto-upgraded to fptrunc
instructions too.
Commit 1eed46960c217f9480865702f06fb730c7521e61 added logic to
reassociate a (add (add x y) -c) operand to a CSEL instruction with a
comparison involving x and c (or a similar constant) in order to obtain
a common (SUBS x c) instruction.
This commit extends this logic to non-constants. In this way, we also
reassociate a (sub (add x y) z) operand of a CSEL instruction to
(add (sub x z) y) if the CSEL compares x and z, for example.
Alive proof: https://alive2.llvm.org/ce/z/SEVpR
This PR fixes operand order of `ILVOD.df` when lowering
`VECTOR_SHUFFLE`, the result was `<y[1], x[1]>` while it should be
`<x[1], y[1]>`.
* This PR is split from #123040.
Add some generated tests with every shuffle permutation
for relevant vector element types and sizes. Not sure if this
is going overboard with the number of tests. I pruned out the largest
cases (16 and 32-bit cases are impractically large), and there's
redundancy when testing the pointer cases (at least for SelectionDAG).
This uses inline assembly to produce sample values because of how the
ABI is lowered when using a function argument. Since we break all
arguments into 32-bit pieces, a shuffle never ends up forming. We
need separate handling to reconstruct shuffles in contexts involving
physical registers in ABI contexts.
I wrote a small tool to generate these, so I can easily change the
exact test body. Not sure if it's worth posting anywhere.
This is in preparation for making better use of v_pk_mov_b32,
v_mov_b64 and s_mov_b64 in shuffles.
This patch adds support for the next-generation arch15
CPU architecture to the SystemZ backend.
This includes:
- Basic support for the new processor and its features.
- Detection of arch15 as host processor.
- Assembler/disassembler support for new instructions.
- Exploitation of new instructions for code generation.
- New vector (signed|unsigned|bool) __int128 data types.
- New LLVM intrinsics for certain new instructions.
- Support for low-level builtins mapped to new LLVM intrinsics.
- New high-level intrinsics in vecintrin.h.
- Indicate support by defining __VEC__ == 10305.
Note: No currently available Z system supports the arch15
architecture. Once new systems become available, the
official system name will be added as supported -march name.
The current lowering for ptr addrspace(7) assumed that the instruction
selector can handle arbtrary LLVM types, which is not the case. Code
generation can't deal with
- Values that aren't 8, 16, 32, 64, 96, or 128 bits long
- Aggregates (this commit only handles arrays of scalars, more may come)
- Vectors of more than one byte
- 3-word values that aren't a vector of 3 32-bit values (for axample, a
<6 x half>)
This commit adds a buffer contents type legalizer that adds the needed
bitcasts, zero-extensions, and splits into subcompnents needed to
convert a load or store operation into one that can be successfully
lowered through code generation.
In the long run, some of the involved bitcasts (though potentially not
the buffer operation splitting) ought to be handled by the instruction
legalizer, but SelectionDAG makes this difficult.
It also takes advantage of the new `nuw` flag on `getelementptr` when
lowering GEPs to offset additions.
We don't currently plumb through `nsw` on GEPs since that should likely
be a separate change and would require declaring what we mean by "the
address" in the context of the GEP guarantees.
The indices of SGPR register pairs need to be 2-aligned and SGPR
quadruplets need to be 4-aligned. With this patch, we report an error
when inline asm register constraints specify a misaligned register
index, instead of silently dropping the specified index.
Fixes#123208
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
After a30e50fc, AMDGPUAAResult is being called in more situations where
BasicAA isn't sure. This exposed some regressions where NoAlias is being
incorrectly returned for two identical pointers.
The fix is to check the underlying objects for equality before returning
NoAlias.
Based on feedback from the clastb codegen PR, I'm refactoring basic codegen for the vector.extract.last.active intrinsic to lower to an ISD node in SelectionDAGBuilder then expand in LegalizeVectorOps, instead of doing everything in the builder.
The new ISD node (vector_find_last_active) only covers finding the index of the last active element of the mask, and extracting the element + handling passthru is left to existing ISD nodes.
Instruction `preld` is used to prefetch one cache-line of data from
memory in advance into the cache.
This commit allows it to be generated automatically.
If linker relaxation enabled, relaxable code sequence expanded from
pseudos should avoid being separated by instruction scheduling. This
commit tags scheduling boundary for them to avoid being scheduled.
(Except for `tls_le/tls_ie` and `call36/tail36`. Because `tls_le/tls_ie`
can be scheduled and have no influence to relax, `call36/tail36` are
expanded later in `LoongArchExpandPseudo` pass.)
A new mask target-flag is added to attach relax relocs to the relaxable
code sequence. (No need to add it for `tls_le` and `call36/tail36`
because we can simply add relax relocs for them according to their
relocs. But for other code sequence, such as `PCALA_{HI20/LO12}`, we
must use the mask flag, mainly because relax should not be added when
code model is large.)
Because of the new mask target-flag, get "direct" flags is necessary
when using their target-flags. In addition, code sequence after being
optimized by `MergeBaseOffset` pass may not relaxable any more, so the
relax "bitmask" flag should be removed.
In the unlikely case where the stack size is greater than 4GB, we may run into
the situation where the local stack size and the callee saved registers stack
size get combined incorrectly when restoring the callee saved registers. This
happens because the stack size in shouldCombineCSRLocalStackBumpInEpilogue
is represented as an 'unsigned', but is passed in as an 'int64_t'. We end up with
something like
$fp, $lr = frame-destroy LDPXi $sp, 536870912
This change just makes 'shouldCombineCSRLocalStackBumpInEpilogue' match
'shouldCombineCSRLocalStackBump' where 'StackBumpBytes' is an 'uint64_t'
This fold already exists in a couple places (DAG and CGP), where an
icmps operands are swapped to allow CSE with a sub. They do not handle
constants though. This patch adds an AArch64 version that can be more
precise.
Prevents infinite loop between combineBitcastToBoolVector and hoistLogicOpWithSameOpcodeHands, which only performs the "logicop(bitcast(A),bitcast(B)) -> bitcast(logicop(A,B))" upto type legalization.
combineBitcastToBoolVector doesn't care much as its mainly for AVX512 cleanup that X86DomainReassignment can't handle for us.
Fixes#123333
mayFoldIntoStore currently just checks the direct (oneuse) user of a
SDValue to check its stored, which prevents cases where we bitcast the
value prior to storing (usually the bitcast will be removed later).
This patch peeks up through oneuse BITCAST nodes chain to see if its
eventually stored.
The main use of mayFoldIntoStore is v8i16 EXTRACT_VECTOR_ELT lowering
which will only use PEXTRW/PEXTRB for index0 extractions (vs the faster
MOVD) if the extracted value will be folded into a store on SSE41+
targets.
Fixes#107086
I'd swapped the %add/%sub names, and then copied that repeatedly.
Oops. While updating, remove the nsw as those should be irrelevant
and is another copy-paste mistake on my part.
A select between an add and a sub can be either a vrsub followed by
add (reducing register pressure), or a vmacc. The former will be
implemented in an upcoming review.
The insturction selector uses the `MachineFunction::copySalvageSSA`
function to insert `DBG_PHIs` or identify a defining instruction for a
copy-like instruction when finalizing Instruction References.
AArch64 has the ORR instruction which is a logical OR with the variants
ORRWrr which refers to a register to register variant, and ORRWrs which
is a register to a shifted register variant.
An ORRWrs where the shift amount is 0, and the zero register ($wzr) is
used is considered a copy, for example:
`$w0 = ORRWrs $wzr, killed $w3, 0`
However an ORRWrr with a zero register is not considered a copy
`$w0 = ORRWrr $wzr, killed $w3`
This causes an issue in the livedebugvalues pass because in aarch64-isel
the instruction is the ORRWrr variant, but is then changed to the ORRWrs
variant before the livedebugvalues pass.
This causes a mismatch between the two passes which leads to a crash in
the livedebugvalues pass.
This patch fixes the issue.
The code pattern that clang will generate for HLSL has changed from the
original plan. This allows the SPIR-V backend to generate code for the
current code generation.
It looks for patterns of the form:
```
%1 = @llvm.spv.resource.handlefrombinding
%2 = @llvm.spv.resource.getpointer(%1, index)
load/store %2
```
These three llvm-ir instruction are treated as a single unit that will
1. Generate or find the global variable identified by the call to
`resource.handlefrombinding`.
2. Generate an OpLoad of the variable to get the handle to the image.
3. Generate an OpImageRead or OpImageWrite using that handle with the
given index.
This will generate the OpLoad in the same BB as the read/write.
Note: Now that `resource.handlefrombinding` is not processed on its own,
many existing tests had to be removed. We do not have intrinsics that
are able to use handles to sampled images, input attachments, etc., so
we cannot generate the load of the handle. These tests are removed for
now, and will be added when those resource types are fully implemented.
