Adding SPIRV to LLVM_ALL_TARGETS
(https://github.com/llvm/llvm-project/pull/119653) revealed a series of
minor compilation problems and sanitizer complaints. This PR is to
address the problem.
Avoid `warning: enumerated mismatch in conditional expression:
'llvm::LoongArchISD::NodeType' vs 'llvm::ISD::NodeType'` while compiling
`LoongArchISelLowering.cpp`.
Inspired by https://reviews.llvm.org/D146600, this commit adds
some TTI hooks for LoongArch to make LoopDataPrefetch pass
really work. Including:
- `getCacheLineSize()`: 64 for loongarch64.
- `getPrefetchDistance()`: After testing SPEC CPU 2017, improvements
taken by prefetching are more obvious when set PrefetchDistance to
200(results shown blow), although different benchmarks fit for different
best choice.
- `enableWritePrefetching()`: store prefetch is supported by LoongArch,
so set WritePrefetching to true in default.
- `getMinPrefetchStride()` and `getMaxPrefetchIterationsAhead()` still
use default values: 1 and UINT_MAX, so not override them.
After this commit, the test added by https://reviews.llvm.org/D146600
can generate llvm.prefetch intrinsic IR correctly.
Results of spec2017rate benchmarks (testing date: ref, copies: 1):
- For all C/C++ benchmarks, compared to O3+novec/lsx/lasx, prefetch can
bring about -1.58%/0.31%/0.07% performance improvement for int
benchmarks and 3.26%/3.73%/3.78% improvement for floating point
benchmarks. (Only O3+novec+prefetch decreases when testing intrate.)
- But prefetch results in performance reduction almost for every Fortran
benchmark compiled by flang. While considering all C/C++/Fortran
benchmarks, prefetch performance will decrease about 1% ~ 5%.
FIXME: Keep `loongarch-enable-loop-data-prefetch` option default to
false for now due to the bad effect for Fortran.
Instruction `preld` is used to prefetch one cache-line of data from
memory in advance into the cache.
This commit allows it to be generated automatically.
Also add support for new relocation types required by debug information.
Constants have been taken from CodeView Symbolic Debug Information
Specification.
If linker relaxation enabled, relaxable code sequence expanded from
pseudos should avoid being separated by instruction scheduling. This
commit tags scheduling boundary for them to avoid being scheduled.
(Except for `tls_le/tls_ie` and `call36/tail36`. Because `tls_le/tls_ie`
can be scheduled and have no influence to relax, `call36/tail36` are
expanded later in `LoongArchExpandPseudo` pass.)
A new mask target-flag is added to attach relax relocs to the relaxable
code sequence. (No need to add it for `tls_le` and `call36/tail36`
because we can simply add relax relocs for them according to their
relocs. But for other code sequence, such as `PCALA_{HI20/LO12}`, we
must use the mask flag, mainly because relax should not be added when
code model is large.)
Because of the new mask target-flag, get "direct" flags is necessary
when using their target-flags. In addition, code sequence after being
optimized by `MergeBaseOffset` pass may not relaxable any more, so the
relax "bitmask" flag should be removed.
In the unlikely case where the stack size is greater than 4GB, we may run into
the situation where the local stack size and the callee saved registers stack
size get combined incorrectly when restoring the callee saved registers. This
happens because the stack size in shouldCombineCSRLocalStackBumpInEpilogue
is represented as an 'unsigned', but is passed in as an 'int64_t'. We end up with
something like
$fp, $lr = frame-destroy LDPXi $sp, 536870912
This change just makes 'shouldCombineCSRLocalStackBumpInEpilogue' match
'shouldCombineCSRLocalStackBump' where 'StackBumpBytes' is an 'uint64_t'
Attempt 32-bit broadcasts first, and then fallback to 64-bit broadcasts on failure.
We lose an explicit assertion for matching operand numbers but X86InstrFoldTables already does something similar.
Pulled out of WIP patch #73509
This fold already exists in a couple places (DAG and CGP), where an
icmps operands are swapped to allow CSE with a sub. They do not handle
constants though. This patch adds an AArch64 version that can be more
precise.
Avoid the use of getIntPtrConstant for anything other than address pointer related code.
Noticed while trying to use getVectorIdxConstant as a breakpoint.
Prevents infinite loop between combineBitcastToBoolVector and hoistLogicOpWithSameOpcodeHands, which only performs the "logicop(bitcast(A),bitcast(B)) -> bitcast(logicop(A,B))" upto type legalization.
combineBitcastToBoolVector doesn't care much as its mainly for AVX512 cleanup that X86DomainReassignment can't handle for us.
Fixes#123333
mayFoldIntoStore currently just checks the direct (oneuse) user of a
SDValue to check its stored, which prevents cases where we bitcast the
value prior to storing (usually the bitcast will be removed later).
This patch peeks up through oneuse BITCAST nodes chain to see if its
eventually stored.
The main use of mayFoldIntoStore is v8i16 EXTRACT_VECTOR_ELT lowering
which will only use PEXTRW/PEXTRB for index0 extractions (vs the faster
MOVD) if the extracted value will be folded into a store on SSE41+
targets.
Fixes#107086
I'm not adding tests for this, as I don't think we usually have tests to
verify correct description of defs and uses in instructions?
This fix will be tested when #122304 lands, as one of the regression
tests in that PR fails without this fix.
The insturction selector uses the `MachineFunction::copySalvageSSA`
function to insert `DBG_PHIs` or identify a defining instruction for a
copy-like instruction when finalizing Instruction References.
AArch64 has the ORR instruction which is a logical OR with the variants
ORRWrr which refers to a register to register variant, and ORRWrs which
is a register to a shifted register variant.
An ORRWrs where the shift amount is 0, and the zero register ($wzr) is
used is considered a copy, for example:
`$w0 = ORRWrs $wzr, killed $w3, 0`
However an ORRWrr with a zero register is not considered a copy
`$w0 = ORRWrr $wzr, killed $w3`
This causes an issue in the livedebugvalues pass because in aarch64-isel
the instruction is the ORRWrr variant, but is then changed to the ORRWrs
variant before the livedebugvalues pass.
This causes a mismatch between the two passes which leads to a crash in
the livedebugvalues pass.
This patch fixes the issue.
The code pattern that clang will generate for HLSL has changed from the
original plan. This allows the SPIR-V backend to generate code for the
current code generation.
It looks for patterns of the form:
```
%1 = @llvm.spv.resource.handlefrombinding
%2 = @llvm.spv.resource.getpointer(%1, index)
load/store %2
```
These three llvm-ir instruction are treated as a single unit that will
1. Generate or find the global variable identified by the call to
`resource.handlefrombinding`.
2. Generate an OpLoad of the variable to get the handle to the image.
3. Generate an OpImageRead or OpImageWrite using that handle with the
given index.
This will generate the OpLoad in the same BB as the read/write.
Note: Now that `resource.handlefrombinding` is not processed on its own,
many existing tests had to be removed. We do not have intrinsics that
are able to use handles to sampled images, input attachments, etc., so
we cannot generate the load of the handle. These tests are removed for
now, and will be added when those resource types are fully implemented.
To deduce whether the optimization is legal we need to compare the target
features between caller and callee versions. The criteria for bypassing
the resolver are the following:
* If the callee's feature set is a subset of the caller's feature set,
then the callee is a candidate for direct call.
* Among such candidates the one of highest priority is the best match
and it shall be picked, unless there is a version of the callee with
higher priority than the best match which cannot be picked from a
higher priority caller (directly or through the resolver).
* For every higher priority callee version than the best match, there
is a higher priority caller version whose feature set availability
is implied by the callee's feature set.
Example:
Callers and Callees are ordered in decreasing priority.
The arrows indicate successful call redirections.
Caller Callee Explanation
=========================================================================
mops+sve2 --+--> mops all the callee versions are subsets of the
| caller but mops has the highest priority
|
mops --+ sve2 between mops and default callees, mops wins
sve sve between sve and default callees, sve wins
but sve2 does not have a high priority caller
default -----> default sve (callee) implies sve (caller),
sve2(callee) implies sve (caller),
mops(callee) implies mops(caller)
This is already defined for each register class in AArch64RegisterInfo,
not hardcoding it here makes these values easier to change (perhaps
based on hardware mode).
This commit add relax relocations for `tls_le` code sequence.
Handwritten assembly and generating source code by clang are both
affected.
Scheduled `tls_le` code sequence can be relaxed normally and we can add
relax relocs when code emitting according to their relocs. Other
relaxable macros' code sequence cannot simply add relax relocs according
to their relocs, such as `PCALA_{HI20/LO12}`, we do not want to add
relax relocs when code model is large. This will be implemented in later
commit.
Emit `R_LARCH_RELAX` relocations when expanding some macros, including:
- `la.tls.ie`, `la.tls.ld`, `la.tls.gd`, `la.tls.desc`,
- `call36`, `tail36`.
Other macros that need to emit `R_LARCH_RELAX` relocations was
implemented in https://github.com/llvm/llvm-project/pull/72961, including:
- `la.local`, `la.pcrel`, `la.pcrel` expanded as `la.abs`, `la`,
`la.global`, `la/la.global` expanded as `la.pcrel`, `la.got`.
Note: `la.tls.le` macro can be relaxed when expanded with
`R_LARCH_TLS_LE_{HI20/ADD/LO12}_R` relocations. But if we do so,
previously handwritten assembly code will occur error due to the
redundant `add.{w/d}` followed by `la.tls.le`. So `la.tls.le` keeps to
expands with `R_LARCH_TLS_LE_{HI20/LO12}`.
This patch fixes:
llvm/lib/Target/AMDGPU/SIISelLowering.cpp:13908:46: error:
comparison of integers of different signs: 'uint32_t' (aka 'unsigned
int') and 'int' [-Werror,-Wsign-compare]
The intention is to use a "copy" instead of a "sub" to handle the high
parts of 64-bit multiply for this specific case.
This unlocks copy prop use cases where the copy can be reused by later
multiply+add sequences if possible.
Fixes: SWDEV-487672, SWDEV-487669
Once again we have excessive TLI hooks with bad defaults. Permit this
for 32-bit element vectors, which are just use-different-register.
We should permit 16-bit vectors as cheap with legal packed instructions,
but I see some mixed improvements and regressions that need investigation.
The motivation for this is to allow us to match strided accesses that
are emitted from the loop vectorizer with EVL tail folding (see #122232)
In these loops the step isn't loop invariant and is based off of
@llvm.experimental.get.vector.length.
We can relax this as long as we make sure to construct the updates after
the definition inside the loop, instead of the preheader.
I presume the restriction was previously added so that the step would
dominate the insertion point in the preheader. I can't think of why it
wouldn't be safe to calculate it in the loop otherwise.
I have a particular user downstream who likes to write shuffles in terms
of unions involving _BitInt(128) types. This isn't completely crazy
because there's a bunch of code in the wild which was written with SSE
in mind, so 128 bits is a common data fragment size.
The problem is that generic lowering scalarizes this to ELEN, and we end
up with really terrible extract/insert sequences if the i128 shuffle is
between other (non-i128) operations.
I explored trying to do this via generic lowering infrastructure, and
frankly got lost. Doing this a target specific DAG is a bit ugly -
really, there's nothing hugely target specific here - but oh well. If
reviewers prefer, I could probably phrase this as a generic DAG combine,
but I'm not sure that's hugely better. If reviewers have a strong
preference on how to handle this, let me know, but I may need a bit of
help.
A couple notes:
* The argument passing weirdness is due to a missing combine to turn a
build_vector of adjacent i64 loads back into a vector load. I'm a bit
surprised we don't get that, but the isel output clearly has the
build_vector at i64.
* The splat case I plan to revisit in another patch. That's a relatively
common pattern, and the fact I have to scalarize that to avoid an
infinite loop is non-ideal.
Support true16 format for v_cndmask_b16 in MC and CodeGen in true16 and
fake16 flow.
Since we are replacing `v_cndmask_b16` to `v_cndmask_b16_t16/fake16`, we
have to at least update the fake16 codeGen to get codeGen test passing.
For this case, we have to update the true16 and with fake16 together,
otherwise some of the true16 tests will fail
- Add support for `@llvm.exp2()`:
- LLVM: `float` -> PTX: `ex2.approx{.ftz}.f32`
- LLVM: `half` -> PTX: `ex2.approx.f16`
- LLVM: `<2 x half>` -> PTX: `ex2.approx.f16x2`
- LLVM: `bfloat` -> PTX: `ex2.approx.ftz.bf16`
- LLVM: `<2 x bfloat>` -> PTX: `ex2.approx.ftz.bf16x2`
- Any operations with non-native vector widths are expanded. On
targets not supporting f16/bf16, values are promoted to f32.
- Add *CONDITIONAL* support for `@llvm.log2()` [^1]:
- LLVM: `float` -> PTX: `lg2.approx{.ftz}.f32`
- Support for f16/bf16 is emulated by promoting values to f32.
[1]: CUDA implements `exp2()` with `ex2.approx` but `log2()` is
implemented differently, so this is off by default. To enable, use the
flag `-nvptx-approx-log2f32`.
