From the legalizer, G_ATOMIC_CMPXCHG is legal with local and region pointers.
Global and flat pointers are converted into G_AMDGPU_ATOMIC_CMPXCHG. Implement
rules for both variants.
The basic register allocator's priority queue exhibited
non-deterministic behavior when multiple LiveIntervals had identical
spill weights. This caused different register allocation decisions
between ASAN and non-ASAN builds, leading to spurious test failures on
some internal tests.
Root Cause:
The CompSpillWeight comparator only compared spill weights:
```
return A->weight() < B->weight();
```
When two LiveIntervals had equal weights, the comparator returned false
for both comp(A,B) and comp(B,A), making them equivalent in the heap
ordering. The C++ standard does not specify the relative order of
equivalent elements in a heap. In practice, the heap's internal
structure for equivalent elements depends on implementation details and
memory layout. ASAN changes memory layout, leading to different valid
heap configurations and thus different dequeue order.
Fix:
Add an explicit stable tie-breaker using virtual register numbers:
```
return std::tuple(A->weight(), A->reg()) <
std::tuple(B->weight(), B->reg());
```
Register numbers are stable integers independent of memory layout,
ensuring deterministic behavior across all build configurations. When
spill weights are equal, higher-numbered virtual registers are allocated
first, which is equally valid and deterministic.
Test Updates:
Updated test expectations in
CodeGen/AMDGPU/register-killed-error-after-alloc-failure0.mir to reflect
the new deterministic allocation order. When LiveIntervals have equal
spill weights, the allocator now consistently processes them in register
number order rather than in an implementation-defined order.
This instruction doesn't use VL and the pseudo doesn't have a VL
operand. According to VSETVLIInserter tests, this instruction only gets
implicit $vtype added to it.
Fixes https://github.com/llvm/llvm-project/issues/159116
Prevent XXPERMDI splat optimization when the splat output register is
used in other instructions, which caused undefined register references.
Also track removed ISEL operands in simplifyToLI to prevent
LiveVariables corruption during ISEL-to-COPY conversion.
On one hand, this patch adds a couple of new testcases related to
SPV_KHR_bit_instructions. On the other hand, it fixes a couple of
testcases that were using `llvm.bitreverse.*`, while they were supposed
to use the SPIRV-friendly equivalent.
---------
Co-authored-by: Marcos Maronas <marcos.maronas@intel.com>
AArch64 has enabled this in https://reviews.llvm.org/D138990, and
the measurement data still stands for RISCV in some cases.
And, similar optimization like #77284 is added too.
After this patch, the highly predictable branch will be converted
back to branches instead using selects.
This pass is disabled by default now, we can enable it by default
after more detailed investigation.
Reviewers: davemgreen, preames, dtcxzyw, lukel97, topperc, asb
Pull Request: https://github.com/llvm/llvm-project/pull/80124
Since we started optimizating always-true branches in the AArch64
backend (like cbz wzr), shrink wrap has been exposed to some block
structures that it does not handle correctly, usually with unreachable
blocks. This prevents the call to FindIDom/findNearestCommonDominator
from failing when looking at the predecessors of a loop if one is
unreachable.
Fixes: #177866
I did not replace riscv.clmulh/clmulr since those require a multiple
instruction pattern match. I wanted to ensure that -O0 will select the
correct instructions without relying on combines.
Reverts llvm/llvm-project#177459
`mlir-nvidia` and `mlir-nvidia-gcc7` Buildbots are failing.
The blamelist is small and likely because of my change. Preemptively
reverting.
Original commit message:
> When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>
>When no SM or PTX version is specified, we now use PTX 3.2 (the minimum
for the default SM 3.0) instead of PTX 6.0.
---
The following commits should fix the failures that arose when I
previously tried to land this commit:
- 9fc5fd0ad689eed94f65b1d6d10f9c5642935e68 should address the
`llvm-nvptx*-nvidia-*` build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
- 600514a63760c6730e4cd970d2fcead9c5a897b3 should address the MLIR
failures
- Make v8f16 a legal type so that arguments can be passed in vector
registers. Handle fp16 vectors so that they have the same ABI as other
fp vectors.
- Set the preferred vector action for fp16 vectors to "split". This will
scalarize all operations, which is not always necessary (like with
memory operations), but it avoids the superfluous operations that result
after first widening and then scalarizing a narrow vector (like v4f16).
Fixes#168992
When analyzing outliner candidates, there is no check that the tail-call
expansion register is live across the candidate call site. That can
result in a situation where the original function sets the volatile
register and uses it in the section that gets outlined. This of course
results in the use of the register receiving the incorrect value. Namely
the address of the outlined function since that is what the tail-call
sequence placed in the register.
Support GlobalISel and switch to checking `nnan` flag on instruction
instead of TargetOptions.
Instruction are renamed to v_cvt_floor and v_cvt_nearest on gfx11+
so add gfx11 tests as well.
Previously, for performance reasons, we would only use precise lifetimes
if they cover all reachable exits. Now, if they do not, we use precise
lifetimes in addition to untagging at every exit that is not dominated
by them.
This is the behavior of ASan.
Reviewers: vitalybuka, pcc
Pull Request: https://github.com/llvm/llvm-project/pull/174875
The code in the test is causing a crash in `SIInstrInfo.cpp`
`fixImplicitOperands()` in `MI.implicit_operands()`:
```
for (auto &Op : MI.implicit_operands()) {
```
MachineInstr.h:
```
mop_range implicit_operands() {
=> return operands_impl().drop_front(getNumExplicitOperands());
}
```
We are trying to drop 1 operand from the operands of MI which are 0.
By early returning we are no longer crashing at that point and we are
getting a more meaningful error message:
```
*** Bad machine code: Too few operands ***
- function: missing_operand_crash
- basic block: %bb.0 (0x5a9d30ced988)
- instruction: S_WAITCNT_DEPCTR
1 operands expected, but 0 given.
```
The code is still crashing at a different location, but at least we are
getting an error message.
In practice when legalizeOperands is called on a PHI node, the result is
never an SGPR class and the operands are never subregs. Simplify the
code accordingly by using the result regclass for all the inputs. This
includes using an AV class where previously we picked either an AGPR or
VGPR class.
Also includes a kind of hacky, minimal change to avoid assertions
when softPromoteHalfType is removed to fix kernel arguments
lowered as f16. Half support was never really implemented
for r600, and there just happened to be a few incidental tests
which included a half argument (which were also not even meaningful,
since the function body just folded to nothing due to no callable
function support).
This is a followup to https://github.com/llvm/llvm-project/pull/171288,
which removed lowering of libcalls to SDAG nodes for most libcalls that
get unconditionally canonicalized to intrinsics. This handles the
remaining fabs case, which I originally skipped due to larger test
impact.
Peek through free scalar_to_vector/extract_vector_elt pairs (of the same
vector width) that might still persist (ideally a better topological
sorting would have removed these already....)
`EstimateFunctionSizeInBytes`, in `ARMFrameLowering.cpp`, provides an
early estimate of the compiled size of a function, in a context that
wants to overestimate rather than underestimate.
In some cases it was underestimating severely, by over 20%. The
discrepancy was entirely accounted for by the fact that `COPY`
operations were not being counted at all, even though each one (or at
least each one that survives any post-regalloc optimizations) takes 2
bytes in Thumb or 4 in Arm. This could lead to a compile failure, if the
underestimated function size led frame lowering to not stack LR, but
later, `ARMConstantIslandsPass` needed to insert an intra-function
branch long enough to require a `bl` instruction, needing LR to have
been stacked.
The result of `EstimateFunctionSizeInBytes` was not directly available
for testing, so I added an `LLVM_DEBUG` at the end of the function. That
way, the test file doesn't need to try to make a >2048 byte function
estimated at <2048 bytes; it just needs to exhibit a function with a
single `COPY` and make sure it's counted.
At the moment, `EstimateFunctionSizeInBytes` is only used at all in
Thumb-1 compilations, to decide whether the function is large enough to
justify stacking LR as a precaution. However, the subroutine
`ARMBaseInstrInfo::getInstSizeInBytes` which counts each individual
`MachineInstr` is called from other contexts too, so I've made it return
a sensible answer for `COPY` nodes in both of Arm and Thumb.
This adds the initial feature for the base RVY extension,
other extensions such as the hybrid mode will be added later.
RVY specification: https://riscv.github.io/riscv-cheri/
Co-authored-by: Jessica Clarke <jrtc27@jrtc27.com>
Co-authored-by: Petr Vesely <petr.vesely@codasip.com>
Pull Request: https://github.com/llvm/llvm-project/pull/176870
This restriction was originally added in
https://reviews.llvm.org/D143256, with the given justification:
> Currently, in TargetLowering, if the target does not support fminnum,
we lower to fminimum if neither operand could be a NaN. But this isn't
quite correct because fminnum and fminimum treat +/-0 differently; so,
we need to prove that one of the operands isn't a zero.
As far as I can tell, this was never correct. Before
https://github.com/llvm/llvm-project/pull/172012, `minnum` and `maxnum`
were nondeterministic with regards to signed zero, so it's always been
perfectly legal to lower them to operations that order signed zeroes.
