[This
case](bc35715696/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp (L2815-L2818))
is not covered by any existing test (checked via code coverage and
inserting an `abort`). New test proposed that covers this line, as
demonstrated by test failure when an `abort` is present in that line.
[This
body](73768f88fb/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp (L857-L860))
is not covered by any existing test (checked via code coverage and
inserting an abort). New test proposed that covers this line, as
demonstrated by test failure when an abort is present in that line.
74e4694 moved ObjCARCContract from running before the codegen pipeline
into addISelPrepare(), which runs after PreISelIntrinsicLowering.
This broke ObjCARCContract's retainRV-to-claimRV optimization because
ObjCARCContract identifies ARC calls via intrinsics, not their lowered
counterparts.
This patch restores the pre-74e4694 ordering by moving ObjCARCContract
to addISelPasses.
The IntrinsicInst.cpp change looks extraneous but is required here:
ObjCARCContract may now rewrite the bundle operand from retainRV to
claimRV. When PreISelIntrinsicLowering then encounters this new
intrinsic use, lowerObjCCall asserts mayLowerToFunctionCall.
Assisted-by: claude
rdar://137997453
- Add CPP_for_OpenCL source language operand
- Handle opencl.cxx.version metadata
Align handling with SPIR-V translator logic and tests presented there
Add tests targeting specific Hexagon CodeGen passes with low coverage:
- peephole-sxtw-combine.mir: HexagonPeephole pass exercising SXTW
removal, combine generation, and LSR copy patterns. Improves
HexagonPeephole.cpp line coverage from 63.89% to 99.31%.
- vec-print-wq.ll: HexagonVectorPrint pass with V (single vector) and W
(double vector) register printing via 128b HVX. Improves
HexagonVectorPrint.cpp line coverage from 71.19% to 87.29%.
- tfr-cleanup-double-imm.mir: HexagonTfrCleanup pass exercising 64-bit
immediate rewrite paths. Improves HexagonTfrCleanup.cpp line coverage
from 80.85% to 88.30%.
- cfgopt-newpt-invert.ll: HexagonCFGOptimizer pass exercising branch
inversion with new-value predicate transfers.
This is a follow-up to commit 3ef59d80c5ce ("[Hexagon] Fix
use-before-def of AP register in prologue CSR spills").
When the AP (alignment pointer) register is used as a base register for
frame index elimination, add it to the basic block's livein set. This
ensures liveness information is accurate for the machine verifier.
The original commit fixed the use-before-def issue by moving PS_aligna
after CSR spills. However, when the prologepilog pass is run in
isolation (as in MIR tests) with expensive checks enabled, the verifier
reports an error because AP is used in blocks where it's not marked as
live-in.
In the full compilation pipeline, the Hexagon Packetizer adds AP as an
implicit operand to instruction bundles, which satisfies the verifier.
However, when running only the prologepilog pass (before packetization),
AP remains an explicit operand and must be in the livein set.
This fix adds AP to liveins when AP is used as the base register,
ensuring correct liveness tracking regardless of whether packetization
has run.
- The SPIR-V spec requires that OpConstantComposite must not reference
spec constant operands. When a composite contains non-constant
constitued, OpSpecConstantComposite should be emitted instead of
OpConstantComposite
- Avoid creating function pointer types when the
SPV_INTEL_function_pointers extension is unavailable, falling back to i8
as the pointee type
- Re-enable spirv-val validation in tests that previously failed due to
this issue, and un-XFAIL the block_w_struct_return and global_block
transcoding tests
related to #60133fixes#186756
The OpBitReverse is available when Shader or SPV_KHR_bit_instructions
extension is enabled. For targets without these capabilities, introduce
software emulation of G_BITREVERSE based on the parallel bit reversal
algorithm:
https://graphics.stanford.edu/~seander/bithacks.html#ReverseParallel
The emulation supports 8/16/32/64-bit scalars and vectors using bitwise
operations (shifts, AND, OR). A helper lambda avoids undefined behavior
when computing masks for 64-bit types.
Tests added for both emulation and native paths across all supported
types.
Assisted-by: Claude Code
The very first step towards #83422 - which will move DAG combines to be
processed in topological order.
There is a lot of churn on existing tests that need to be addressed
before this can be switched on globally, this patch gives the ability to
enable it both on a per-target basis, and via a command line option to
assist with testing and triage.
At the moment I'm focusing on addressing the x86 regressions (example in
the patch's basic test coverage) as that's the target I'm most familiar
with and will help with many other targets as well, but there might be
other/simpler targets that would benefit from earlier handling.
HexagonLoadStoreWidening pass was incorrectly using DoubleRegs registers
where IntRegs was required when widening load/store pairs. When the
S2_addasl_rrri instruction used a subreg (e.g., %2.isub_lo), the
widening pass extracted only the base register without preserving the
subreg, causing machine verifier errors.
PS_aligna initializes the AP register (eg:callee-saved R16) with an
aligned value derived from FP. It was being placed before the
CSR spills, causing the spill of R16 to save the AP value instead of the
caller's original R16, breaking the callee-saved register contract and
it must be defined before any AP-relative stack accesses. Fix by moving
PS_aligna to after all CSR spills in insertCSRSpillsInBlock().
Fixes#184531
Add uniform and divergent register bank legalization rules for the amdgcn_perm intrinsic (v_perm_b32). Since this is a VALU-only instruction, the uniform case maps the destination to UniInVgprB32 and all source operands to VgprB32.
Add register bank legalization rules for the amdgcn_permlane64 intrinsic
in the new RegBankLegalize framework.
After GISel legalization, permlane64 always operates on S32 — sub-32-bit
types are anyext'd to S32 and types wider than 32 bits are split into
S32 parts by legalizeLaneOp. Add rules for B32 type.
Also enable -new-reg-bank-select in the permlane64 lit test and update
affected check lines.
v_cvt_scalef32_2xpk16_fp6_f32 and v_cvt_scalef32_2xpk16_bf6_f32, as multipass instructions,
the destination operand must not overlap with any of the source operands.
In this work, we apply Constraints = "@earlyclobber $vdst" to these two instructions.
Fixes: LCCOMPILER-561
This reverts commit 08a4085. The change breaks `nvro.cpp` in the
debugging tests on the buildbot
(https://lab.llvm.org/buildbot/#/builders/46/builds/32873) but works
locally for me. It might be because the buildbot is using an older
Windows SDK.
In addition, it reverts parts of #188769 (using `.` over `->`).
Adds the `GroupMemoryBarrier()` HLSL function to SPIRV and DirectX with
additional tests for the different backends.
When this moves in, will create another PR with this as a template for
the other Barriers:
- `AllMemoryBarrier()` #99076
- `AllMemoryBarrierWithGroupSync()` #99090
- `DeviceMemoryBarrier()` #99105
- `DeviceMemoryBarrierWithGroupSync()` #99106
`Barrier()` does not have support for SPIRV, so I will exclude that from
the next PR.
- [x] Implement GroupMemoryBarrier clang builtin,
- [x] Link GroupMemoryBarrier clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for GroupMemoryBarrier to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for GroupMemoryBarrier to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/GroupMemoryBarrier.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/GroupMemoryBarrier-errors.hlsl
- [x] Create the int_dx_GroupMemoryBarrier intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_GroupMemoryBarrier to 80 in
DXIL.td
- [x] Create the GroupMemoryBarrier.ll and GroupMemoryBarrier_errors.ll
tests in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_GroupMemoryBarrier intrinsic in
IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the GroupMemoryBarrier
lowering and map it to int_spv_GroupMemoryBarrier in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/GroupMemoryBarrier.ll
<!-- branch-stack-start -->
<!-- branch-stack-end -->
If we we are going to legalize to a vector with the same element type
and mulh or mul_lohi are supported, allow the optimization before type
legalization.
RISC-V will widen vectors using vp.udiv/sdiv that doesn't support
division by constant optimization. In addition, type legalization will create
a build_vector with undef elements making it hard to match after type
legalization.
Other targets may need to widen by a combination of vector and scalar
divisions to avoid traps if we widen a vector with garbage.
I had to enable the MULHU->SRL DAG combine before type legalization to
prevent regressions. After type legalization, the multiply constant
build_vector will have undef elements and the combine won't trigger.
It's the a continuation of previously reverted
https://github.com/llvm/llvm-project/pull/178420
The patch removes custom AMDGPUISD::FFBH_I32 SelectionDAG node. Call
sites that need raw hardware semantics (LowerINT_TO_FP32, legalizeITOFP)
now use amdgcn_sffbh intrinsic directly. ISD::CTLS is added as a Custom
operation for i32.
Previous attempt had an issue:
The hardware v_ffbh_i32 instruction (v_cls_i32 on newer targets) has
different semantics than ISD::CTLS:
-sffbh returns [1, BitWidth-1] for normal values, -1 for
all-same-bits
-CTLS returns [0, BitWidth-2] for normal values, BitWidth-1 for
all-same-bits
Now LowerCTLS handles this by: sffbh -> umin(sffbh, BitWidth) -> sub 1.
Current patch also adds DAG combine to recognize the common CTLS idiom:
sub(ctlz(xor(x, sra(x, BitWidth-1))), 1) -> ctls(x)
and an optimization in performMinMaxCombine to fold away umin
when the input is not all-same-bits.
Partially addresses #177635
K is an unsigned, it will be zero extended to uint64_t for
the APInt constructor. If the ShSVT has more than 32 bits, we won't
create an all ones ConstantSDNode.
To fix this, explicitly push an all ones constant to KAmts. This
also fixes an APInt ImplicitTrunc.
This allows turnVectorIntoSplatVector to work for this case.
Modelling these as ll/sc style instructions as `memory(argmem: read)`
and `memory(argmem: write)` is not correct. The read/write additional
microarchitectural state, so they should at least read/write
`inaccessiblemem` as well.
Treating these are purely reading/writing can result in miscompilations
-- this came up in https://github.com/llvm/llvm-project/pull/169379
where the stcx ended up being hoisted outside the atomicrmw loop.
Add register bank legalize rules for the amdgcn_permlane16_var and
amdgcn_permlanex16_var intrinsics. All register operands (result, old,
src0, src1) map to VGPR since these are VALU lane permutation
operations.
Enable -new-reg-bank-select in the permlane16.var test.
This PR adds the patterns` // fold (A - (0 - B)) to (A + B)` and `//
fold A - (A - B) -> B` from SelectionDAG:
838be78e44/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (L4312)
These rewrites should not need HasOneUse checks, since even with
multiple uses, they cannot increase the instruction count.
In CodeView we had the limitation that we couldn't express locations
like `DW_OP_deref, DW_OP_plus_uconst 8` (i.e. indirect loads with an
offset). `S_DEFRANGE_REGISTER_REL_INDIR` allows us to represent this.
It's essentially `S_DEFRANGE_REGISTER_REL` (`Register + Offset`) with an
additional load afterward (`*(Register + Offset) + OffsetInUdt`).
These indirect locations are used in C++ 17 structured bindings and the
compiler generated C++ 20 coroutine stubs.
Before, locations that would only do a dereference without an added
offset afterward were represented by
`S_DEFRANGE_(REGISTER|FRAMEPOINTER)_REL` where the local had a reference
type:
```cpp
struct Foo {
int a;
int b;
};
int main() {
Foo f{1, 2};
auto &[a, b] = f;
// │ ╰─ Not present
// ╰─ S_LOCAL{ type: int&, FRAMEPOINTER_REL{ offset=0 } }
return a + b;
}
```
With this PR, both `a` and `b` will be present as non-reference types:
```cpp
// ...
int main() {
Foo f{1, 2};
auto &[a, b] = f;
// │ ╰─ S_LOCAL{ type: int, REGISTER_REL_INDIR{ register: RSP, offset: 0, offset in udt: 4 } }
// ╰─ S_LOCAL{ type: int, REGISTER_REL_INDIR{ register: RSP, offset: 0, offset in udt: 0 } }
return a + b;
}
```
One downside of this is that all variables like `a` now need a larger
record. If it used `FRAMEPOINTER_REL` before, it now takes 8 bytes more
(there's no `FRAMEPOINTER_REL_INDIR` where we could omit the register).
I removed the `UseReferenceType` workaround. If we need three
dereferences, that could be added back, but I don't know any construct
that uses this.
Closes#34392.
Update `optimizeIntraBlock` to use the `tryOptimizePair` method instead
of attempting to optimise directly. This unifies optimisation logic
between the intra- and cross-block paths and extends intra-block to
support the differs-by-two case
If Subtarget is null during global emission, it is now retrieved
from TargetMachine to prevent crashes caused by empty VTs in
wasmSymbolSetType.
Fixed: https://github.com/llvm/llvm-project/issues/181527
Add a tune feature that reverses the callee-saved register spill/restore
order so that addresses are ascending, enabling LD/ST fusion on
processors like SpacemiT X60.
Co-authored-by: LiqinWeng <liqin.weng@spacemit.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The VarIndex might come from (like load) another alloca which maybe
promoted before. The value will replaced in this case. WeakVH correctly
handles this.
Resolves: https://github.com/llvm/llvm-project/issues/171676
Related: https://github.com/llvm/llvm-project/pull/185117 (AArch64 side)
The issue points out that `Fold ADD(ADC(Y,0,W),X) -> ADC(X,Y,W)` is
optimized and that SBB can be optimized similarly:
`Fold ADD(SBB(Y,0,W),C) -> SBB(Y,-C,W)`.
With the changes from this branch, a new clang will compile the example
code:
```c
#include <stdint.h>
uint64_t f(uint64_t a, uint64_t b) {
uint64_t x;
x += __builtin_add_overflow(a, b, &x);
return x + 10;
}
uint64_t g(uint64_t a, uint64_t b) {
uint64_t x;
x -= __builtin_sub_overflow(a, b, &x);
return x + 10;
}
```
And it's optimized for the sub case as well, instead of emitting a leaq
on x86, it folds it in:
```asm
f:
movq %rdi, %rax
addq %rsi, %rax
adcq $10, %rax
retq
g:
movq %rdi, %rax
subq %rsi, %rax
sbbq $-10, %rax
retq
```
