The DebugValueManager constructor scans the basic block to find all DEBUG_VALUE
uses of a def. This can lead to quadratic behavior as reported in #168326.
Instead, use MRI.use_instructions to find only the def's uses to limit the
search.
We still scan the BB so that we get the uses in order, and so we can stop
when a new def is found.
- Add CPP_for_OpenCL source language operand
- Handle opencl.cxx.version metadata
Align handling with SPIR-V translator logic and tests presented there
This is a follow-up to commit 3ef59d80c5ce ("[Hexagon] Fix
use-before-def of AP register in prologue CSR spills").
When the AP (alignment pointer) register is used as a base register for
frame index elimination, add it to the basic block's livein set. This
ensures liveness information is accurate for the machine verifier.
The original commit fixed the use-before-def issue by moving PS_aligna
after CSR spills. However, when the prologepilog pass is run in
isolation (as in MIR tests) with expensive checks enabled, the verifier
reports an error because AP is used in blocks where it's not marked as
live-in.
In the full compilation pipeline, the Hexagon Packetizer adds AP as an
implicit operand to instruction bundles, which satisfies the verifier.
However, when running only the prologepilog pass (before packetization),
AP remains an explicit operand and must be in the livein set.
This fix adds AP to liveins when AP is used as the base register,
ensuring correct liveness tracking regardless of whether packetization
has run.
- The SPIR-V spec requires that OpConstantComposite must not reference
spec constant operands. When a composite contains non-constant
constitued, OpSpecConstantComposite should be emitted instead of
OpConstantComposite
- Avoid creating function pointer types when the
SPV_INTEL_function_pointers extension is unavailable, falling back to i8
as the pointee type
- Re-enable spirv-val validation in tests that previously failed due to
this issue, and un-XFAIL the block_w_struct_return and global_block
transcoding tests
related to #60133fixes#186756
The OpBitReverse is available when Shader or SPV_KHR_bit_instructions
extension is enabled. For targets without these capabilities, introduce
software emulation of G_BITREVERSE based on the parallel bit reversal
algorithm:
https://graphics.stanford.edu/~seander/bithacks.html#ReverseParallel
The emulation supports 8/16/32/64-bit scalars and vectors using bitwise
operations (shifts, AND, OR). A helper lambda avoids undefined behavior
when computing masks for 64-bit types.
Tests added for both emulation and native paths across all supported
types.
Assisted-by: Claude Code
HexagonLoadStoreWidening pass was incorrectly using DoubleRegs registers
where IntRegs was required when widening load/store pairs. When the
S2_addasl_rrri instruction used a subreg (e.g., %2.isub_lo), the
widening pass extracted only the base register without preserving the
subreg, causing machine verifier errors.
PS_aligna initializes the AP register (eg:callee-saved R16) with an
aligned value derived from FP. It was being placed before the
CSR spills, causing the spill of R16 to save the AP value instead of the
caller's original R16, breaking the callee-saved register contract and
it must be defined before any AP-relative stack accesses. Fix by moving
PS_aligna to after all CSR spills in insertCSRSpillsInBlock().
Fixes#184531
Preserving all analyses likely is a wrong assertion since we do not know
what other analyses might exist that we've updated.
Updated to preserve CFG, LiveIntervalsAnalysis and SlotIndexAnalysis
(which is transitively preserved from LiveIntervalsAnalysis)
Add uniform and divergent register bank legalization rules for the amdgcn_perm intrinsic (v_perm_b32). Since this is a VALU-only instruction, the uniform case maps the destination to UniInVgprB32 and all source operands to VgprB32.
Add register bank legalization rules for the amdgcn_permlane64 intrinsic
in the new RegBankLegalize framework.
After GISel legalization, permlane64 always operates on S32 — sub-32-bit
types are anyext'd to S32 and types wider than 32 bits are split into
S32 parts by legalizeLaneOp. Add rules for B32 type.
Also enable -new-reg-bank-select in the permlane64 lit test and update
affected check lines.
v_cvt_scalef32_2xpk16_fp6_f32 and v_cvt_scalef32_2xpk16_bf6_f32, as multipass instructions,
the destination operand must not overlap with any of the source operands.
In this work, we apply Constraints = "@earlyclobber $vdst" to these two instructions.
Fixes: LCCOMPILER-561
This custom legalization was preserving splat values in widened
build_vector to allow the div by constant optimization to work.
We now allow division by constant optimization on narrow vector types
before type legalization so we no longer need this.
Adds the `GroupMemoryBarrier()` HLSL function to SPIRV and DirectX with
additional tests for the different backends.
When this moves in, will create another PR with this as a template for
the other Barriers:
- `AllMemoryBarrier()` #99076
- `AllMemoryBarrierWithGroupSync()` #99090
- `DeviceMemoryBarrier()` #99105
- `DeviceMemoryBarrierWithGroupSync()` #99106
`Barrier()` does not have support for SPIRV, so I will exclude that from
the next PR.
- [x] Implement GroupMemoryBarrier clang builtin,
- [x] Link GroupMemoryBarrier clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for GroupMemoryBarrier to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for GroupMemoryBarrier to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/GroupMemoryBarrier.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/GroupMemoryBarrier-errors.hlsl
- [x] Create the int_dx_GroupMemoryBarrier intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_GroupMemoryBarrier to 80 in
DXIL.td
- [x] Create the GroupMemoryBarrier.ll and GroupMemoryBarrier_errors.ll
tests in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_GroupMemoryBarrier intrinsic in
IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the GroupMemoryBarrier
lowering and map it to int_spv_GroupMemoryBarrier in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/GroupMemoryBarrier.ll
<!-- branch-stack-start -->
<!-- branch-stack-end -->
It's the a continuation of previously reverted
https://github.com/llvm/llvm-project/pull/178420
The patch removes custom AMDGPUISD::FFBH_I32 SelectionDAG node. Call
sites that need raw hardware semantics (LowerINT_TO_FP32, legalizeITOFP)
now use amdgcn_sffbh intrinsic directly. ISD::CTLS is added as a Custom
operation for i32.
Previous attempt had an issue:
The hardware v_ffbh_i32 instruction (v_cls_i32 on newer targets) has
different semantics than ISD::CTLS:
-sffbh returns [1, BitWidth-1] for normal values, -1 for
all-same-bits
-CTLS returns [0, BitWidth-2] for normal values, BitWidth-1 for
all-same-bits
Now LowerCTLS handles this by: sffbh -> umin(sffbh, BitWidth) -> sub 1.
Current patch also adds DAG combine to recognize the common CTLS idiom:
sub(ctlz(xor(x, sra(x, BitWidth-1))), 1) -> ctls(x)
and an optimization in performMinMaxCombine to fold away umin
when the input is not all-same-bits.
Partially addresses #177635
Modelling these as ll/sc style instructions as `memory(argmem: read)`
and `memory(argmem: write)` is not correct. The read/write additional
microarchitectural state, so they should at least read/write
`inaccessiblemem` as well.
Treating these are purely reading/writing can result in miscompilations
-- this came up in https://github.com/llvm/llvm-project/pull/169379
where the stcx ended up being hoisted outside the atomicrmw loop.
Add register bank legalize rules for the amdgcn_permlane16_var and
amdgcn_permlanex16_var intrinsics. All register operands (result, old,
src0, src1) map to VGPR since these are VALU lane permutation
operations.
Enable -new-reg-bank-select in the permlane16.var test.
Elevate findCondCodeUseOperandIdxForBranchOrSelect to a class member on
AArch64InstrInfo and use it in updateCondInstr to replace the hardcoded
switch on condition code operand indices.
Update `optimizeIntraBlock` to use the `tryOptimizePair` method instead
of attempting to optimise directly. This unifies optimisation logic
between the intra- and cross-block paths and extends intra-block to
support the differs-by-two case
If Subtarget is null during global emission, it is now retrieved
from TargetMachine to prevent crashes caused by empty VTs in
wasmSymbolSetType.
Fixed: https://github.com/llvm/llvm-project/issues/181527
Add a tune feature that reverses the callee-saved register spill/restore
order so that addresses are ascending, enabling LD/ST fusion on
processors like SpacemiT X60.
Co-authored-by: LiqinWeng <liqin.weng@spacemit.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The VarIndex might come from (like load) another alloca which maybe
promoted before. The value will replaced in this case. WeakVH correctly
handles this.
Resolves: https://github.com/llvm/llvm-project/issues/171676
Related: https://github.com/llvm/llvm-project/pull/185117 (AArch64 side)
The issue points out that `Fold ADD(ADC(Y,0,W),X) -> ADC(X,Y,W)` is
optimized and that SBB can be optimized similarly:
`Fold ADD(SBB(Y,0,W),C) -> SBB(Y,-C,W)`.
With the changes from this branch, a new clang will compile the example
code:
```c
#include <stdint.h>
uint64_t f(uint64_t a, uint64_t b) {
uint64_t x;
x += __builtin_add_overflow(a, b, &x);
return x + 10;
}
uint64_t g(uint64_t a, uint64_t b) {
uint64_t x;
x -= __builtin_sub_overflow(a, b, &x);
return x + 10;
}
```
And it's optimized for the sub case as well, instead of emitting a leaq
on x86, it folds it in:
```asm
f:
movq %rdi, %rax
addq %rsi, %rax
adcq $10, %rax
retq
g:
movq %rdi, %rax
subq %rsi, %rax
sbbq $-10, %rax
retq
```
The byte type was introduced to LLVM IR in #186888. DXILBitcodeWriter
needs to handle this type by generating valid LLVM 3.7 era bitcode,
which is as simple as treating byte equivalently to an integer.
This logic was previously embedded in NVPTXLowerArgs, but is split into
a separate pass as it is an optional optimization orthogonal to kernel
parameter lowering. This pass can be run earlier making alignment
available to generic IR passes (like infer-alignment) and a separate
pass makes the alignment logic independently testable. It also
simplifies NVPTXLowerArgs by removing the duplicate handling of
alignment for kernel vs. device functions.
This patch fixes the logic when lowering firsbithigh and low when
targeting spirv. It adds an aditional check to make sure the second half
is also -1, if it is it returns -1 otherwise returns the resulting sum.
Fix: #143003 and #143171
---------
Co-authored-by: Finn Plummer <mail@inbelic.dev>
When a f16 from a true16 insts is passed to a wmma, the src mod try to
pack it to a v4f16 using v_perm_b32. In true16 mode this is causing an
issue since v_perm_b32 takes vgpr32. Create a vgpr_32 for 16-bit src
before passing to v_perm_b32 in true16 mode so that the reg size
matched.
Ideailly we should use reg_sequence to replace v_perm_b32 in true16
mode. However, it currently hit a problem with bad code quality. With
current optimization it only shows better code quality when .hi16 is
selected in vector shuffle. Will fix it when reg allocator and coalescer
can reduce the extra mov
This PR adds QuadReadAcrossY intrinsic support in HLSL with codegen for
both DirectX and SPIRV backends. Resolves
https://github.com/llvm/llvm-project/issues/99176.
- [x] Implement `QuadReadAcrossY` clang builtin,
- [x] Link `QuadReadAcrossY` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `QuadReadAcrossY` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `QuadReadAcrossY` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/QuadReadAcrossY.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/QuadReadAcrossY-errors.hlsl`
- [x] Create the `int_dx_QuadReadAcrossY` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_QuadReadAcrossY` to `123` in
`DXIL.td`
- [x] Create the `QuadReadAcrossY.ll` and `QuadReadAcrossY_errors.ll`
tests in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_QuadReadAcrossY` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `QuadReadAcrossY`
lowering and map it to `int_spv_QuadReadAcrossY` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/QuadReadAcrossY.ll`
This adds two related changes to HLSL debug info support in the SPIR-V
backend. It's a first small step towards the plan I described in
https://discourse.llvm.org/t/hlsl-spirv-nsdi-debug-info-support-for-clang-dxc/90149.
## Tag HLSL shaders with `DW_LANG_HLSL` in the front-end
`GetSourceLanguage()` in `clang/lib/CodeGen/CGDebugInfo.cpp` checked
`LO.CPlusPlus` before `LO.HLSL`. Since HLSL is compiled as C++, the HLSL
check was never reached. Shaders compiled with `-g` were tagged with
`DW_LANG_C_plus_plus_14` instead of `DW_LANG_HLSL`. The NSDI pass
already had the correct mapping for `DW_LANG_HLSL` but it was never
triggered.
This fixes#136929 and #136995.
## Make `SPIRVEmitNonSemanticDI` activate automatically when `-g` is
used
`SPIRVPassConfig::addPreEmitPass()` only scheduled
`SPIRVEmitNonSemanticDI` when `--spv-emit-nonsemantic-debug-info` was
set or the target vendor was AMD. Passing `-g` to clang had no effect on
the SPIR-V backend pass.
The pass is now added unconditionally and self-activates by checking for
`llvm.dbg.cu` in the module. When no debug metadata is present it exits
early with no effect. This avoids the need to inspect module metadata at
pass-configuration time, which is not reliably available.
`--spv-emit-nonsemantic-debug-info` is now a deprecated synonym for
`-g`.
The alternative to the unconditional pass approach is to check at
pass-configuration time whether the module was compiled with debug info
(e.g. via `TargetOptions::DebugInfoForProfiling` or a similar flag
forwarded from the driver). I went with the unconditional approach
because it is simpler and the pass is cheap to enter and exit when no
`llvm.dbg.cu` is present.
I'm not sure whether adding a pass unconditionally is acceptable. Does
this sound reasonable, or would it be better to implement the
flag-forwarding approach?
Changes to tests:
- `clang/test/CodeGenHLSL/` (new): verifies that `-g` on an HLSL SPIR-V
target produces `DebugCompilationUnit` with language code 5
(`DW_LANG_HLSL`).
-
`llvm/test/CodeGen/SPIRV/debug-info/hlsl-debug-info-auto-activation.ll`
(new): verifies that a module with `llvm.dbg.cu` and `DW_LANG_HLSL`
produces `DebugCompilationUnit` without
`--spv-emit-nonsemantic-debug-info`.
- Existing `debug-compilation-unit.ll`, `debug-type-basic.ll`,
`debug-type-pointer.ll`: updated to verify NSDI is emitted whenever
debug metadata is present.
- `llc-pipeline.ll`: updated to reflect that `SPIRVEmitNonSemanticDI` is
now always in the pipeline.
---------
Co-authored-by: Eric Christopher <echristo@gmail.com>
