Adds the `GroupMemoryBarrier()` HLSL function to SPIRV and DirectX with
additional tests for the different backends.
When this moves in, will create another PR with this as a template for
the other Barriers:
- `AllMemoryBarrier()` #99076
- `AllMemoryBarrierWithGroupSync()` #99090
- `DeviceMemoryBarrier()` #99105
- `DeviceMemoryBarrierWithGroupSync()` #99106
`Barrier()` does not have support for SPIRV, so I will exclude that from
the next PR.
- [x] Implement GroupMemoryBarrier clang builtin,
- [x] Link GroupMemoryBarrier clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for GroupMemoryBarrier to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for GroupMemoryBarrier to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/GroupMemoryBarrier.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/GroupMemoryBarrier-errors.hlsl
- [x] Create the int_dx_GroupMemoryBarrier intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_GroupMemoryBarrier to 80 in
DXIL.td
- [x] Create the GroupMemoryBarrier.ll and GroupMemoryBarrier_errors.ll
tests in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_GroupMemoryBarrier intrinsic in
IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the GroupMemoryBarrier
lowering and map it to int_spv_GroupMemoryBarrier in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/GroupMemoryBarrier.ll
<!-- branch-stack-start -->
<!-- branch-stack-end -->
The byte type was introduced to LLVM IR in #186888. DXILBitcodeWriter
needs to handle this type by generating valid LLVM 3.7 era bitcode,
which is as simple as treating byte equivalently to an integer.
This PR adds QuadReadAcrossY intrinsic support in HLSL with codegen for
both DirectX and SPIRV backends. Resolves
https://github.com/llvm/llvm-project/issues/99176.
- [x] Implement `QuadReadAcrossY` clang builtin,
- [x] Link `QuadReadAcrossY` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `QuadReadAcrossY` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `QuadReadAcrossY` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/QuadReadAcrossY.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/QuadReadAcrossY-errors.hlsl`
- [x] Create the `int_dx_QuadReadAcrossY` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_QuadReadAcrossY` to `123` in
`DXIL.td`
- [x] Create the `QuadReadAcrossY.ll` and `QuadReadAcrossY_errors.ll`
tests in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_QuadReadAcrossY` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `QuadReadAcrossY`
lowering and map it to `int_spv_QuadReadAcrossY` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/QuadReadAcrossY.ll`
Implments the Texture2D::operator[] method. It uses the same design as
Buffer::operator[]. However, this requires us to chagne the
resource_getpointer intrinsic to accept integer vectors for the index.
Assisted-by: Gemini
Fixes#187225.
The `TypedUAVLoadAdditionalFormats` shader flag was being set for all
TypedBuffer vector loads, so loading from a `Buffer<int64_t2>` was
incorrectly triggering this flag and causing the mismatch. This PR
changes it so the flag is only set for UAV loads.
this patch updates the shader flags to account for 0 being used to
represent unbounded arrays. This was a missed updated from the previous
pr #186022. This change is required to make sure the following offload
test pass dxv validation:
```
OffloadTest-clang-d3d12 :: Feature/ResourceArrays/multi-dim-unbounded-array-nuri.test
OffloadTest-clang-d3d12 :: Feature/ResourceArrays/multi-dim-unbounded-array.test
OffloadTest-clang-d3d12 :: Feature/ResourceArrays/unbounded-array-nuri.test
OffloadTest-clang-d3d12 :: Feature/ResourceArrays/unbounded-array.test
```
---------
Co-authored-by: Joao Saffran <jderezende@microsoft.com>
This PR adds QuadReadAcrossX intrinsic support in HLSL with codegen for
both DirectX and SPIRV backends. Resolves
https://github.com/llvm/llvm-project/issues/99175.
- [x] Implement QuadReadAcrossX clang builtin
- [x] Link QuadReadAcrossX clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for QuadReadAcrossX to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for QuadReadAcrossX to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/QuadReadAcrossX.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/QuadReadAcrossX-errors.hlsl
- [x] Create the int_dx_QuadReadAcrossX intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_QuadReadAcrossX to 123 in
DXIL.td
- [x] Create the QuadReadAcrossX.ll and QuadReadAcrossX_errors.ll tests
in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_QuadReadAcrossX intrinsic in IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the QuadReadAcrossX
lowering and map it to int_spv_QuadReadAcrossX in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/QuadReadAcrossX.ll
When `llvm.global_ctors` has no elements (e.g., when all resources are
unused in a shader library), its initializer is a `zeroinitializer`
(`ConstantAggregateZero`) rather than a `ConstantArray`. The previous
code used `cast<ConstantArray>` which asserts on incompatible types:
> "cast<Ty>() argument of incompatible type!"
This patch uses `dyn_cast` and returns early if the initializer is not a
`ConstantArray`, handling the edge case gracefully.
Fixes#178993.
Co-authored-by: Kaitlin Peng <kaitlinpeng@microsoft.com>
SPIRV backend uses 0 to represent unbounded arrays. This patch makes
unbounded resources be represented with 0 when binding them, as well as
makes sure the backend uses OpTypeRuntimeArray to represent such cases.
Fix: https://github.com/llvm/llvm-project/issues/183367
Fixes#184922
- [x] Implement `transpose` clang builtin in `Builtins.td`
- [x] Link `transpose` clang builtin with `hlsl_alias_intrinsics.h`
- [x] Add sema checks for `transpose` to `CheckHLSLBuiltinFunctionCall`
in `SemaHLSL.cpp`
- [x] Add codegen for `transpose` to `EmitHLSLBuiltinExpr` in
`CGHLSLBuiltins.cpp`
- `transpose` lowers to the `llvm.matrix.transpose` intrinsic
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/transpose.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/transpose-errors.hlsl`
- [x] Implement lowering of the `llvm.matrix.transpose` intrinsic in the
DXIL backend in `DXILIntrinsicExpansion.cpp`
- The intrinsic lowers to a shufflevector like in DXC
https://hlsl.godbolt.org/z/Gj959q6sq
- [x] Add DXIL lowering tests to
`llvm/test/CodeGen/DirectX/matrix-transpose.ll`
Assisted-by: claude-opus-4.6
If a vector has more than 4 elements, the `resource.load` and
`resource.store`
intrinsics aren't able to handle it. Split these into multiple calls.
Fixes#167542
Adds the WaveActiveBitOr intrinsic from issue #99167. This intrinsic
required a bit more work than the last intrinsics that I have done.
There are some peculiarities, which I verified with dxcompiler:
- WaveActiveBitOr only works on uint and uint64_t, no other types are
allowed
- There is no 16 bit version of WaveActiveBitOr
Followed the checklist:
- [x] Implement WaveActiveBitOr clang builtin,
- [x] Link WaveActiveBitOr clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for WaveActiveBitOr to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for WaveActiveBitOr to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/WaveActiveBitOr.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/WaveActiveBitOr-errors.hlsl
- [x] Create the int_dx_WaveActiveBitOr intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_WaveActiveBitOr to 120 in
DXIL.td
- [x] Create the WaveActiveBitOr.ll and WaveActiveBitOr_errors.ll tests
in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_WaveActiveBitOr intrinsic in IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the WaveActiveBitOr
lowering and map it to int_spv_WaveActiveBitOr in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveBitOr.ll
offload test: https://github.com/llvm/offload-test-suite/pull/487
PR #184882 was missing `half` type-specific overloads for `mul`.
This PR introduces `half` type-specific overloads for `mul` and
additional codegen tests for the half type.
Also added f16 tests for the lowering of llvm.matrix.multiply.
The offload test suite already has a `mul.fp16` test for exercising half
types at runtime, so no change is needed there.
Assisted-by: claude-opus-4.6
From issue #99165, adds the implementation of WaveActiveProduct. This
time with the new types for SPIRVTypeInst
- [x] Implement WaveActiveProduct clang builtin,
- [x] Link WaveActiveProduct clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for WaveActiveProduct to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for WaveActiveProduct to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/WaveActiveProduct.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/WaveActiveProduct-errors.hlsl
- [x] Create the int_dx_WaveActiveProduct intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_WaveActiveProduct to 119 in
DXIL.td
- [x] Create the WaveActiveProduct.ll and WaveActiveProduct_errors.ll
tests in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_WaveActiveProduct intrinsic in
IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the WaveActiveProduct
lowering and map it to int_spv_WaveActiveProduct in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveProduct.ll
Fixes#99138
- Defines a `__builtin_hlsl_mul` clang builtin in `Builtins.td`.
- Links the `__builtin_hlsl_mul` clang builtin with
`hlsl_alias_intrinsics.h` under the name `mul` for matrix cases
- Implement scalar and vector elementwise multiplication cases of the
`mul` function in `hlsl_intrinsics.h` and `hlsl_intrinsic_helpers.h`
- Adds sema for `__builtin_hlsl_mul` to `CheckBuiltinFunctionCall` in
`SemaHLSL.cpp`
- Adds codegen for `__builtin_hlsl_mul` to `EmitHLSLBuiltinExpr` in
`CGHLSLBuiltins.cpp`
- Vector-vector cases lower to `dot` (except double vectors, which
expands to scalar multiply-adds).
- Matrix-matrix, matrix-vector, and vector-matrix multiplication lower
to the `llvm.matrix.multiply` intrinsic
- Adds codegen tests to `clang/test/CodeGenHLSL/builtins/mul.hlsl`
- Adds sema tests to `clang/test/SemaHLSL/BuiltIns/mul-errors.hlsl`
- Implements lowering of the `llvm.matrix.multiply` intrinsic to DXIL in
`DXILIntrinsicExpansion.cpp`
Note: Currently the SPIRV backend does not support row-major matrix
memory layouts when lowering matrix multiply, and just assumes
column-major layout. Therefore this PR also makes the DirectX backend
only assume column-major layout. Implementing support for row-major
order shall be done in a separate PR. (Tracked by
https://github.com/llvm/llvm-project/issues/184906)
This PR locally passes the `mul` offload tests in both DirectX 12 and
Vulkan: https://github.com/llvm/offload-test-suite/pull/941
Assisted-by: claude-opus-4.6
This PR adds the WaveActiveAllEqual function to HLSL.
It also adds extra macro logic to CGHLSLBuiltins so that you can specify
a different intrinsic name for the SPIRV intrinsic.
Fixes https://github.com/llvm/llvm-project/issues/99162
This change resolves handles (or corresponding ptr) that all point into
a unique global resource by propagating an index into that global
resource through control flow.
If a unique global resource can't be resolved, an error is reported
instead.
This specifically resolves all handles that point into the same global
resource array.
Resolves: https://github.com/llvm/llvm-project/issues/165288
By reporting an error, this is part of resolving
https://github.com/llvm/llvm-project/issues/179303.
This patch removes optnone from HLSL entry functions and instead uses
`dx.disable_optimizations` module flag to know when to enable
`DisableOptimization` Shader Flag. This is part of: #167936
---------
Co-authored-by: Joao Saffran <jderezende@microsoft.com>
From issue #99165, adds the implementation of WaveActiveProduct. Mainly
followed how WaveActiveSum was implemented
- [x] Implement WaveActiveProduct clang builtin,
- [x] Link WaveActiveProduct clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for WaveActiveProduct to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for WaveActiveProduct to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/WaveActiveProduct.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/WaveActiveProduct-errors.hlsl
- [x] Create the int_dx_WaveActiveProduct intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_WaveActiveProduct to 119 in
DXIL.td
- [x] Create the WaveActiveProduct.ll and WaveActiveProduct_errors.ll
tests in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_WaveActiveProduct intrinsic in
IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the WaveActiveProduct
lowering and map it to int_spv_WaveActiveProduct in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveProduct.ll
Also created the [offload
tests](https://github.com/llvm/offload-test-suite/pull/486)
By marking `dx.resource.getpointer` as convergent, we are able to
prevent unwanted code transforms that make resource access look illegal.
Namely, this prevents the creation of a ptr into a resource handle to be
moved throughout control flow (eg in `SimplifyCFG`), further preventing
sink/hoist optimizations on the returned ptr during `InstCombine`.
Previously, these transforms were undone by the `phiNodeReplacement`
function in `dxil-resource-access`. However, in general, we would like
to follow a policy of preventing these transforms from occurring rather
than having to undo the work. This change is aligned with this.
This is a pre-requisite to resolving
https://github.com/llvm/llvm-project/issues/165288.
This PR adds WavePrefixProduct intrinsic support in HLSL with codegen
for both DirectX and SPIRV backends. Resolves
https://github.com/llvm/llvm-project/issues/99173.
- [x] Implement `WavePrefixProduct` clang builtin
- [x] Link `WavePrefixProduct` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `WavePrefixProduct` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `WavePrefixProduct` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/WavePrefixProduct.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/WavePrefixProduct-errors.hlsl`
- [x] Create the `int_dx_WavePrefixProduct` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_WavePrefixProduct` to `121`
in `DXIL.td`
- [x] Create the `WavePrefixProduct.ll` and
`WavePrefixProduct_errors.ll` tests in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_WavePrefixProduct` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In `SPIRVInstructionSelector.cpp` create the `WavePrefixProduct`
lowering and map it to `int_spv_WavePrefixProduct` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WavePrefixProduct.ll`
Issue: https://github.com/llvm/llvm-project/issues/99172
- [x] Implement `WavePrefixSum` clang builtin
- [x] Link `WavePrefixSum` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `WavePrefixSum` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `WavePrefixSum` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/WavePrefixSum.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/WavePrefixSum-errors.hlsl`
- [x] Create the `int_dx_WavePrefixSum` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_WavePrefixSum` to `121` in
`DXIL.td`
- [x] Create the `WavePrefixSum.ll` and `WavePrefixSum_errors.ll` tests
in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_WavePrefixSum` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `WavePrefixSum`
lowering and map it to `int_spv_WavePrefixSum` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WavePrefixSum.ll`
I also added a new macro
`GENERATE_HLSL_INTRINSIC_FUNCTION_SELECT_UNSIGNED` in conjunction with
the new function `getUnsignedIntrinsicVariant` to make selecting
unsigned variants of the intrinsic easier. As a result, I was able to
replace `getWaveActiveSumIntrinsic`, `getWaveActiveMaxIntrinsic`, and
`getWaveActiveMinIntrinsic` using the new macro.
The previous WaveActiveBallot implementation did not account for the
fact that the DXC implementation of the intrinsic returns a struct type
with 4 uints, rather than a vector of 4 uints. This must be respected,
otherwise the validator will reject the uses of WaveActiveBallot that
return a vector of 4 uints.
This PR updates the return type and adds the DXC-specific return type
`fouri32` to use for the intrinsic.
This PR adds a new CC1 option and a new dxc driver option. The DXC
option, when set, is translated into the new CC1 option.
The `all-resources-bound` dxc option will create a metadata module flag,
and the print-dx-shader-flags pass will set the appropriate shader
module flag from this metadata module flag.
Fixes https://github.com/llvm/llvm-project/issues/112264
Raw (as in ByteAddress) buffer accesses in DXIL must specify
ElementIndex as undef, and Structured buffer accesses must specify a
value. Ensure that we do this correctly in DXILResourceAccess, and
enforce that the operations are valid in DXILOpLowering.
Fixes#173316
This introduces the DXILMemIntrinsics pass and moves memset and memcpy
handling from DXILLegalize to here. We need to do this so that we can
handle memory intrinsics before the DXILResourceAccess pass so that we
can properly deal with arrays and large structures in resources.
Instead of trying to precalculate GEP offsets ahead of time and then
process resource accesses based off of these offsets, traverse the GEP
chain inline for each access. This makes it easier to get the types
correct when translating GEPs for cbuffer and structured buffer
accesses, which in turn lets us access individual elements of those
structures directly.
Fixes#160208, #164517, and #169430
Continuation of PR #169848 to address PR comments.
This PR makes the test more strict by adding CHECKs to ensure the loads
are indeed using the same or different GEPs.
Fixes an LLVM DirectX codegen test after it broke due to #169141
The CBuffer loads and GEPs are no longer duplicated when there are two
or more accesses within the same basic block.
This PR removes the duplicate check for CBuffer load and GEP from the
original test function `@f` and adds a new test function `@g` which
places duplicate CBuffer loads into separate basic blocks.
- The DXIL data scalarizer only needs to change vectors into arrays. It
does not need to change the types of GEPs to match the pointer type.
This PR simplifies the `visitGetElementPtrInst` method to do just that
while also accounting for nested GEPs from ConstantExprs. (Before this
PR, there were still vector types lingering in nested GEPs with
ConstantExprs.)
- The `equivalentArrayTypeFromVector` function was awkwardly placed near
the top of the file and away from the other helper functions. The
function is now moved next to the other helper functions.
- Removed an unnecessary `||` condition from `isVectorOrArrayOfVectors`
Related tests have also been cleaned up, and the test CHECKs have been
modified to account for the new simplified behavior.
Adds the fwidth intrinsic for HLSL.
The DXIL path only requires modification to the hlsl headers.
The SPIRV path implements the OpFwidth builtin in Clang and instruction
selection for the OpFwidth instruction in LLVM.
Also adds shader stage tests to the ddx_coarse and ddy_coarse
instructions used by fwidth.
Closes#99120
---------
Co-authored-by: Alexander Johnston <alexander.johnston@amd.com>
This change drops the use of the "Layout" type and instead uses explicit
padding throughout the compiler to represent types in HLSL buffers.
There are a few parts to this, though it's difficult to split them up as
they're very interdependent:
1. Refactor HLSLBufferLayoutBuilder to allow us to calculate the padding
of arbitrary types.
2. Teach Clang CodeGen to use HLSL specific paths for cbuffers when
generating aggregate copies, array accesses, and structure accesses.
3. Simplify DXILCBufferAccesses such that it directly replaces accesses
with dx.resource.getpointer rather than recalculating the layout.
4. Basic infrastructure for SPIR-V handling, but the implementation
itself will need work in follow ups.
Fixes several issues, including #138996, #144573, and #156084.
Resolves#147352.
