This patch implements the SPIR-V lowering for the following HLSL
intrinsics:
- SampleBias
- SampleGrad
- SampleLevel
- SampleCmp
- SampleCmpLevelZero
It defines the required LLVM intrinsics in 'IntrinsicsDirectX.td' and
'IntrinsicsSPIRV.td'.
It updates 'SPIRVInstructionSelector.cpp' to handle the new intrinsics
and
generates the correct 'OpImageSample*' instructions with the required
operands
(Bias, Grad, Lod, ConstOffset, MinLod, etc.).
CodeGen tests are added to verify the implementation for images with
dimension 1D, 2D, 3D, and Cube.
Assisted-by: Gemini
selectConst() was asserting for constants wider than 64 bits. Add APInt
overloads of getOrCreateConstInt and getOrCreateConstVector that avoid
the uint64_t truncation.
`constrainSelectedInstRegOperands` always returns `true`; so it can be
safely transformed to return `void` instead.
A follow-up patch should update `MachineInstrBuilder::constrainAllUses`.
According to SPIRV:
```
PackedVectorFormat4x8Bit (PackedVectorFormat4x8BitKHR)
Interpret 32-bit scalar integer operands as vectors of four 8-bit
components. Vector components follow byte significance order with the
lowest-numbered component stored in the least significant byte.
```
And in OpSDot / OpUDot:
```
When Vector 1 and Vector 2 are scalar integer types, Packed Vector
Format must be specified to select how the integers are to be
interpreted as vectors.
```
Issue: https://github.com/llvm/llvm-project/issues/99172
- [x] Implement `WavePrefixSum` clang builtin
- [x] Link `WavePrefixSum` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `WavePrefixSum` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `WavePrefixSum` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/WavePrefixSum.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/WavePrefixSum-errors.hlsl`
- [x] Create the `int_dx_WavePrefixSum` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_WavePrefixSum` to `121` in
`DXIL.td`
- [x] Create the `WavePrefixSum.ll` and `WavePrefixSum_errors.ll` tests
in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_WavePrefixSum` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `WavePrefixSum`
lowering and map it to `int_spv_WavePrefixSum` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WavePrefixSum.ll`
I also added a new macro
`GENERATE_HLSL_INTRINSIC_FUNCTION_SELECT_UNSIGNED` in conjunction with
the new function `getUnsignedIntrinsicVariant` to make selecting
unsigned variants of the intrinsic easier. As a result, I was able to
replace `getWaveActiveSumIntrinsic`, `getWaveActiveMaxIntrinsic`, and
`getWaveActiveMinIntrinsic` using the new macro.
For compute we don't run structurizer hence we won't be able to preserve
loop metadata via LoopMerge instruction. So
SPV_INTEL_unstructured_loop_controls is the only way we can preserve the
info in unstructured control flow.
The ReduceMax, ReduceMin, and ReduceSum selectors were all doing the
samething with the exception of which opcode they were using.
This change unifies these implementations and allows pick the opcode via
a helper lambda.
The extension adds support for the `OpFmaKHR` instruction, which
provides a native SPIR-V instruction for fused multiply-add operations
as an alternative to using OpenCL.std::Fma extended instruction.
Translate both LLVM fma intrinsics as well as OCL builtins to `OpFmaKHR`
if the extension is available.
Specification:
https://github.khronos.org/SPIRV-Registry/extensions/KHR/SPV_KHR_fma.html
This patch implements the `sample` and `sample_clamp` intrinsics for
HLSL
resources in the SPIR-V backend. It adds the necessary intrinsic
definitions
in `IntrinsicsDirectX.td` and `IntrinsicsSPIRV.td`, and implements the
instruction selection logic in `SPIRVInstructionSelector.cpp`.
Key changes:
- Added `int_dx_resource_sample` and `int_dx_resource_sample_clamp`
intrinsics.
- Added `int_spv_resource_sample` and `int_spv_resource_sample_clamp`
intrinsics.
- Implemented `selectSampleIntrinsic` to handle
`OpImageSampleImplicitLod` generation.
- Added `ResourceDimension` enum in `DXILABI.h` and `HLSLResource.h`.
- Added a new test case
`llvm/test/CodeGen/SPIRV/hlsl-resources/Sample.ll` to verify the
implementation.
The previous WaveActiveBallot implementation did not account for the
fact that the DXC implementation of the intrinsic returns a struct type
with 4 uints, rather than a vector of 4 uints. This must be respected,
otherwise the validator will reject the uses of WaveActiveBallot that
return a vector of 4 uints.
This PR updates the return type and adds the DXC-specific return type
`fouri32` to use for the intrinsic.
SPIR-V doesn't currently have a way to encode the `externally_initialized` LLVM concept. Unfortunately, certain HIP constructs (e.g. `__managed__` or `__constant__` variables) rely on it. Hence, this patch allows AMDGCN flavoured SPIR-V to encode `externally_initialized` via the `HostAccessINTEL` decoration, which approximates some of the semantics; the decoration is handled during reverse translation. It also appears that we never fully implemented the decoration, which I intend to handle in a subsequent, independent patch.
#166494 added support for using `inttoptr` in global initialisation, and
lowering int into `OpSpecConstantOp OpConvertUToPtr`. Unfortunately, it
slightly more subtle case / exposed an existing issue around the `COPY`
pseudo-op. This patch ensures that we glance through a `COPY` when
figuring out whether an `OpConvertUToPtr` is actually operating on a
global. We also correctly handle the case where a `G_PTR_ADD` is used by
an `OpSpecConstantOp` in the context of global initialisation, which
would otherwise lead to broken SPIR-V wherein the latter would reference
a non constant Op.
---------
Co-authored-by: Marcos Maronas <marcos.maronas@intel.com>
This patch handles the special case where an extract value yields an
aggregate result, which then is used as an argument to a store. The
SPIRV BE uses special intrinsics (`spv_extractv` and `spv_store`) to
represent these through IRTranslator, however this creates a problem:
`spv_store` is called as a function, and IRTranslator cannot handle
arguments that take more than a vreg. For other functions, the aggregate
argument replacement pass would have solved things, but it does not
apply here. Hence, we apply the same mutate-into-Int32 solution here
when dealing with stores, and restore the extract value's type (which we
have available as a ValueAttr) during instruction selection.
Implements initial support for vk::push_constant.
As is, this allows handling simple push constants, but has one
main issue: layout can be incorrect (See #168401). The layout
issue being not only push-constant related, it's ignored for this PR.
The frontend part of the implementation is straightforward:
- adding a new attribute
- when targeting vulkan/spirv, we process it
- global variables with this attribute gets a new AS:
hlsl_push_constant
The IR has nothing specific, only some RO globals in this new AS.
On the SPIR-V side, we not convert this AS into a PushConstant storage
class. But this creates some issues: the variables in this storage class
must have a specific set of decoration to define their layout.
Current infra to create the SPIR-V types lacks the context required to
make this decision: no indication on the AS or context around the type
being created. Refactoring this would be a heavy task as it would
require getting this information in every place using the GR for type
creation.
Instead, we do something similar to CBuffers:
- find all globals with this address space, and change their type to
a target-specific type.
- insert a new intrinsic in place of every reference to this global
variable.
This allow the backend to handle both layout variables loads and type
lowering independently.
Type lowering has nothing specific: when we encounter a target extension
type with spirv.PushConstant, we lower this to the correct SPIR-V type
with the proper offset & block decorations.
As for the intrinsic, it's mostly a no-op, but required since we have
this target-specific type.
Note: this implementation prevents the static declaration of multiple
push constants in a single shader module. The actual specification is
more relaxed: there can be only one **used** push constant block per
entrypoint. To correctly implement this, we'd require to keep some
additional state to determine the list of statically used resources per
entrypoint. This shall be addressed as a follow-up (see #170310)
Implements the ddx and ddy HLSL intrinsics. The DXIL intrinsics call the
ddx_coarse and ddy_builtins as they lower to the coarse variants.
The SPIRV intrinsics lower to their own opcodes OpDPdx and OpDPdy. Tests
are added to ensure the SPIRV builtins are not available outside of
shaders stage.
Closes https://github.com/llvm/llvm-project/issues/99096
Closes https://github.com/llvm/llvm-project/issues/99099
This adds support for the `SPV_NV_shader_atomic_fp16_vector` extension,
and then uses it to enable lowering of atomic add, sub, min and max on 2
and 4 component vectors of FP16, which are rather common options in ML
workloads. Even though `bfloat16` also works in practice, we do not
enable it since it's not specified in the extension (which might need
updating / promoting to KHR at least). A `TODO` is also inserted in
`SPIRVModuleAnalysis.cpp' regarding the need to upgrade its ample usage
of `report_fatal_error`; I have a WiP patch for that, but it still needs
a bit of baking. Finally, a paired patch will be necessary in the
Translator, as it's not aware of the extension either - I'll update this
review to reference the PR once I create it.
This patch introduces the necessary infrastructure to legalize vector
operations on vectors that are longer than what the SPIR-V target
supports. For instance, shaders only support vectors up to 4 elements.
The legalization is done by splitting the long vectors into smaller
vectors of a legal size.
Specifically, this patch does the following:
- Introduces `vectorElementCountIsGreaterThan` and
`vectorElementCountIsLessThanOrEqualTo` legality predicates.
- Adds legalization rules for `G_SHUFFLE_VECTOR`,
`G_EXTRACT_VECTOR_ELT`,
`G_BUILD_VECTOR`, `G_CONCAT_VECTORS`, `G_SPLAT_VECTOR`, and
`G_UNMERGE_VALUES`.
- Handles `G_BITCAST` of long vectors by converting them to
`@llvm.spv.bitcast` intrinsics which are then legalized.
- Updates `selectUnmergeValues` to handle extraction of both scalars
and vectors from a larger vector, using `OpCompositeExtract` and
`OpVectorShuffle` respectively.
Fixes https://github.com/llvm/llvm-project/pull/165444
This commit modifies the SPIRV instruction selector to emit
`OpCopyMemory`
instead of `OpCopyMemorySized` when generating SPIRV for logical
addressing.
Previously, `G_MEMCPY` was translated to `OpCopyMemorySized`, which
requires an
explicit size operand. However, for logical SPIRV, the size of the
pointee type
is implicitly known. This change ensures that `OpCopyMemory` is used,
which is
more appropriate for logical SPIRV and aligns with the SPIR-V
specification for
logical addressing.
The instruction selection pass for SPIR-V now performs dead code
elimination (DCE).
This change removes unused instructions, leading to more optimized
SPIR-V output.
As a consequence of this, several tests were updated to ensure their
continued
correctness and to prevent previously tested code from being optimized
away.
Specifically:
- Many tests now store computed values into global variables to ensure
they are
not eliminated by DCE, allowing their code generation to be verified.
- The test `keep-tracked-const.ll` was removed because it no longer
tested
its original intent. The check statements in this test were for
constants
generated when expanding a G_TRUNC instruction, which is now removed by
DCE
instead of being expanded.
- A new test, `remove-dead-type-intrinsics.ll`, was added to confirm
that dead
struct types are correctly removed by the compiler.
These updates improve the SPIR-V backends optimization capabilities and
maintain the robustness of the test suite.
---------
Co-authored-by: Nathan Gauër <github@keenuts.net>
This commit improves the handling of GetElementPtr (GEP) instructions
for
Logical SPIR-V. It includes:
- Rewriting of GEPs that are not allowed in Logical SPIR-V
(specifically,
handling non-zero first indices by rebuilding access chains or adjusting
types).
- Better deduction of element types for pointer casting.
- Updates to instruction selection to ensure GEPs are correctly lowered
to
OpAccessChain or OpInBoundsAccessChain only when valid (e.g. first index
0).
- Support for standard HLSL cbuffer layouts in tests.
Adds the fwidth intrinsic for HLSL.
The DXIL path only requires modification to the hlsl headers.
The SPIRV path implements the OpFwidth builtin in Clang and instruction
selection for the OpFwidth instruction in LLVM.
Also adds shader stage tests to the ddx_coarse and ddy_coarse
instructions used by fwidth.
Closes#99120
---------
Co-authored-by: Alexander Johnston <alexander.johnston@amd.com>
`inttoptr` usage in global initialisers is valid, and rather common when it comes to the machinery around vtables (construction vtables are particularly fond). This was not handled by the BE, even though SPIR-V allows forming `SpecConstantOp`s with the `OpConvertUToPtr` opcode, which is what these would map to. We augment instruction selection to address this.
Implement the f16tof32() intrinsic, including DXILand SPIRV codegen, and
associated tests.
Fixes#99112
---------
Co-authored-by: Tim Corringham <tcorring@amd.com>
The spv_bitcast intrinsic is currently replaced by an OpBitcast
during prelegalization. This will cause a problem when we need to
legalize the OpBitcast. The legalizer assumes that instruction
already lowered to a target specific opcode is legal.
We cannot lower it to a G_BITCAST because the bitcasts sometimes
the LLT type will be the same, causing an error in the verifier,
even if the SPIR-V types will be different.
This commit keeps the intrinsic around until instruction selection.
We can create rules to legalize a G_INTRINISIC* instruction, and
it does not create problem for the verifier.
No tests are updated because this change should be invisible to users.
Adds the WaveActiveMin intrinsic from #99169. I think I did all of the
required things on the checklist:
- [x] Implement `WaveActiveMin` clang builtin,
- [x] Link `WaveActiveMin` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `WaveActiveMin` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `WaveActiveMin` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/WaveActiveMin.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/WaveActiveMin-errors.hlsl`
- [x] Create the `int_dx_WaveActiveMin` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_WaveActiveMin` to `119` in
`DXIL.td`
- [x] Create the `WaveActiveMin.ll` and `WaveActiveMin_errors.ll` tests
in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_WaveActiveMin` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `WaveActiveMin`
lowering and map it to `int_spv_WaveActiveMin` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveMin.ll
But as some of the code has changed and was moved around (E.G.
`CGBuiltin.cpp` -> `CGHLSLBuiltins.cpp`) I mostly followed how
`WaveActiveMax()` is implemented.
I have not been able to run the tests myself as I am unsure which
project runs the correct test. Any guidance on how I can test myself
would be helpful.
Also added some tests to the offload-test-suite
https://github.com/llvm/offload-test-suite/pull/478
Currently, the spir-v validator fails if `OpImageRead` instruction is
used when loading from read-only `Buffer`.
### Unit Test
```hlsl
RWBuffer<uint> rwbuff;
Buffer<uint> buff;
[numthreads(1,1,1)]
void main() {
rwbuff[99] = buff[98];
rwbuff[97] = rwbuff[96];
}
```
This also unblocks adding a test case that adds a special capability
when using a non-uniform index on `Buffer` arrays.
(https://github.com/llvm/llvm-project/pull/162540).
Resolves https://github.com/llvm/llvm-project/issues/162891
There was some repeated code that was used to deduce the
SPIRV::LinkageType from a GlobalVariable/Function.
At several related parts of the code we also had functions taking 2
parameters:
a 'hasLinkage' bool, and a 'LinkageType'. This is error-prone since the
later parameter's meaning depends on the first. This patch also merges
these
two options into a single `std::optional<SPIRV::LinkageType>`.
Support `@llvm.spv.resource.nonuniformindex` in SPIRV Codegen.
- Add `NonUniformEXT` decoration to the registers marked as
`nonuniformindex`, and recursively decorate its child registers (e.g.
Copy, AccessChain, Load) that access such index.
- `OpCapability ShaderNonUniformEXT` is already added in the code.
-
[SPV_EXT_descriptor_indexing](https://github.khronos.org/SPIRV-Registry/extensions/EXT/SPV_EXT_descriptor_indexing.html)
is skipped because it's added to SPIRV Core in 1.5.
## Unit test
- The unit test checks that the register being used in the final
Store/Load/Write instruction is decorated, as required by the spec.
- The implementation follows [DXC](https://godbolt.org/z/zhqGThcaf) in
that it recursively decorates all the child elements until the end.
```hlsl
RWStructuredBuffer<uint4> StructuredOut[64];
RWBuffer<uint> UnStructuredOut[64];
[numthreads(64,1,1)]
void main(uint3 GTID: SV_GroupThreadID) {
StructuredOut[(NonUniformResourceIndex(GTID.x + 1))][98][0] = 99;
UnStructuredOut[(NonUniformResourceIndex(GTID.x))][96] = 95;
}
```
Resolves https://github.com/llvm/llvm-project/issues/160231,
https://github.com/llvm/llvm-project/issues/161852.
Verified
[offload-test-suite](cfc37840c8/test/Feature/ResourceArrays/unbounded-array-nuri.test)
started passing for clang.
This commit implements the backend portion of the typed buffer counter
proposal described in
https://github.com/llvm/wg-hlsl/blob/main/proposals/0023-typed-buffer-counters.md.
This is the second part of the implementation, focusing on the LLVM IR
and SPIR-V backend.
Specifically, this commit implements the "LLVM IR Generation and Backend
Handling"
section of the proposal. This includes:
- Adding the `llvm.spv.resource.counterhandlefromimplicitbinding` and
`llvm.spv.resource.counterhandlefrombinding` intrinsics.
- Implementing the selection of these intrinsics in the SPIRV backend to
generate the correct `OpVariable` and `OpDecorate` instructions for
the counter buffer.
- Handling `IncrementCounter` and `DecrementCounter` via a new
`llvm.spv.resource.updatecounter` intrinsic, which is lowered to
`OpAtomicIAdd`.
- Adding a new test file to verify the implementation.
Contributes to https://github.com/llvm/llvm-project/issues/137032
---------
Co-authored-by: Marcos Maronas <marcos.maronas@intel.com>
Implementation of
[SPV_KHR_float_controls2](https://github.khronos.org/SPIRV-Registry/extensions/KHR/SPV_KHR_float_controls2.html)
extension, and corresponding tests.
Some of the tests make use of `!spirv.ExecutionMode` LLVM named
metadata. This is because some SPIR-V instructions don't have a direct
equivalent in LLVM IR, so the SPIR-V Target uses different LLVM named
metadata to convey the necessary information. Below, you will find an
example from one of the newly added tests:
```
!spirv.ExecutionMode = !{!19, !20, !21, !22, !23, !24, !25, !26, !27}
!19 = !{ptr @k_float_controls_float, i32 6028, float poison, i32 131079}
!20 = !{ptr @k_float_controls_all, i32 6028, float poison, i32 131079}
!21 = !{ptr @k_float_controls_float, i32 31}
!22 = !{ptr @k_float_controls_all, i32 31}
!23 = !{ptr @k_float_controls_float, i32 4461, i32 32}
!24 = !{ptr @k_float_controls_all, i32 4461, i32 16}
!25 = !{ptr @k_float_controls_all, i32 4461, i32 32}
!26 = !{ptr @k_float_controls_all, i32 4461, i32 64}
!27 = !{ptr @k_float_controls_all, i32 4461, i32 128}
```
`!spirv.ExecutionMode` contains a list of metadata nodes, and each of
them specifies the required operands for expressing a particular
`OpExecutionMode` instruction in SPIR-V. For example, `!19 = !{ptr
@k_float_controls_float, i32 6028, float poison, i32 131079}` will be
lowered to `OpExecutionMode [[k_float_controls_float_ID]]
FPFastMathDefault [[float_type_ID]] 131079`.
---------
Co-authored-by: Dmitry Sidorov <dmitry.sidorov@intel.com>
--Added legalizer for @llvm.lround.* and @llvm.llround.* inrinsic
--Added Instruction Selector for @llvm.lround.* and @llvm.llround.*
intrinsic
--Added tests for @llvm.lround.* and @llvm.llround.* intrinsic
fixes#148051
- update EmitHLSLBuiltinExpr in CGHLSLBuiltins.cpp to toggle intrinsics
by target
- Add a GENERATE_HLSL_INTRINSIC_FUNCTION for isinf in CGHLSLRuntime.h
- Update the SPIRVInstructionSelector.cpp to emit the OpSinf instruction
- Updates the isinf.hlsl test to check spirv intrinsic generation
- add OpIsinf.ll tests
Removes uniformity bit from resource initialization intrinsics `llvm.{dx|spv}.resource.handlefrombinding` and `llvm.{dx|spv}.resource.handlefromimplicitbinding`. The flag currently always set to `false`. It should be derived from resource analysis and not provided by codegen.
Closes#135452
fixes#135572
There are two problems that are causing problems first register types
are copied from older registers instead of evaluating the spirv types.
Second the way OpSelect is defined in SPIRVInstrInfo.td we always
default to integer for TernOpTyped. There seems to be a problem of
multiple matches in the getMatchTable so when executeMatchTable runs we
aren't getting the right opSelect.
Correcting the tablegen wasn't very easy so instead created an emitter
for Select that evaluated the register types. this passes the original
llvm/test/CodeGen/SPIRV/instructions/select.ll tests and the new float
ones I'm adding in issue-135572-emit-float-opselect.ll
