This change drops the use of the "Layout" type and instead uses explicit
padding throughout the compiler to represent types in HLSL buffers.
There are a few parts to this, though it's difficult to split them up as
they're very interdependent:
1. Refactor HLSLBufferLayoutBuilder to allow us to calculate the padding
of arbitrary types.
2. Teach Clang CodeGen to use HLSL specific paths for cbuffers when
generating aggregate copies, array accesses, and structure accesses.
3. Simplify DXILCBufferAccesses such that it directly replaces accesses
with dx.resource.getpointer rather than recalculating the layout.
4. Basic infrastructure for SPIR-V handling, but the implementation
itself will need work in follow ups.
Fixes several issues, including #138996, #144573, and #156084.
Resolves#147352.
This commit adds support for legalizing pointer casts between array and
vector types within the SPIRV backend.
This is necessary to handle cases where a vector is loaded from or
stored to an array, which can occur with HLSL matrix types.
The following changes are included:
- Added to load a vector from an array.
- Added to store a vector to an array.
- Added the test case to verify the functionality.
In HIP, dynamic LDS variables are represented using `0-element` global
arrays in the `__shared__` language address-space.
```cpp
extern __shared__ int LDS[];
```
These are not representable in SPIRV directly.
To represent them, for AMD, we use an array with `UINT32_MAX`-elements.
These are reverse translated to 0-element arrays later in AMD's SPIRV
runtime pipeline (in
[SPIRVReader.cpp](8cb74e264d/lib/SPIRV/SPIRVReader.cpp (L358))).
When compiling with `-fembed-bitcode-marker`, Clang inserts a
placeholder
for the bitcode. This placeholder is a `[0 x i8]` array, which we cannot
represent in SPIRV.
For AMD flavored SPIRV, we extend the `llvm.embedded.module` global to a
`zeroinitializer [1 x i8]` array.
To achieve this, this patch adds a new pass, `SPIRVPrepareGlobals`, that
we can use to write global variable's _non-trivial-to-lower-IR_ ->
_trivial-to-lower-IR_ mappings.
This is a second attempt at
https://github.com/llvm/llvm-project/pull/162082, but cleaner.
In the translator something similar is done for every 0-element array
since https://github.com/KhronosGroup/SPIRV-LLVM-Translator/pull/2743 .
But I don't think we want to do this mapping for all cases.
According to SPIR-V spec:
> It is invalid to decorate any given id or structure member more than
one time with the same
[decoration](https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#Decoration),
unless explicitly allowed below for a specific decoration.
`FuncParamAttr` explicitly allows multiple uses of the decoration on the
same id, so this patch honors it.
Add an error test to check that a suitable error diagnostic is generated
for the use of the GL::unpackhalf2x16 operation in
invalid contexts.
Fixes#166965
Co-authored-by: Tim Corringham <tcorring@amd.com>
With `+SPV_KHR_float_controls2` and when there is a non-int
`OpConstantNull` we
would call `MI.getOperand(1).getImm()` when `MI` was not an `OpTypeInt`
(the
associated test has an `OpTypeArray` zeroinitialized).
Under this conditions an assertion is triggered.
This patch adds the missing condition.
This patch doesn't change anything. Just adds more explicit checks to
verify what is generated in this case when an alloca has a zero-sized
array.
I'd expect an `OpRuntimeArray`, but nothing is generated.
This enables support for atomic RMW ops (add, sub, min and max to be
precise) with `bfloat16` operands, via the [SPV_INTEL_16bit_atomics
extension](https://github.com/intel/llvm/pull/20009). It's logically a
successor to #166031 (I should've used a stack), but I'm putting it up
for early review.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
`inttoptr` usage in global initialisers is valid, and rather common when it comes to the machinery around vtables (construction vtables are particularly fond). This was not handled by the BE, even though SPIR-V allows forming `SpecConstantOp`s with the `OpConvertUToPtr` opcode, which is what these would map to. We augment instruction selection to address this.
This ports rather useful functionality that was already available in the
Translator, and was mostly implemented in the BE. Today, if we encounter
an unknown intrinsic, we pipe it through and hope for the best, which in
practice yields either obtuse ISEL errors, or potentially impossible to
use SPIR-V. With this change, if instructed via a newly added
`--spv-allow-unknown-intrinsics` flag, we emit allowed intrinsics as
calls to extern (import) functions. The test is also mostly lifted from
the Translator.
Implement the f16tof32() intrinsic, including DXILand SPIRV codegen, and
associated tests.
Fixes#99112
---------
Co-authored-by: Tim Corringham <tcorring@amd.com>
Enable the `SPV_INTEL_bfloat16_arithmetic` extension, which allows arithmetic, relational and `OpExtInst` instructions to take `bfloat16` arguments. This patch only adds support to arithmetic and relational ops. The extension itself is rather fresh, but `bfloat16` is ubiquitous at this point and not supporting these ops is limiting.
This adds BE support for the
[`SPV_INTEL_kernel_attributes`](https://github.khronos.org/SPIRV-Registry/extensions/INTEL/SPV_INTEL_kernel_attributes.html)
extension. The extension is necessary to encode the rather useful
`max_work_group_size` kernel attribute, via `OpExecutionMode
MaxWorkgroupSizeINTEL`, which is the only Execution Mode added by the
extension that this patch adds full processing for. Future patches will
add the other Execution Modes and Capabilities. The test is adapted from
the equivalent Translator test; it depends on #165815.
AMDGCN flavoured SPIR-V has slightly different defaults from what the BE
adopts: it assumes all extensions are enabled, and expects nonsemantic
debug info to be generated. Furthermore, it is necessary to encode in
the resulting SPIR-V binary that what was generated was AMDGCN
flavoured, which we do by setting the Generator Version to `UINT16_MAX`
(which matches what we expect to see at reverse translation). We will
register this generator version at
<https://github.com/KhronosGroup/SPIRV-Headers>. This is a preliminary
patch out of a series of patches that are needed for adopting the BE for
AMDGCN flavoured SPIR-V generation.
Due to limitations in GISel / IRTranslator, the SPIR-V BE replaces aggregate function args with `i32` placeholders, which are subsequently used to retrieve the original type after IR translation, from metadata. Due to what appears to be an oversight, the current implementation only handles a single mutation, as it does not traverse the metadata, but rather only takes the first operand. This patch addresses that limitation by correctly iterating the metadata.
The previous check for vector bitcasts in `loadVectorFromVector` only
compared the number of elements, which is insufficient when the element
types differ. This can lead to incorrect assumptions about the validity
of the cast.
This commit replaces the element count check with a comparison of the
total size of the vectors in bits. This ensures that the bitcast is
only performed between vectors of the same size, preventing potential
miscompilations.
Part of https://github.com/llvm/llvm-project/issues/153091
Adds the WaveActiveMin intrinsic from #99169. I think I did all of the
required things on the checklist:
- [x] Implement `WaveActiveMin` clang builtin,
- [x] Link `WaveActiveMin` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `WaveActiveMin` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `WaveActiveMin` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/WaveActiveMin.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/WaveActiveMin-errors.hlsl`
- [x] Create the `int_dx_WaveActiveMin` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_WaveActiveMin` to `119` in
`DXIL.td`
- [x] Create the `WaveActiveMin.ll` and `WaveActiveMin_errors.ll` tests
in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_WaveActiveMin` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `WaveActiveMin`
lowering and map it to `int_spv_WaveActiveMin` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveMin.ll
But as some of the code has changed and was moved around (E.G.
`CGBuiltin.cpp` -> `CGHLSLBuiltins.cpp`) I mostly followed how
`WaveActiveMax()` is implemented.
I have not been able to run the tests myself as I am unsure which
project runs the correct test. Any guidance on how I can test myself
would be helpful.
Also added some tests to the offload-test-suite
https://github.com/llvm/offload-test-suite/pull/478
In binary form, 64-bit values are split into two 32-bit values as per
the spec. Naturally this works fine with all tools.
However, the text format does not have a formal specification but
SPIR-V-Tools, which we already rely on in the SPIRV workflow (clang
calls `spirv-as` for example), expects the full 64 bit value, but today
we print the two 32-bit values. causing the tool to error and report
that the format is invalid.
The SPIR-V Translator also prints a single 64-bit value for text format.
This case is already handled specifically for `OpConstant`, but
`OpSwitch` was missed. The SPIR-V translator also has special code in
`OpSwitch` handling for this case.
Recombine the two 32-bit operands into a single 64-bit value to print in
`AsmPrinter`. The actual ASM (aka binary form) emission is unchanged.
---------
Signed-off-by: Sarnie, Nick <nick.sarnie@intel.com>
The comment here pointed out that RAUW would fall over given a
constantexpr, but then proceeded to just do what RAUW does by hand,
which falls over in the same way. Instead, convert constantexprs
involving cbuffer globals to instructions before processing them.
The test update just modifies the existing cbuffer test, since it
implied it was trying to test this exact case anyways.
Currently, the spir-v validator fails if `OpImageRead` instruction is
used when loading from read-only `Buffer`.
### Unit Test
```hlsl
RWBuffer<uint> rwbuff;
Buffer<uint> buff;
[numthreads(1,1,1)]
void main() {
rwbuff[99] = buff[98];
rwbuff[97] = rwbuff[96];
}
```
This also unblocks adding a test case that adds a special capability
when using a non-uniform index on `Buffer` arrays.
(https://github.com/llvm/llvm-project/pull/162540).
Resolves https://github.com/llvm/llvm-project/issues/162891
These tests verify SPIR-V code generation for LLVM intrinsics and atomic
operations, ensuring correct translation of fcmp false, llvm.fake.use,
and atomic_compare_exchange.
---------
Co-authored-by: Michal Paszkowski <michal@michalpaszkowski.com>
- Lowering of fcmp false in llvm IR
- Handling duplicate builtins
- Pointer conversion and address space cast
- Dominator ordering
---------
Co-authored-by: Michal Paszkowski <michal@michalpaszkowski.com>
`@llvm.compiler.used` holds a series of global values and prevents the compiler from optimizing them out.
However, the symbols in the variable can be optimized after compilation
as usual by a linker (notice that `@llvm.used` instead doesn't allow for
the linker to optimize-out the global variables referenced in it).
This was already done for `@llvm.global.annotations`.
Support `@llvm.spv.resource.nonuniformindex` in SPIRV Codegen.
- Add `NonUniformEXT` decoration to the registers marked as
`nonuniformindex`, and recursively decorate its child registers (e.g.
Copy, AccessChain, Load) that access such index.
- `OpCapability ShaderNonUniformEXT` is already added in the code.
-
[SPV_EXT_descriptor_indexing](https://github.khronos.org/SPIRV-Registry/extensions/EXT/SPV_EXT_descriptor_indexing.html)
is skipped because it's added to SPIRV Core in 1.5.
## Unit test
- The unit test checks that the register being used in the final
Store/Load/Write instruction is decorated, as required by the spec.
- The implementation follows [DXC](https://godbolt.org/z/zhqGThcaf) in
that it recursively decorates all the child elements until the end.
```hlsl
RWStructuredBuffer<uint4> StructuredOut[64];
RWBuffer<uint> UnStructuredOut[64];
[numthreads(64,1,1)]
void main(uint3 GTID: SV_GroupThreadID) {
StructuredOut[(NonUniformResourceIndex(GTID.x + 1))][98][0] = 99;
UnStructuredOut[(NonUniformResourceIndex(GTID.x))][96] = 95;
}
```
Resolves https://github.com/llvm/llvm-project/issues/160231,
https://github.com/llvm/llvm-project/issues/161852.
Verified
[offload-test-suite](cfc37840c8/test/Feature/ResourceArrays/unbounded-array-nuri.test)
started passing for clang.
This PR introduces the support for the SPIR-V extension
`SPV_INTEL_predicated_io`. This extension adds predicated load and store
instructions. Predicated load performs load from memory if predicate is
true; otherwise, it uses default_value as a result. Predicated store
performs store of value to memory if predicate is true; otherwise, it
does nothing.
Reference Specification:
https://github.com/intel/llvm/blob/sycl/sycl/doc/design/spirv-extensions/SPV_INTEL_predicated_io.asciidoc
This commit implements the backend portion of the typed buffer counter
proposal described in
https://github.com/llvm/wg-hlsl/blob/main/proposals/0023-typed-buffer-counters.md.
This is the second part of the implementation, focusing on the LLVM IR
and SPIR-V backend.
Specifically, this commit implements the "LLVM IR Generation and Backend
Handling"
section of the proposal. This includes:
- Adding the `llvm.spv.resource.counterhandlefromimplicitbinding` and
`llvm.spv.resource.counterhandlefrombinding` intrinsics.
- Implementing the selection of these intrinsics in the SPIRV backend to
generate the correct `OpVariable` and `OpDecorate` instructions for
the counter buffer.
- Handling `IncrementCounter` and `DecrementCounter` via a new
`llvm.spv.resource.updatecounter` intrinsic, which is lowered to
`OpAtomicIAdd`.
- Adding a new test file to verify the implementation.
Contributes to https://github.com/llvm/llvm-project/issues/137032
---------
Co-authored-by: Marcos Maronas <marcos.maronas@intel.com>
Generate `OpBitCast` instruction for pointer cast operation if the
element type is different.
The HLSL for the unit test is
```hlsl
StructuredBuffer<uint2> In : register(t0);
RWStructuredBuffer<double2> Out : register(u2);
[numthreads(1,1,1)]
void main() {
Out[0] = asdouble(In[0], In[1]);
}
```
Resolves https://github.com/llvm/llvm-project/issues/153513
Prevent adding duplicate instructions for implicit bindings when they
are from the same resource. The fix is to store and check if the binding
number is already assigned for each `OrderId`.
Resolves https://github.com/llvm/llvm-project/issues/160716
This PR resolves the current failure in the `integer-casts.ll` SPIR-V
test during CI runs in `llvm-project`.
The failure occurs because the SPIR-V instruction `OpQuantizeToF16`
requires the `Capability::Shader`. However, the function in
`integer-casts.ll` is written as a kernel function and executed in a
kernel environment. Therefore, `Capability::Kernel` is emitted instead
of `Capability::Shader`. To fix this, we remove the `QuantizeToF16` test
from`integer-casts.ll` in this PR.
Implementation of
[SPV_KHR_float_controls2](https://github.khronos.org/SPIRV-Registry/extensions/KHR/SPV_KHR_float_controls2.html)
extension, and corresponding tests.
Some of the tests make use of `!spirv.ExecutionMode` LLVM named
metadata. This is because some SPIR-V instructions don't have a direct
equivalent in LLVM IR, so the SPIR-V Target uses different LLVM named
metadata to convey the necessary information. Below, you will find an
example from one of the newly added tests:
```
!spirv.ExecutionMode = !{!19, !20, !21, !22, !23, !24, !25, !26, !27}
!19 = !{ptr @k_float_controls_float, i32 6028, float poison, i32 131079}
!20 = !{ptr @k_float_controls_all, i32 6028, float poison, i32 131079}
!21 = !{ptr @k_float_controls_float, i32 31}
!22 = !{ptr @k_float_controls_all, i32 31}
!23 = !{ptr @k_float_controls_float, i32 4461, i32 32}
!24 = !{ptr @k_float_controls_all, i32 4461, i32 16}
!25 = !{ptr @k_float_controls_all, i32 4461, i32 32}
!26 = !{ptr @k_float_controls_all, i32 4461, i32 64}
!27 = !{ptr @k_float_controls_all, i32 4461, i32 128}
```
`!spirv.ExecutionMode` contains a list of metadata nodes, and each of
them specifies the required operands for expressing a particular
`OpExecutionMode` instruction in SPIR-V. For example, `!19 = !{ptr
@k_float_controls_float, i32 6028, float poison, i32 131079}` will be
lowered to `OpExecutionMode [[k_float_controls_float_ID]]
FPFastMathDefault [[float_type_ID]] 131079`.
---------
Co-authored-by: Dmitry Sidorov <dmitry.sidorov@intel.com>
--Test for signed arithmetic overflow intrinsics, which is for now
expectedly failing
--Test checking that no duplicate image types are emitted.
---------
Co-authored-by: Michal Paszkowski <michal@michalpaszkowski.com>
- Added opencl Pipe builtins
- Pipe instructions were added in tablegen and lowered in
SPIRVBuiltins.cpp
---------
Co-authored-by: Michal Paszkowski <michal@michalpaszkowski.com>
Co-authored-by: Dmitry Sidorov <dmitry.sidorov@intel.com>
Added constraints related to Addressing model as specified in the
specification.
It conforms with the implementation in translator
Same as PR #160089
Solved all issues
The code was incorrectly converting all `undef` arguments to `i32`,
while the `spv_insertv` intrinsics only expects that for the first
operand, representing the aggregate type.
Fixes https://github.com/llvm/llvm-project/issues/127977
---------
Co-authored-by: Michal Paszkowski <michal@michalpaszkowski.com>
