Resolves https://github.com/llvm/llvm-project/issues/99161
- [x] Implement `WaveActiveAllTrue` clang builtin,
- [x] Link `WaveActiveAllTrue` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `WaveActiveAllTrue` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `WaveActiveAllTrue` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/WaveActiveAllTrue.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/WaveActiveAllTrue-errors.hlsl`
- [x] Create the `int_dx_WaveActiveAllTrue` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_WaveActiveAllTrue` to `114`
in `DXIL.td`
- [x] Create the `WaveActiveAllTrue.ll` and
`WaveActiveAllTrue_errors.ll` tests in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_WaveActiveAllTrue` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `WaveActiveAllTrue`
lowering and map it to `int_spv_WaveActiveAllTrue` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveAllTrue.ll`
This PR adds the following features:
* saturation and float rounding mode decorations,
* arithmetic constrained floating-point intrinsics (strict_fadd,
strict_fsub, strict_fmul, strict_fdiv, strict_frem, strict_fma and
strict_fldexp),
* and SPV_INTEL_float_controls2 extension,
* using recent improvements of emit-intrinsics step, this PR also
simplifies pre- and post-legalizer steps and improves instruction
selection.
Support HLSL SV_GroupThreadId attribute.
For `directx` target, translate it into `dx.thread.id.in.group` in clang
codeGen and lower `dx.thread.id.in.group` to `dx.op.threadIdInGroup` in
LLVM DirectX backend.
For `spir-v` target, translate it into `spv.thread.id.in.group` in clang
codeGen and lower `spv.thread.id.in.group` to a `LocalInvocationId`
builtin variable in LLVM SPIR-V backend.
Fixes: #70122
This PR improves general validity of emitted code between passes due to
generation of `TargetOpcode::PHI` instead of `SPIRV::OpPhi` after
Instruction Selection, fixing generation of OpTypePointer instructions
and using of proper virtual register classes.
Using `TargetOpcode::PHI` instead of `SPIRV::OpPhi` after Instruction
Selection has a benefit to support existing optimization passes
immediately, as an alternative path to disable those passes that use
`MI.isPHI()`. This PR makes it possible thus to revert
https://github.com/llvm/llvm-project/pull/116060 actions and get back to
use the `MachineSink` pass.
This PR is a solution of the problem discussed in details in
https://github.com/llvm/llvm-project/pull/110507. It accepts an advice
from code reviewers of the PR #110507 to postpone generation of OpPhi
rather than to patch CodeGen. This solution allows to unblock
improvements wrt. expensive checks and makes it unrelated to the general
points of the discussion about OpPhi vs. G_PHI/PHI.
This PR contains numerous small patches of emitted code validity that
allows to substantially pass rate with expensive checks. Namely, the
test suite with expensive checks set ON now has only 12 fails out of 569
total test cases.
FYI @bogner
This OpenCL extension extends the subgroup block read and write
functions defined by `cl_intel_subgroups` (and its `char`, `short`, and
`long` versions) to support reading from and writing to pointers to the`
__local` memory address space in addition to pointers to the `__global`
memory address space.
The builtins are translated to SPIR-V using `SPV_INTEL_subgroups`
extension.
This PR is to fix the following issues:
* the SPIR-V Backend didn't generate Alignment decoration for alloca
instructions,
* we need to use types from demangled function declarations to specify
types for opaque pointers.
The spec is available here:
https://github.com/intel/llvm/pull/12497
The PR doesn't add OpCooperativeMatrixApplyFunctionINTEL instruction as
it's still experimental and not properly tested E2E.
The PR also fixes few bugs in the related code:
1. CooperativeMatrixMulAddKHR optional operand must be literal, not a
constant;
2. Fixed available capabilities table creation for a case, when a single
extension adds few capabilities, that occupy not contiguous op codes.
---------
Signed-off-by: Sidorov, Dmitry <dmitry.sidorov@intel.com>
This PR fixes:
* emission of OpNames (added newly inserted internal intrinsics and
basic blocks)
* emission of function attributes (SRet is added)
* implementation of SPV_INTEL_optnone so that it emits OptNoneINTEL
Function Control flag, and add implementation of the SPV_EXT_optnone
SPIR-V extension.
This PR resolved the following issues:
(1) There are rare but possible cases when there are bitcasts between
pointers intertwined in a sophisticated way with loads, stores, function
calls and other instructions that are part of type deduction. In this
case we must account for inserted bitcasts between pointers rather than
just ignore them.
(2) Null pointers have the same constant representation but different
types. Type info from Intrinsic::spv_track_constant() refers to the
opaque (untyped) pointer, so that each MF/v-reg pair would fall into the
same Const record in Duplicate Tracker and would be represented by a
single OpConstantNull instruction, unless we use precise pointee type
info. We must be able to distinguish one constant (null) pointer from
another to avoid generating invalid code with inconsistent types of
operands.
Re-land of #116636
Adds a new address spaces: hlsl_private. Variables with such address
space will be emitted with a Private storage class.
This is useful for variables global to a SPIR-V module, since up to now,
they were still emitted with a Function storage class, which is wrong.
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
Adds a new address spaces: `hlsl_private`. Variables with such address
space will be emitted with a `Private` storage class.
This is useful for variables global to a SPIR-V module, since up to now,
they were still emitted with a `Function` storage class, which is wrong.
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
Goals of the PR are:
* to ensure that correct types are applied to virtual registers which
were used as return values in call lowering. A reproducer is attached as
a new test case, before the PR it fails because spirv-val considers
output invalid due to wrong result/operand types in OpPhi's;
* improve type inference by speeding up postprocessing of types: by
limiting iterations by checking what remains to process, and processing
each instruction just once for any number of operands with uncomplete
types;
* improve type inference by more accurate work with uncomplete types
(pass uncomplete property to dependent operands, ensure consistency of
uncomplete-types data structure);
* change processing order and add traversing of PHI nodes when type
inference apply instructions results to specify/update/cast operands
type (fixes an issue with OpPhi's result type mismatch with operand
types).
Add testing for the visitor and added a note explaining irreducible CFG
are not supported.
Related to #116692
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
Block sorting was assuming reducible CFG. Meaning we always had a best
node to continue with. Irreducible CFG makes breaks this assumption, so
the algorithm looped indefinitely because no node was a valid candidate.
Fixes#116692
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
This PR fixes generation of OpConstantFunctionPointerINTEL instruction
for the SPIR-V extension SPV_INTEL_function_pointers. Result type of
OpConstantFunctionPointerINTEL must be OpTypePointer with Storage Class
operand equal to CodeSectionINTEL.
See also https://github.com/llvm/llvm-project/pull/116636
CC: @MrSidims
```
- use the new OpSDot/OpUDot instructions when capabilites allow in SPIRVInstructionSelector.cpp
- correct functionality of capability check onto input operand and not return operand type in SPIRVModuleAnalysis.cpp
- add test cases to demonstrate use case in idot.ll
```
Resolves#114632
Resolves https://github.com/llvm/llvm-project/issues/99160
- [x] Implement `WaveActiveAnyTrue` clang builtin,
- [x] Link `WaveActiveAnyTrue` clang builtin with `hlsl_intrinsics.h`
- [x] Add sema checks for `WaveActiveAnyTrue` to
`CheckHLSLBuiltinFunctionCall` in `SemaChecking.cpp`
- [x] Add codegen for `WaveActiveAnyTrue` to `EmitHLSLBuiltinExpr` in
`CGBuiltin.cpp`
- [x] Add codegen tests to
`clang/test/CodeGenHLSL/builtins/WaveActiveAnyTrue.hlsl`
- [x] Add sema tests to
`clang/test/SemaHLSL/BuiltIns/WaveActiveAnyTrue-errors.hlsl`
- [x] Create the `int_dx_WaveActiveAnyTrue` intrinsic in
`IntrinsicsDirectX.td`
- [x] Create the `DXILOpMapping` of `int_dx_WaveActiveAnyTrue` to `113`
in `DXIL.td`
- [x] Create the `WaveActiveAnyTrue.ll` and
`WaveActiveAnyTrue_errors.ll` tests in `llvm/test/CodeGen/DirectX/`
- [x] Create the `int_spv_WaveActiveAnyTrue` intrinsic in
`IntrinsicsSPIRV.td`
- [x] In SPIRVInstructionSelector.cpp create the `WaveActiveAnyTrue`
lowering and map it to `int_spv_WaveActiveAnyTrue` in
`SPIRVInstructionSelector::selectIntrinsic`.
- [x] Create SPIR-V backend test case in
`llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveAnyTrue.ll`
---------
Co-authored-by: Finn Plummer <50529406+inbelic@users.noreply.github.com>
Co-authored-by: Greg Roth <grroth@microsoft.com>
This commit adds an intrinsic that will write to an image buffer. We
chose to match the name of the DXIL intrinsic for simplicity in clang.
We cannot reuse the existing openCL write_image function because that is
not a reserved name in HLSL. There is not much common code to factor
out.
This PR is to solve several intertwined issues with type inference while
adding support for builtins for OpIAddCarry and OpISubBorrow:
* OpIAddCarry and OpISubBorrow generation in a way of supporting SPIR-V
friendly builtins `__spirv_...` -- introduces a new element to account
for, namely, `ptr sret (%struct) %0` argument that is a place to put a
result of the instruction;
* fix early definition of SPIR-V types during call lowering -- namely,
the goal of the PR is to ensure that correct types are applied to
virtual registers which were used as arguments in call lowering and so
caused early definition of SPIR-V types; reproducers are attached as a
new test cases;
* improve parsing of builtin names (e.g., understand a name of a kind
`"anon<int, int> __spirv_IAddCarry<int, int>(int, int)"` that was
incorrectly parsed as `anon` before the PR);
* improve type inference and fix access to erased from parent after
visit instructions -- before the PR visiting of instructions in
emitintrinsics pass replaced old alloca's, bitcast's, etc. instructions
with a newly generated internal SPIR-V intrinsics and after erasing old
instructions there were still references to them in a postprocessing
working list, while records for newly deduced pointee types were lost;
this PR fixes the issue by adding as consistent wrt. internal data
structures action `SPIRVEmitIntrinsics::replaceAllUsesWith()` that fixes
above mentioned problems;
* LLVM IR add/sub instructions result in logical SPIR-V instructions
when applied to bool type;
* fix validation of pointer types for frexp and lgamma_r,
* fix hardcoded reference to AS0 as a Function storage class in
lib/Target/SPIRV/SPIRVBuiltins.cpp -- now it's
`storageClassToAddressSpace(SPIRV::StorageClass::Function)`,
* re-use the same OpTypeStruct for two identical references to struct's
in arithmetic with overflow instructions.
This commit adds an intrinsic that will read from an image buffer. We
chose to match the name of the DXIL intrinsic for simplicity in clang.
We cannot reuse the existing openCL readimage function because that is
not a reserved name in HLSL.
I considered trying to refactor generateReadImageInst, so that we could
share code between the two implementations. However, most of the code in
generateReadImageInst is concerned with trying to figure out which type
of image read is being done. Once we factor out the code that will be
common, then we end up with just a single call to the MIRBuilder being
common.
```
- add codegen for llvm builtin to spirv/directx intrinsic in CGBuiltin.cpp
- add lowering of spirv intrinsic to spirv backend in SPIRVInstructionSelector.cpp
- add lowering of directx intrinsic to dxil op in DXIL.td
- add test cases to illustrate passes
- add test case for semantic analysis
```
Resolves#80176
Fixes#88052
- Added the following intrinsics:
- `int_spv_uclamp`
- `int_spv_sclamp`
- `int_spv_fclamp`
- Updated DirectX counterparts to have the same three clamp intrinsics.
- Update the clamp.hlsl unit tests to include SPIRV
- Added the SPIRV specific tests
```- create a clang built-in in Builtins.td
- link dot4add_u8packed in hlsl_intrinsics.h
- add lowering to spirv backend through expansion of operation as OpUDot is missing up to SPIRV 1.6 in SPIRVInstructionSelector.cpp
- add lowering to spirv backend using OpUDot if applicable SPIRV version or SPV_KHR_integer_dot_product is enabled
- add dot4add_u8packed intrinsic to IntrinsicsDirectX.td and mapping to DXIL.td op Dot4AddU8Packed
- add tests for HLSL intrinsic lowering to dx/spv intrinsic in dot4add_u8packed.hlsl
- add tests for sema checks in dot4add_u8packed-errors.hlsl
- add test of spir-v lowering in SPIRV/dot4add_u8packed.ll
- add test to dxil lowering in DirectX/dot4add_u8packed.ll
```
Resolves#99219
Implements elementwise firstbithigh hlsl builtin.
Implements firstbituhigh intrinsic for spirv and directx, which handles
unsigned integers
Implements firstbitshigh intrinsic for spirv and directx, which handles
signed integers.
Fixes#113486Closes#99115
OpenCL Extended Instruction Set Specification defines relations between
return/operand types and pointee type of pointer arguments in case of
remquo, fract, frexp, lgamma_r, modf, sincos and prefetch instructions
(https://registry.khronos.org/SPIR-V/specs/unified1/OpenCL.ExtendedInstructionSet.100.html).
This PR ensures correct pointee types of those OpenCL Extended
Instructions' pointer arguments.
The goal of the PR is to ensure that if module contains functions with
mutated signature (due to preprocessing of aggregate types), functions
still are going through re-creating of function type to preserve pointee
type information for arguments.
This fixes a bug when a module with (1) a function having aggregate
arguments and/or return, and (2) at least two functions with signatures
different only wrt. pointee types is translated so that one of two
similar functions gets an incorrect OpFunctionParameter type that is
different from the corresponding OpTypeFunction definition.
A reproducer is attached as a new test case.
- create a clang built-in in Builtins.td
- link dot4add_i8packed in hlsl_intrinsics.h
- add lowering to spirv backend through expansion of operation as OPSDot
is missing up to SPIRV 1.6 in SPIRVInstructionSelector.cpp
- add lowering to spirv backend using OpSDot in applicable SPIRV version
or if SPV_KHR_integer_dot_product is enabled
- add dot4add_i8packed intrinsic to IntrinsicsDirectX.td and mapping to
DXIL.td op Dot4AddI8Packed
- add tests for HLSL intrinsic lowering to dx/spv intrinsic in
dot4add_i8packed.hlsl
- add tests for sema checks in dot4add_i8packed-errors.hlsl
- add test of spir-v lowering in SPIRV/dot4add_i8packed.ll
- add test to dxil lowering in DirectX/dot4add_i8packed.ll
Resolves#99220
SPIR-V doesn't currently encode "native" integer bit-widths in its
datalayout(s). This is problematic as it leads to optimisation passes,
such as InstCombine, getting ideas and e.g. shrinking to non
byte-multiple integer types, which is not desirable and can lead to
breakage further down in the toolchain. This patch addresses that by
encoding `i8`, `i16`, `i32` and `i64` as native types for vanilla SPIR-V
(the spec natively supports them), and `i32` and `i64` for AMDGCNSPIRV
(where the hardware targets are known). We also set the stack alignment
on the latter, as it is overaligned (32-bit vs 8-bit).
In SPIR-V, OpDecorate instructions are allowed to forward-declare a
virtual register. But while we are at the MIR level, we must comply with
stricter rules, meaning OpDecorate should be emited after, not before
the reg definition.
(In some cases, we defined those just before, switching to just after).
Related to #110652
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
Many tests had this flag removed because of the G_BITCAST emission
issue. Now that the PR is merged, we can re-enable this additional
check.
2 tests (basic_int_types) just have the TODO removed because they are
not useful for SPIR-V as-is: SPIR-V requires reg2mem/mem2reg to run,
which removes all the body. Integers are used in other spirv tests, and
seems like testing for spirv32/64 and relying on others for the logical
target coverage should be fine.
Signed-off-by: Nathan Gauër <brioche@google.com>
This PR implements instruction selection for G_BITCAST on an earlier
stage to avoid MachineVerifier complains on subtle semantics difference
between G_BITCAST and OpBitcast.
We do instruction selections for OpBitcast after IR Translation instead
of calling MIB.buildBitcast() generating the general op code G_BITCAST,
because when MachineVerifier validates G_BITCAST we see a check of a
kind: 'if Source Type is equal to Destination Type then report error
"bitcast must change the type"'. This doesn't take into account the
notion of a typed pointer that is important for SPIR-V where a user may
and should use bitcast between pointers with different pointee types
(https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#OpBitcast).
It's important for correct lowering in SPIR-V, because interpretation of
the data type is not left to instructions that utilize the pointer, but
encoded by the pointer declaration, and the SPIRV target can and must
handle the declaration and use of pointers that specify the type of data
they point to.
It's not feasible to improve validation of G_BITCAST using just
information provided by low level types of source and destination.
Therefore we don't produce G_BITCAST as the general op code with
semantics different from OpBitcast, but rather lower to OpBitcast
immediately.
See discussion in https://github.com/llvm/llvm-project/pull/110270 for
even more context.
G_BITCAST emission in the SPIR-V backend is not accepted by the
verifier. DIsabling verifier for impacted tests until
https://github.com/llvm/llvm-project/pull/114216 is merged.
Signed-off-by: Nathan Gauër <brioche@google.com>
The "topological" sorting was behaving incorrectly in some cases:
the exit of a loop could have a lower rank than a node in the loop.
This causes issues when structurizing some patterns, and also codegen
issues as we could generate BBs in the incorrect order in regard to the
SPIR-V spec.
Fixing this ordering alone broke other parts of the structurizer, which
by luck worked. Had to fix those.
Added more test cases, especially to test basic patterns.
I also needed to tweak/disable some tests for 2 reasons:
- SPIR-V now required reg2mem/mem2reg to run. Meaning dead stores
are optimized away. Some tests require tweaks to avoid having the
whole function removed.
- Mem2Reg will generate variable & load/stores. This generates
G_BITCAST in several cases. And there is currently something wrong
we do with G_BITCAST which causes MIR verifier to complain.
Until this is resolved, I disabled -verify-machineinstrs flag on
those tests.
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
Line ending policies were changed in the parent, dccebddb3b80. To make
it easier to resolve downstream merge conflicts after line-ending
policies are adjusted this is a separate whitespace-only commit. If you
have merge conflicts as a result, you can simply `git add --renormalize
-u && git merge --continue` or `git add --renormalize -u && git rebase
--continue` - depending on your workflow.
- create a clang built-in in Builtins.td
- add semantic checking in SemaHLSL.cpp
- link the WaveReadLaneAt api in hlsl_intrinsics.h
- add lowering to spirv backend op GroupNonUniformShuffle
with Scope = 2 (Group) in SPIRVInstructionSelector.cpp
- add WaveReadLaneAt intrinsic to IntrinsicsDirectX.td and mapping
to DXIL.td
- add tests for HLSL intrinsic lowering to spirv intrinsic in
WaveReadLaneAt.hlsl
- add tests for sema checks in WaveReadLaneAt-errors.hlsl
- add spir-v backend tests in WaveReadLaneAt.ll
- add test to show scalar dxil lowering functionality
- note that this doesn't include support for the scalarizer to
handle WaveReadLaneAt will be added in a future pr
This is the first part #70104
