When running SPIR-V Backend with optimization levels higher than 0, we
observe GEP Operator's as a new factor, massively used to convey the
semantics of the original LLVM IR. Previously, an issue related to GEP
Operator was mentioned and fixed on the consumer side of toolchains
(see, for example, Khronos Trandslator Issue
https://github.com/KhronosGroup/SPIRV-LLVM-Translator/issues/2486 and PR
https://github.com/KhronosGroup/SPIRV-LLVM-Translator/pull/2487).
However, there is a case when GenCode creates G_PTR_ADD to convey the
original semantics under optimization levels higher than 0 where it's
SPIR-V Backend that fails to translate source LLVM IR correctly.
Consider the following reproducer:
```
%struct = type { i32, [257 x i8], [257 x i8], [129 x i8], i32, i64, i64, i64, i64, i64, i64 }
@Mem = linkonce_odr dso_local addrspace(1) global %struct zeroinitializer, align 8
define weak dso_local spir_func void @__devicelib_assert_fail(ptr addrspace(4) noundef %expr, i32 noundef %line, i1 %fl) {
entry:
%cmp = icmp eq i32 %line, 0
br i1 %cmp, label %lbl, label %exit
lbl:
store i32 %line, ptr addrspace(1) getelementptr inbounds (i8, ptr addrspace(1) @Mem, i64 648), align 8
br i1 %fl, label %lbl, label %exit
exit:
ret void
}
```
converted to the following machine instructions by SPIR-V Backend:
```
%4:type(s64) = OpTypeInt 32, 0
%22:type(s64) = OpTypePointer 5, %4:type(s64)
%2:type(s64) = OpTypeInt 8, 0
%28:type(s64) = OpTypePointer 5, %2:type(s64)
%10:pid(p1) = G_GLOBAL_VALUE @Mem
%36:type(s64) = OpTypeStruct %4:type(s64), %32:type(s64), %32:type(s64), %34:type(s64), %4:type(s64), %35:type(s64), %35:type(s64), %35:type(s64), %35:type(s64), %35:type(s64), %35:type(s64)
%37:iid(s32) = G_INTRINSIC_W_SIDE_EFFECTS intrinsic(@llvm.spv.const.composite)
%8:iid(s32) = ASSIGN_TYPE %37:iid(s32), %36:type(s64)
G_INTRINSIC_W_SIDE_EFFECTS intrinsic(@llvm.spv.init.global), %10:pid(p1), %8:iid(s32)
%29:pid(p1) = nuw G_PTR_ADD %10:pid, %16:iid(s64)
%15:pid(p1) = nuw ASSIGN_TYPE %29:pid(p1), %28:type(s64)
%27:pid(p2) = G_BITCAST %15:pid(p1)
%17:pid(p2) = ASSIGN_TYPE %27:pid(p2), %22:type(s64)
G_STORE %1:iid(s32), %17:pid(p2) :: (store (s32) into %ir.3, align 8, addrspace 1)
```
On the next stage of instruction selection this `G_PTR_ADD`-related
pattern would be interpreted as an initialization of a global variable
and converted to an invalid constant GEP pattern that, in its turn,
would fail to be verified by LLVM during back translation from SPIR-V to
LLVM IR.
This PR introduces a fix for the problem by adding one more case of
`G_PTR_ADD` translation, when we use a non-const GEP to convey the
meaning. The reproducer is attached as a new test case.
This commits add the WaveIsFirstLane() hlsl intrinsinc. This intrinsic
uses the convergence intrinsincs for the SPIR-V backend. On the DXIL
side, I'm not sure what the strategy is for convergence, so I
implemented that like in DXC: a normal builtin function.
Signed-off-by: Nathan Gauër <brioche@google.com>
This adds the SPIRV fdot, sdot, and udot intrinsics and allows them to
be created at codegen depending on the target architecture. This
required moving some of the DXIL-specific choices to DXIL instruction
expansion out of codegen and providing it with at a more generic fdot
intrinsic as well.
Removed some stale comments that gave the obsolete impression that type
conversions should be expected to match overloads.
The SPIRV intrinsic handling involves generating multiply and add
operations for integers and the existing OpDot operation for floating
point.
New tests for generating SPIRV float and integer dot intrinsics are
added as well as expanding HLSL tests to include SPIRV generation
Used new dot product intrinsic generation to implement normalize() in SPIRV
Incidentally changed existing dot intrinsic definitions to use
DefaultAttrsIntrinsic to match the newly added inrinsics
Fixes#88056
This PR continues https://github.com/llvm/llvm-project/pull/101732
changes in virtual register processing aimed to improve correctness of
emitted MIR between passes from the perspective of MachineVerifier.
Namely, the following changes are introduced:
* register classes (lib/Target/SPIRV/SPIRVRegisterInfo.td) and
instruction patterns (lib/Target/SPIRV/SPIRVInstrInfo.td) are corrected
and simplified (by removing unnecessary sophisticated options) -- e.g.,
this PR gets rid of duplicating 32/64 bits patterns, removes ANYID
register class and simplifies definition of the rest of register
classes,
* hardcoded LLT scalar types in passes before instruction selection are
corrected -- the goal is to have correct bit width before instruction
selection, and use 64 bits registers for pattern matching in the
instruction selection pass; 32-bit registers remain where they are
described in such terms by SPIR-V specification (like, for example,
creation of virtual registers for scope/mem semantics operands),
* rework virtual register type/class assignment for calls/builtins
lowering,
* a series of minor changes to fix validity of emitted code between
passes:
- ensure that that bitcast changes the type,
- fix the pattern for instruction selection for OpExtInst,
- simplify inline asm operands usage,
- account for arbitrary integer sizes / update legalizer rules;
* add '-verify-machineinstrs' to existed test cases.
See also https://github.com/llvm/llvm-project/issues/88129 that this PR
may resolve.
This PR fixes a great number of issues reported by MachineVerifier and,
as a result, reduces a number of failed test cases for the mode with
expensive checks set on from ~200 to ~57.
Implement support for HLSL intrinsic saturate.
Implement DXIL codegen for the intrinsic saturate by lowering it to DXIL
Op dx.saturate.
Implement SPIRV codegen by transforming saturate(x) to clamp(x, 0.0f,
1.0f).
Add tests for DXIL and SPIRV CodeGen.
This PR addresses a TODO in
lib/Target/SPIRV/SPIRVInstructionSelector.cpp by adding implementation
of the non-const G_BUILD_VECTOR, and fix emission of the
OpGroupBroadcast instruction for the case when the `..._group_broadcast`
builtin has more than one `local_id` argument and `OpGroupBroadcast`
requires a newly constructed vector with 2 or 3 components instead of
originally passed series of `local_id` arguments.
This PR may resolve https://github.com/llvm/llvm-project/issues/97310 if
the reason for the reported fail is an incorrectly generated
OpGroupBroadcast instruction that was definitely a case.
Existing test is hardened and a new test is added to cover this special
case of the OpGroupBroadcast instruction emission.
This PR finishes #99134 by lowering the length function to the SPIR-V
backend. A test was added to verify that the generated SPIR-V is
correct.
Fixes#99134
This PR contains changes in virtual register processing aimed to improve
correctness of emitted MIR between passes from the perspective of
MachineVerifier. This potentially helps to detect previously missed
flaws in code emission and harden the test suite. As a measure of
correctness and usefulness of this PR we may use a mode with expensive
checks set on, and MachineVerifier reports problems in the test suite.
In order to satisfy Machine Verifier requirements to MIR correctness not
only a rework of usage of virtual registers' types and classes is
required, but also corrections into pre-legalizer and instruction
selection logics. Namely, the following changes are introduced:
* scalar virtual registers have proper bit width,
* detect register class by SPIR-V type,
* add a superclass for id virtual register classes,
* fix Tablegen rules used for instruction selection,
* fixes of minor existed issues (missed flag for proper representation
of a null constant for OpenCL vs. HLSL, wrong usage of integer virtual
registers as a synonym of any non-type virtual register).
The MachineModuleInfo reference was removed from the MachineFunction in
#100357, which broke this code. Instead of finding another way to get at
the MMI, just avoid using it - we can get at the LLVMContext from the
MachineFunction itself.
This PR improves type inference of operand presented by opaque pointers
and aggregate types:
* tries to restore original function return type for aggregate types so
that it's possible to deduce a correct type during emit-intrinsics step
(see llvm/test/CodeGen/SPIRV/SpecConstants/restore-spec-type.ll for the
reproducer of the previously existed issue when spirv-val found a
mismatch between object and ptr types in OpStore due to the incorrect
aggregate types tracing),
* explores untyped pointer operands of store to deduce correct pointee
types,
* creates an extension type to track pointee types from emit-intrinsics
step and further instead of direct and naive usage of TypePointerType
that led previously to crashes due to ban of creation of Value of
TypePointerType type,
* tracks instructions with uncomplete type information and tries to
improve their type info after pass calculated types for all machine
functions (it doesn't traverse a code but rather checks only those
instructions which were tracked as uncompleted),
* address more cases of removing unnecessary bitcasts (see, for example,
changes in test/CodeGen/SPIRV/transcoding/OpGenericCastToPtr.ll where
`CHECK-SPIRV-NEXT` in LIT checks show absence of unneeded bitcasts and
unmangled/mangled versions have proper typing now with equivalent type
info),
* address more cases of well known types or relations between types
within instructions (see, for example, atomic*.ll test cases and
Event-related test cases for improved SPIR-V code generated by the
Backend),
* fix the issue of removing unneeded ptrcast instructions in
pre-legalizer pass that led to creation of new assign-type instructions
with the same argument as source in ptrcast and caused errors in type
inference (the reproducer `complex.ll` test case is added to the PR).
This PR fixes the issue
https://github.com/llvm/llvm-project/issues/96614 by improve pattern
matching and tracking of constant integers. The attached test is
successful if it doesn't crash and generate valid SPIR-V code for both
32 and 64 bits targets.
This PR fixes https://github.com/llvm/llvm-project/issues/96513.
The way of creation of array type constant was incorrect: instead of
creating [1, 1, 1] or [1, 1, 1, 1, 1, ....] constants, the same [1]
constant was always created, substituting original composite constants.
This in its turn led to a situation when only one of constants might
exist in the code without emitting invalid code, the second constant
would be eventually rewritten to the first constant, because a key to
address both was an array of a single element (like [1]).
This PR fixes the issue and purges from the code unneeded copy/pasted
clone of the function that creates an array constant.
This PR continues https://github.com/llvm/llvm-project/pull/94467 and
contains fixes in emission of type intrinsics, constant recording and
corresponding test cases:
* type-deduce-global-dup.ll -- fix of integer constant emission on
32-bit platforms and correct type deduction for globals
* type-deduce-simple-for.ll -- fix of GEP translation (there was an
issue previously that led to incorrect translation/broken logic of
for-range implementation)
This PR also:
* fixes a cast between identical storage classes and updates the test
case to include validation run by spirv-val,
* ensures that Bitcast for pointers satisfies the requirement that the
address spaces must match and adds the corresponding test case,
* improve encode in Tablegen and decode in code of dependencies between
SPIR-V entities and required capability/extensions,
* prevent emission of identical OpTypePointer instructions.
This change is an implementation of #87367's investigation on supporting
IEEE math operations as intrinsics.
Which was discussed in this RFC:
https://discourse.llvm.org/t/rfc-all-the-math-intrinsics/78294
If you want an overarching view of how this will all connect see:
https://github.com/llvm/llvm-project/pull/90088
Changes:
- `llvm/docs/GlobalISel/GenericOpcode.rst` - Document the `G_FTAN`
opcode
- `llvm/include/llvm/IR/Intrinsics.td` - Create the tan intrinsic
- `llvm/include/llvm/Support/TargetOpcodes.def` - Create a `G_FTAN`
Opcode handler
- `llvm/include/llvm/Target/GenericOpcodes.td` - Define the `G_FTAN`
Opcode
- `llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp` Map the tan intrinsic
to `G_FTAN` Opcode
- `llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp` - Map the
`G_FTAN` opcode to the GLSL 4.5 and openCL tan instructions.
- `llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp` - Define `G_FTAN` as a
legal spirv target opcode.
Translating global values, IRTranslator pass can sometimes generates
code patterns that require additional efforts during pre-legalization.
This PR addresses this problem to support G_PTRTOINT instruction used in
initialization of GV.
This PR fixes the issue
https://github.com/llvm/llvm-project/issues/88908
Attached test case is updated to check that OpSConvert/OpUConvert is not
generated when input and result types are identical.
This PR resolves the issue that SPIR-V Backend uses the notion of a
pointer size of the target, most notably, in legalizer code, but
Tablegen instruction selection in SPIR-V Backend doesn't account for a
pointer size of the target. See
https://github.com/llvm/llvm-project/issues/88723 for a detailed
description. There are 3 test cases attached to the PR that reproduced
the issue, when dealing with spirv32-spirv64 differences, and are
working correctly now with this PR.
- `CGBuiltin.cpp` - Switch to using
`CGM.getHLSLRuntime().get##NAME##Intrinsic()`
- `CGHLSLRuntime.h` - Add any to backend intrinsic abstraction
- `IntrinsicsSPIRV.td` - Add any intrinsic to SPIR-V.
- `SPIRVInstructionSelector.cpp` - Add means of selecting any intrinsic.
Any and All share the same behavior up to the opCode. They are only
different in vector cases.
Completes #88045
The main point of this change was to add support for HLSL's all
intrinsic.
In the process of doing that I found a few issues around creating an
`OpConstantComposite` via `buildZerosVal`.
First the current code didn't support floats so the process of adding
`buildZerosValF` meant I needed a
float version of `getOrCreateIntConstVector`. After doing so I renamed
both versions to `getOrCreateConstVector`. That meant I needed to create
a float type version of `getOrCreateIntCompositeOrNull`. Luckily the
type information was low for this function so was able to split it out
into a helpwe and rename `getOrCreateIntCompositeOrNull` to
`getOrCreateCompositeOrNull` With the exception of type handling
differences of the code and Null vs 0 Constant Op codes these functions
should be identical.
To handle scalar floats I could not use `buildConstantFP` like this PR
did:
https://github.com/llvm/llvm-project/commit/0a2aaab5aba46#diff-733a189c5a8c3211f3a04fd6e719952a3fa231eadd8a7f11e6ecf1e584d57411R1603
because that would create too many superfluous registers (that causes
problems in the validator), I had to create a float version of
`getOrCreateConstInt` which I called `getOrCreateConstFP`.
similar problems with doing it like this:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp#L1540.
`buildZerosValF` also has a use of a function `getZeroFP`. This is
because half, float, and double scalar values of 0 would collide in
`SPIRVDuplicatesTracker<Constant> CT` if you use `APFloat(0.0f)`.
`getORCreateConstFP` needed its own version of `getOrCreateConstIntReg`
which I called `getOrCreateConstFloatReg` The one difference in this
function is `getOrCreateConstFloatReg` returns a bit width so we don't
have to call `getScalarOrVectorBitWidth` twice ie when it is used again
in `getOrCreateConstFP` for `OpConstantF` `addNumImm`.
`getOrCreateConstFloatReg` needed an `assignFloatTypeToVReg` helper
which called a `getOrCreateSPIRVFloatType` helper. There was no
equivalent IntegerType::get for floats so I handled this with a switch
statement on bit widths to get the right LLVM float type.
Finally, there is the use of `bool ZeroAsNull = STI.isOpenCLEnv();` This
is partly a cosmetic change. When Zeros are treated as nulls, we don't
create `OpConstantComposite` vectors which is something we do in the
DXCs SPIRV backend. The DXC SPIRV backend also does not use
`OpConstantNull`. Finally, I needed a means to test the behavior of the
OpConstantNull and `OpConstantComposite` changes and this was one way I
could do that via the same tests.
SPIR-V intsruction selection was mapping the LLVM float min/max
intrinsics to FMin and FMax respectively for GL/Vulkan environments,
which does not match the intrinsics' documented treatment of NaN
operands. This patch switches the mapping to the correctly matched NMin
and NMax operations.
Fixes#87072
This PR:
* fixes OpVariable instructions place in a function (see
https://github.com/llvm/llvm-project/issues/66261),
* improves type inference,
* helps avoiding unneeded bitcasts when validating function call's
This allows to improve existing and add new test cases with more strict
checks. OpVariable fix refers to "All OpVariable instructions in a
function must be the first instructions in the first block" requirement
from SPIR-V spec.
This PR improves type inference in general and deduces types of
composite data structures in particular. Also added a way to insert a
bitcast to make a fun call valid in case of arguments types mismatch due
to opaque pointers type inference.
The attached test `pointers/nested-struct-opaque-pointers.ll`
demonstrates new capabilities: the SPIRV code emitted for this test is
now (1) valid in a sense of data field types and (2) accepted by
`spirv-val`.
More strict LIT checks, support of more composite data structures and
improvement of fun calls from the perspective of type correctness are
main todo's at the moment.
This PR fixes illegal use of OpConstantComposite with non-constant
constituents. The test attached to the PR is able now to satisfy
`spirv-val` check. Before the fix SPIR-V Backend produced for the
attached test case a pattern like
```
%a = OpVariable %_ptr_CrossWorkgroup_uint CrossWorkgroup %uint_123
%11 = OpConstantComposite %_struct_6 %a %a
```
so that `spirv-val` complained with
```
error: line 25: OpConstantComposite Constituent <id> '10[%a]' is not a constant or undef.
%11 = OpConstantComposite %_struct_6 %a %a
```
This PR improves type inference in SPIR-V Backend for opaque pointers,
accounting or a case when there is a chain of function calls that allows
to deduce formal parameter types from actual arguments. The attached
test demonstrates the case.
This PR:
* adds __spirv_ builtins for existing instructions;
* fixes parsing of "syncscope" values in atomic instructions;
* fix a special case of binary header emision.
This PR:
* adds Lifetime intrinsics/instructions
* fixes how the binary header is emitted (correct version and better
approximation of Bound)
* add validation into more test cases
This PR:
* adds support for G_SPLAT_VECTOR generic opcode that may be legally
generated instead of G_BUILD_VECTOR by previous passes of the translator
(see https://github.com/llvm/llvm-project/pull/80378 for the source of
breaking changes);
* improves deduction of types for opaque pointers.
This PR also fixes the following issues:
* if a function has ptr argument(s), two functions that have different
SPIR-V type definitions may get identical LLVM function types and break
agreements of global register and duplicate checker;
* checks for pointer types do not account for TypedPointerType.
Update of tests:
* A test case is added to cover the issue with function ptr parameters.
* The first case, that is support for G_SPLAT_VECTOR generic opcode, is
covered by existing test cases.
* Multiple additional checks by `spirv-val` is added to cover more
possibilities of generation of invalid code.
Add SPIR-V backend support for the HLSL SV_DispatchThreadID semantic
attribute, which is lowered to a @llvm.dx.thread.id intrinsic in LLVM
IR. In the SPIR-V backend, this is now correctly translated to a
`GlobalInvocationId` builtin variable.
Fixes#82534
This PR is to fix a way how SPIR-V Backend describes legality of
OpBitcast instruction and how it is validated on a step of instruction
selection. Instead of checking a size of virtual registers (that makes
no sense due to lack of guarantee of direct relations between size of
virtual register and bit width associated with the type size), this PR
allows to legalize OpBitcast without size check and postpones validation
to the instruction selection step.
As an example, let's consider the next example that was copied as is
from a bigger test suite:
```
%355:id(s16) = G_BITCAST %301:id(s32)
%303:id(s16) = ASSIGN_TYPE %355:id(s16), %349:type(s32)
%644:fid(s32) = G_FMUL %645:fid, %646:fid
%301:id(s32) = ASSIGN_TYPE %644:fid(s32), %40:type(s32)
```
Without the PR this leads to a crash with complains to an illegal
bitcast, because %355 is s16 and %301 is s32. However, we must check not
virtual registers in this case, but types of %355 and %301, i.e.,
%349:type(s32) and %40:type(s32), which are perfectly well compatible in
a sense of OpBitcast in this case.
In a test case that is a part of this PR OpBitcast is legal, being
applied for `OpTypeInt 16` and `OpTypeFloat 16`, but would not be
legalized without this PR due to virtual registers defined as having
size 16 and 32.
This PR is to add vector reduction instructions according to
https://llvm.org/docs/GlobalISel/GenericOpcode.html#vector-reduction-operations
and widen in such a way a range of successful supported conversions,
covering new cases of vector reduction instructions which IRTranslator
is unable to resolve.
By legalizing vector reduction instructions we introduce a new
instruction patterns that should be addressed, including patterns that
are delegated to pre-legalize step. To address this problem, a new pass
is added that is to bring newly generated instructions after
legalization to an aspect required by instruction selection.
Expected overheads for existing cases is minimal, because a new pass is
working only with newly introduced instructions, otherwise it's just a
additional code traverse without any actions.
This PR adds SPIR-V extension SPV_INTEL_variable_length_array that
allows to allocate local arrays whose number of elements is unknown at
compile time:
* add a new SPIR-V internal intrinsic:int_spv_alloca_array
* legalize G_STACKSAVE and G_STACKRESTORE
* implement allocation of arrays (previously getArraySize() of
AllocaInst was not used)
* add tests
This PR is to add support for the 'freeze' instruction:
https://llvm.org/docs/LangRef.html#freeze-instruction
There is no way to implement `freeze` correctly without support on
SPIR-V standard side, but we may at least address a simple (static) case
when undef/poison value presence is obvious. The main benefit of even
incomplete `freeze` support is preventing of translation from crashing
due to lack of support on legalization and instruction selection steps.
This PR adds support for atomic instruction on floating-point numbers:
* SPV_EXT_shader_atomic_float_add
* SPV_EXT_shader_atomic_float_min_max
* SPV_EXT_shader_atomic_float16_add
and fixes asm printer output for half floating-type.
This PR adds support for the SPV_KHR_linkonce_odr extension and modifies
existing negative test with a positive check for the extension and
proper linkage type in case when the extension is enabled.
SPV_KHR_linkonce_odr adds a "LinkOnceODR" linkage type, allowing proper
translation of, for example, C++ templates classes merging during
linking from different modules and supporting any other cases when a
global variable/function must be merged with equivalent global
variable(s)/function(s) from other modules during the linking process.
The goal of this PR is to fix the issue of global unnamed variables
causing SPIR-V Backend code generation to crash:
https://github.com/llvm/llvm-project/issues/78278
The reason for the crash is that GlobalValue's getGlobalIdentifier()
would fail for unnamed global variable when trying to access the first
character of the name (see lib/IR/Globals.cpp:150). This leads to assert
in Debug and undefined behaviour in Release builds.
The proposed fix generates a name of an unnamed global variable as
__unnamed_<unsigned number>, in a style of similar existing LLVM
implementation (see lib/IR/Mangler.cpp:131). A new class member variable
is added into `SPIRVInstructionSelector` class to keep track of the
number we give to anonymous global values to generate the same name
every time when this is needed.
The patch adds a new LIT test with the smallest implementation of
reproducer ll code.
Since efe0e10718 changes in tests are required. Need to add extension to
Extensions list
and command line to enable use of the extension for test runs.
There is no log10 instruction in the GLSL Extended Instruction Set so to
implement the HLSL log10 intrinsic when targeting Vulkan this change
adds the logic to derive the result using the following formula:
```
log10(x) = log2(x) * (1 / log2(10))
= log2(x) * 0.30103
```
