There are llvm intrinsics which we do not expect to actually represent
code after lowering or which are not implemented yet but can be found in
a customer's LLVM IR input. We do not want translation to crash when
these llvm intrinsics are found, and this PR fixes the issue with
translation crash for some known cases, aligned with Khronos Translator.
This PR improves type inference and fixes inconsistency between
previously deduced element type of a pointer and function's return type.
It fixes https://github.com/llvm/llvm-project/issues/109401 by ensuring
that OpPhi is consistent with respect to operand types.
Two main goals of this PR are:
* to support "Arithmetic with Overflow" intrinsics, including the
special case when those intrinsics are being generated by the
CodeGenPrepare pass during translations with optimization;
* to redirect intrinsics with aggregate return type to be lowered via
GlobalISel operations instead of SPIRV-specific unfolding/lowering (see
https://github.com/llvm/llvm-project/pull/95012).
There is a new test case
`llvm/test/CodeGen/SPIRV/passes/translate-aggregate-uaddo.ll` that
describes and checks the general logics of the translation.
This PR continues a series of PRs aimed to identify and fix flaws in
code emission, to improve pass rates for the mode with expensive checks
set on (see https://github.com/llvm/llvm-project/pull/101732,
https://github.com/llvm/llvm-project/pull/104104,
https://github.com/llvm/llvm-project/pull/106966), having in mind the
ultimate goal of proceeding towards the non-experimental status of
SPIR-V Backend.
The reproducers are:
1) consider `llc -O3 -mtriple=spirv64-unknown-unknown ...` with:
```
define spir_func i32 @foo(i32 %a, ptr addrspace(4) %p) {
entry:
br label %l1
l1:
%e = phi i32 [ %a, %entry ], [ %i, %body ]
%i = add nsw i32 %e, 1
%fl = icmp eq i32 %i, 0
br i1 %fl, label %exit, label %body
body:
store i8 42, ptr addrspace(4) %p
br label %l1
exit:
ret i32 %i
}
```
2) consider `llc -O0 -mtriple=spirv64-unknown-unknown ...` with:
```
define spir_func i32 @foo(i32 %a, ptr addrspace(4) %p) {
entry:
br label %l1
l1: ; preds = %body, %entry
%e = phi i32 [ %a, %entry ], [ %math, %body ]
%0 = call { i32, i1 } @llvm.uadd.with.overflow.i32(i32 %e, i32 1)
%math = extractvalue { i32, i1 } %0, 0
%ov = extractvalue { i32, i1 } %0, 1
br i1 %ov, label %exit, label %body
body: ; preds = %l1
store i8 42, ptr addrspace(4) %p, align 1
br label %l1
exit: ; preds = %l1
ret i32 %math
}
```
This change adds support for correctly lowering the `__scoped` Clang
builtins, and corresponding scoped LLVM instructions. These were
previously unconditionally lowered to Device scope, which is possibly incorrect.
Furthermore, the default / implicit scope is changed from Device (an
OpenCL assumption) to AllSvmDevices (aka System), since the SPIR-V BE is not
OpenCL specific / can ingest IR coming from other language front-ends. OpenCL
defaulting to Device scope is now reflected in the front-end handling of atomic
ops, which seems preferable.
An unordered_set include was removed from a header in 04ebd1907c05
"[MC][NFC] Statically allocate storage for decoded pseudo probes and
function records (#102789)", but SPIRVEmitIntrinsics was getting the
definition from that transitive include. Fix the build by including the
header explicitly.
This PR improves type inference of operand presented by opaque pointers
and aggregate types:
* tries to restore original function return type for aggregate types so
that it's possible to deduce a correct type during emit-intrinsics step
(see llvm/test/CodeGen/SPIRV/SpecConstants/restore-spec-type.ll for the
reproducer of the previously existed issue when spirv-val found a
mismatch between object and ptr types in OpStore due to the incorrect
aggregate types tracing),
* explores untyped pointer operands of store to deduce correct pointee
types,
* creates an extension type to track pointee types from emit-intrinsics
step and further instead of direct and naive usage of TypePointerType
that led previously to crashes due to ban of creation of Value of
TypePointerType type,
* tracks instructions with uncomplete type information and tries to
improve their type info after pass calculated types for all machine
functions (it doesn't traverse a code but rather checks only those
instructions which were tracked as uncompleted),
* address more cases of removing unnecessary bitcasts (see, for example,
changes in test/CodeGen/SPIRV/transcoding/OpGenericCastToPtr.ll where
`CHECK-SPIRV-NEXT` in LIT checks show absence of unneeded bitcasts and
unmangled/mangled versions have proper typing now with equivalent type
info),
* address more cases of well known types or relations between types
within instructions (see, for example, atomic*.ll test cases and
Event-related test cases for improved SPIR-V code generated by the
Backend),
* fix the issue of removing unneeded ptrcast instructions in
pre-legalizer pass that led to creation of new assign-type instructions
with the same argument as source in ptrcast and caused errors in type
inference (the reproducer `complex.ll` test case is added to the PR).
This PR improves type inference for a known instruction's builtin:
OpGroupAsyncCopy:
* deduce a type of one source/destination pointer when it's possible to
deduce a type of another argument, and
* validate src and dest types and tries to unfold a parameter if it's a
structure wrapper around a scalar/vector type.
This PR completes implementation of insertion of OpGenericCastToPtr
using builtin functions started by
https://github.com/llvm/llvm-project/pull/95055 by:
* fixing errors in Tablegen definition,
* adding type inference info for `__spirv_GenericCastToPtrExplicit` kind
of wrappers, and
* hardening the test case to check correct translation of
`__spirv_GenericCastToPtrExplicit` kind of wrappers.
Uses the new InsertPosition class (added in #94226) to simplify some of
the IRBuilder interface, and removes the need to pass a BasicBlock
alongside a BasicBlock::iterator, using the fact that we can now get the
parent basic block from the iterator even if it points to the sentinel.
This patch removes the BasicBlock argument from each constructor or call
to setInsertPoint.
This has no functional effect, but later on as we look to remove the
`Instruction *InsertBefore` argument from instruction-creation
(discussed
[here](https://discourse.llvm.org/t/psa-instruction-constructors-changing-to-iterator-only-insertion/77845)),
this will simplify the process by allowing us to deprecate the
InsertPosition constructor directly and catch all the cases where we use
instructions rather than iterators.
This PR implements insertion of OpGenericCastToPtr using builtin
functions (both opencl `to_global|local|private` and `__spirv_`
wrappers), and improves type inference.
This PR continues https://github.com/llvm/llvm-project/pull/94467 and
contains fixes in emission of type intrinsics, constant recording and
corresponding test cases:
* type-deduce-global-dup.ll -- fix of integer constant emission on
32-bit platforms and correct type deduction for globals
* type-deduce-simple-for.ll -- fix of GEP translation (there was an
issue previously that led to incorrect translation/broken logic of
for-range implementation)
This PR also:
* fixes a cast between identical storage classes and updates the test
case to include validation run by spirv-val,
* ensures that Bitcast for pointers satisfies the requirement that the
address spaces must match and adds the corresponding test case,
* improve encode in Tablegen and decode in code of dependencies between
SPIR-V entities and required capability/extensions,
* prevent emission of identical OpTypePointer instructions.
This PR is to add validation to the test case with
get_image_array_size/get_image_dim calls
(transcoding/check_ro_qualifier.ll). This test case didn't pass
validation because of invalid emission of OpCompositeExtract instruction
(Result Type must be the same type as Composite.).
In order to fix the problem this PR improves type inference in general
and partially addresses issues:
* https://github.com/llvm/llvm-project/issues/91998
* https://github.com/llvm/llvm-project/issues/91997
A reproducer from the description of the latter issue is added as a new
test case as a part of this PR.
The structurizer required regions to be SESE: single entry, single exit.
This new pass transforms multiple-exit regions into single-exit regions.
```
+---+
| A |
+---+
/ \
+---+ +---+
| B | | C | A, B & C belongs to the same convergence region.
+---+ +---+
| |
+---+ +---+
| D | | E | C & D belongs to the parent convergence region.
+---+ +---+ This means B & C are the exit blocks of the region.
\ / And D & E the targets of those exits.
\ /
|
+---+
| F |
+---+
```
This pass would assign one value per exit target:
B = 0
C = 1
Then, create one variable per exit block (B, C), and assign it to the
correct value: in B, the variable will have the value 0, and in C, the
value 1.
Then, we'd create a new block H, with a PHI node to gather those 2
variables, and a switch, to route to the correct target.
Finally, the branches in B and C are updated to exit to this new block.
```
+---+
| A |
+---+
/ \
+---+ +---+
| B | | C |
+---+ +---+
\ /
+---+
| H |
+---+
/ \
+---+ +---+
| D | | E |
+---+ +---+
\ /
\ /
|
+---+
| F |
+---+
```
Note: the variable is set depending on the condition used to branch. If
B's terminator was conditional, the variable would be set using a
SELECT.
All internal edges of a region are left intact, only exiting edges are
updated.
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
This PR implements correct zeroinitializer for extension types in SPIR-V
Backend.
Previous version has just created 0 of 32/64 integer type (depending on
target machine word size), that caused re-use and type re-write of the
corresponding integer constant 0 with a potential crash on wrong usage
of the constant (i.e., 0 of integer type expected but extension type
found). E.g., the following code would crash without the PR:
```
%r1 = icmp ne i64 %_arg_i, 0
%e1 = tail call spir_func target("spirv.Event") @__spirv_GroupAsyncCopy(i32 2, ptr addrspace(3) %_arg_local, ptr addrspace(1) %_arg_ptr, i64 1, i64 1, target("spirv.Event") zeroinitializer)
```
because 0 in icmp would eventually be of `Event` type.
This PR introduces support for inline assembly calls for SPIR-V Backend
in general, and support for SPV_INTEL_inline_assembly [1] extension in
particular. The former part of the PR is agnostic towards
vendor-specific requirements and resolves the task of supporting
successful transformation of inline assembly as long as it's possible
without specific SPIR-V instruction codes.
As a part of the PR there appears an opportunity to bring coherent
inline assembly information up to latest passes of the transformation
process (emitting final SPIR-V instructions), so that PR makes it easy
to add any another required flavor of inline assembly, other then
supported by the vendor specific SPV_INTEL_inline_assembly extension,
if/when needed.
At the moment, however, SPV_INTEL_inline_assembly is the only
implemented way to bring LLVM IR inline assembly calls up to valid
SPIR-V instructions and also the default one. This means that inline
assembly calls will generate an error message of such extension is not
used to prevent LLVM-generated error messages at the final stages of
translation. When the SPV_INTEL_inline_assembly extension is mentioned
among supported, translation of inline assembly is intercepted by this
extension implementation on a pre-legalizer step, and this is a place
where support for a new inline assembly extension may be added if
needed.
This PR also extends support for register classes, improves type
inference during pre-legalizer pass, and fixes a minor bug with
asm-printing of string literals.
[1]
https://github.com/intel/llvm/blob/sycl/sycl/doc/design/spirv-extensions/SPV_INTEL_inline_assembly.asciidoc
The goal of the PR is to ensure that newly inserted `ptrcast` internal
intrinsic functions are inserted at the correct positions, and don't
break rules of instruction domination and PHI nodes grouping at top of
basic block.
This PR generation of argument types of internal intrinsic functions
`spv_const_composite` and `spv_track_constant`, so that composite
constants of ConstantVector type preserve their correct type in
transformation passes and can be successfully used further by LLVM
intrinsic functions.
The added test case serves two purposes: it is to check the above
mentioned fix and to demonstrate that a call to __builtin_alloca() maps
to instructions from SPV_INTEL_variable_length_array when this extension
is available.
This PR is to ensure that internal intrinsic functions for PHI's operand
are inserted at the correct positions and don't break rules of
instruction domination and PHI nodes grouping at top of basic block.
This PR is to improve builtins matching and type inference in SPIR-V
Backend. The model test case is printf call from OpenCL.std that has
several features allowing for a wider look at builtins support/type
inference:
(1) call in a "spirv-friendly" style (prefixed by __spirv_ocl_)
(2) restricted type of the 1st argument
Attached test cases checks several possible inputs. Support of the
extension SPV_EXT_relaxed_printf_string_address_space is to do (see:
https://github.com/KhronosGroup/SPIRV-Registry/blob/main/extensions/EXT/SPV_EXT_relaxed_printf_string_address_space.asciidoc).
This PR also fixes target ext type constants and
OpGroupAsyncCopy/OpGroupWaitEvents generation. A new test case is
attached.
This PR contains a series of fixes which are to improve type inference
and instruction selection.
Namely, it includes:
* fix OpSelect to support operands of a pointer type, according to the
SPIR-V specification (previously only integer/float/vectors of integer
or float were supported) -- a new test case is added and existing test
case is updated;
* fix TableGen typo's in definition of register classes and introduce a
new reg class that is a vector of pointers;
* fix usage of a machine function context when there is a need to switch
between different machine functions to infer/validate correct types;
* add usage of TypedPointerType instead of PointerType so that later
stages of type inference are able to distinguish pointer types by their
element types, effectively supporting hierarchy of pointer/pointee types
and avoiding more complicated recursive type matching on level of
machine instructions in favor of direct pointer comparison using LLVM's
`Type *` values;
* extracting detailed information about operand types using known type
rules for some llvm instructions (for instance, by deducing PHI's
operand pointee types if PHI's results type was deducted on previous
stages of type inference), and adding correspondent
`Intrinsic::spv_assign_ptr_type` to keep type info along consequent
passes,
* ensure that OpConstantComposite reuses a constant when it's already
created and available in the same machine function -- otherwise there is
a crash while building a dependency graph, the corresponding test case
is attached,
* implement deduction of function's return type for opaque pointers, a
new test case is attached,
* make 'emit intrinsics' a module pass to resolve function return types
over the module -- first types for all functions of the module must be
calculated, and only after that it's feasible to deduct function return
types on this earlier stage of translation.
This PR fixes issue https://github.com/llvm/llvm-project/issues/87763
and preserves valid CFG in cases when previous scheme failed to generate
valid code for a switch statement. The PR hardens one existing test case
and adds one more test case as a validation of a new switch generation.
Tests are passing spirv-val now.
This PR also improves validation of forward calls.
This PR improves type inference in general and deduces types of
composite data structures in particular. Also added a way to insert a
bitcast to make a fun call valid in case of arguments types mismatch due
to opaque pointers type inference.
The attached test `pointers/nested-struct-opaque-pointers.ll`
demonstrates new capabilities: the SPIRV code emitted for this test is
now (1) valid in a sense of data field types and (2) accepted by
`spirv-val`.
More strict LIT checks, support of more composite data structures and
improvement of fun calls from the perspective of type correctness are
main todo's at the moment.
This PR improves type inference in SPIR-V Backend for opaque pointers,
accounting or a case when there is a chain of function calls that allows
to deduce formal parameter types from actual arguments. The attached
test demonstrates the case.
This PR:
* adds support for G_SPLAT_VECTOR generic opcode that may be legally
generated instead of G_BUILD_VECTOR by previous passes of the translator
(see https://github.com/llvm/llvm-project/pull/80378 for the source of
breaking changes);
* improves deduction of types for opaque pointers.
This PR also fixes the following issues:
* if a function has ptr argument(s), two functions that have different
SPIR-V type definitions may get identical LLVM function types and break
agreements of global register and duplicate checker;
* checks for pointer types do not account for TypedPointerType.
Update of tests:
* A test case is added to cover the issue with function ptr parameters.
* The first case, that is support for G_SPLAT_VECTOR generic opcode, is
covered by existing test cases.
* Multiple additional checks by `spirv-val` is added to cover more
possibilities of generation of invalid code.
This PR introduces a step after instruction selection where instructions
can be traversed from the perspective of their validity from the
specification point of view. The PR adds also a way to correct
load/store when there is a type mismatch contradicting the specification
-- an additional bitcast is inserted to keep types consistent.
Correspondent test cases are added and existing test cases are
corrected.
This PR helps to successfully validate with the `spirv-val` tool
(https://github.com/KhronosGroup/SPIRV-Tools) some output that
previously led to validation errors and crashes of back translation from
SPIRV to LLVM IR from the side of SPIRV Translator project
(https://github.com/KhronosGroup/SPIRV-LLVM-Translator).
The added step of bringing instructions to required by the specification
type correspondence can be (should be and will be) extended beyond
load/store instructions to ensure validity rules of other SPIRV
instructions related to type inference.
The architecture of SPIRVEmitIntrinsics is build in such way that every
private method is called by one main function runOnFunction which then
calls private methods. Private member IRB is allocated in runOnFunction
method but it's not freed. Due to that every time when IR function
contains intrinsics to emit, runOnFunction is entered and memory is
leaked on exit. It's especially true when there are two or more IR
functions to emit. IRB is set to nullptr during construction of object
and it's left without pointing resource until runOnFunction is entered.
This also create possibility of simple mistake when private method is
called but there is no resource pointed. Change requires passing
IRBuilder by reference to private methods. The visit* functions create
it's own IRBuilder thus IRB is eliminated from class scope. In addition
there is a small performance improvement because IRBuilder is not
allocated by heap.
This pull request aims to remove any dependency on OpenCL/SPIR-V type
information in LLVM IR metadata. While, using metadata might simplify
and prettify the resulting SPIR-V output (and restore some of the
information missed in the transformation to opaque pointers), the
overall methodology for resolving kernel parameter types is highly
inefficient.
The high-level strategy is to assign kernel parameter types in this order:
1. Resolving the types using builtin function calls as mangled names
must contain type information or by looking up builtin definition in
SPIRVBuiltins.td. Then:
- Assigning the type temporarily using an intrinsic and later setting
the right SPIR-V type in SPIRVGlobalRegistry after IRTranslation
- Inserting a bitcast
2. Defaulting to LLVM IR types (in case of pointers the generic i8*
type or types from byval/byref attributes)
In case of type incompatibility (e.g. parameter defined initially as
sampler_t and later used as image_t) the error will be found early on
before IRTranslation (in the SPIRVEmitIntrinsics pass).
This PR adds SPIR-V extension SPV_INTEL_variable_length_array that
allows to allocate local arrays whose number of elements is unknown at
compile time:
* add a new SPIR-V internal intrinsic:int_spv_alloca_array
* legalize G_STACKSAVE and G_STACKRESTORE
* implement allocation of arrays (previously getArraySize() of
AllocaInst was not used)
* add tests
This pull request fixes an issue with missing vector element count
immediate in OpExtInst calls and adds a case for generating bitcasts
before GEPs for kernel arguments of non-matching pointer type. The new
LITs are based on basic/vload_local and basic/vload_global OpenCL CTS
tests. The tests after this change pass SPIR-V validation.
The goal of this PR is to fix the issue when use of token type in LLVM
intrinsic causes undefined behavior of SPIR-V Backend code generator
when assertions are disabled:
https://github.com/llvm/llvm-project/issues/78434
Among possible fix options, discussed in the
https://github.com/llvm/llvm-project/issues/78434 issue description, the
option to generate a meaningful error before execution arrives at the
`llvm_unreachable` call looks like a better solution for now, because
SPIR-V doesn't support token type anyway without additional extensions.
The PR is to generate a user-friendly error message and exit without
generating a stack dump when such a usage of token type was detected
that would lead to undefined behavior of SPIR-V Backend code generator.
Handle a special case when StoreInst's value operand is a kernel
argument of a pointer type. Since these arguments could have either a
basic element type (e.g. float*) or OpenCL builtin type (sampler_t),
bitcast the StoreInst's value operand to default pointer element type
(i8).
This pull request addresses the issue
https://github.com/llvm/llvm-project/issues/72864
Prior to this change spv_ptrcast (and OpBitcast) was never emitted for
GEP resulting pointers. While such SPIR-V was (mostly) accepted by the
NEO GPU driver, the generated SPIR-V was incorrect.
The newly added test (pointers/getelementptr-bitcast-load.ll) verifies
that a correct bitcast is added for more complex cases and passes
spirv-val. The test is based on an OpenCL CTS test (basic/prefetch).
This patch introduces a new spv_ptrcast intrinsic for tracking expected
pointer types. The change fixes multiple OpenCL CTS regressions due the
switch to opaque pointers (e.g. basic/hiloeo).
- After #73887, spirv.Image cannot has a zeroinitializer, even though
it's only used in metadata to pass down the type info to the backend.
Instead of creating zeros, create undef ones of that target-specific
types.
This commit adds 2 new instructions in the selector:
- OpAccessChain
- OpInBoundsAccessChain.
The choice between the two relies on the `inbounds` marker.
Those instruction are not used for OpenCL, to maintain the same
behavior as previously. They are only added when building for logical
SPIR-V, as it doesn't support the pointer equivalent.
Because logical SPIR-V doesn't support pointer cast either, the
assign_ptr_type intrinsic need to be generated so OpAccessChain gets
lowered with the correct pointer type, instead of i8*.
Fixes#66107
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
