Only with SPV_INTEL_arbitrary_precision_integers SPIR-V Backend creates
arbitrary sized integer types (<= 64 bits). Without such extension and
according to the SPIR-V specification
`SPIRVGlobalRegistry::getOpTypeInt()` rounds integer sizes other than
8,16,32,64 up, to one of defined by the specification sizes. For the
`DuplicateTracker` class this means that several original LLVM types
(e.g., i2, i4) map to the same "OpTypeInt 8" instruction. This breaks
`DuplicateTracker`'s logic and leads to generation of invalid SPIR-V
code eventually.
For example,
```
define spir_func void @foo(i2 %a, i4 %b) {
entry:
%res2 = tail call i2 @llvm.bitreverse.i2(i2 %a)
%res4 = tail call i4 @llvm.bitreverse.i4(i4 %b)
ret void
}
declare i2 @llvm.bitreverse.i2(i2)
declare i4 @llvm.bitreverse.i4(i4)
```
after translation to SPIR-V would fail during validation (`spirv-val`)
due to two `OpTypeInt 8 0` instructions.
This PR fixes the issue by changing source LLVM type according to the
SPIR-V type that will be used in the emitted code.
Add check of number of operands for concat_vectors being equal to 2.
This can
avoid crash if there are more than 2 operands for concat_vectors and
operand 0 & 1 are undef value.
Implement emitPrologue/emitEpilogue methods, determine/spill/restore
callee saved registers functionality with test. Also implement lowering
of the DYNAMIC_STACKALLOC/STACKSAVE/STACKRESTORE stack operations with
tests.
This adds compare and branch instructions fusion for Neoverse V2.
According to the Software Optimization Guide:
Specific Aarch64 instruction pairs that can be fused are as follows:
CMP/CMN (immediate) + B.cond
CMP/CMN (register) + B.cond
Performance for SPEC2017 is neutral, but another benchmark improves
significantly.
Results for SPEC2017 on a Neoverse V2:
500.perlbench 0%
502.gcc_r 0%
505.mcf_r -0.15%
523.xalancbmk_r -0.43%
525.x264_r 0%
531.deepsjeng_r 0%
541.leela_r -0.16%
557.xz_r -0.47%
Remove support for the icmp and fcmp constant expressions.
This is part of:
https://discourse.llvm.org/t/rfc-remove-most-constant-expressions/63179
As usual, many of the updated tests will no longer test what they were
originally intended to -- this is hard to preserve when constant
expressions get removed, and in many cases just impossible as the
existence of a specific kind of constant expression was the cause of the
issue in the first place.
These blocks usually show up in the form of branches within inline
assembly. Since it's hard to rewire them, we fully omit paths with such
blocks from path cloning.
- Fix build with `EXPENSIVE_CHECKS`
- Remove unused `PassName::ID` to resolve warning
- Mark `~SelectionDAGISel` virtual so AArch64 backend can work properly
Adds AArch64 support for the `preserve_none` calling convention.
Registers X0-X7, X9-X15 and X19-X28 are caller save, and can be used to
pass arguments. Delegates to AAPCS for all other registers.
Closes#87423
Having the version in `get_run_handlers` meant that a RUN line without the
version in it would not match, and therefore UTC would generate a useless
catch-all check of the form:
```
;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
; CHECK: {{.*}}
```
This patch also adds `arm64_32` as a base-level run handler, and assumes it
will always apply to a darwin target (which is currently the case for all
arm64_32-* triples I see in tree currently).
Since pointers in memory, as well as the index type are both 32 bits,
but in registers pointers are 64 bits, the mask generated by
llvm.ptrmask needs to be zero-extended.
Fixes: #94075
Fixes: rdar://125263567
The various preloaded registers (workitem IDs, workgroup IDs, and
various implicit pointers) always have a finite, invariant, well-defined
value throughout a well-defined program.
In cases where the compiler infers or the user declares that some
implicit input will not be used (ex. via amdgcn-no-workitem-id-y), the
behavior of the entire program is undefined, since that misdeclaration
can cause arbitrary other preloaded-register intrinsics to access the
wrong register. This case is not expected to arise in practice, but
could occur when the no implicit argument attributes were not cleared
correctly in the presence of external functions, indrect calls, or other
means of executing un-analyzable code. Failure to detect that case would
be a bug in the attributor.
This commit updates the documentation to reflect this long-standing
reality.
Then, on the basis that all implicit arguments are defined in all
correct programs, the intrinsics that return those values are
annototated with `noundef``. Some implicit pointer arguments gain a
`nonnull`, but the kernel argument segment pointer or implicit argument
pointers don't necessarily have this property.
This will prevent spurious calls to `freeze` in front-end optimizations
that destroy user-provided ranges on built-in IDs.
(While I'm here, this commit adds a test for `noundef` on kernel
arguments which is currently unimplemented)
This reduce the time complexity of the main loop of `findCandidates()`
method from $O(n^2)$ to $O(n \log n)$.
For small $n$, the modification does not regress the build time, but it
helps significantly when $n$ is large.
For one application, this reduces the runtime of the main loop from 120
seconds to 28 seconds.
This is the first commit for an enhanced version of machine outliner --
see
[RFC](https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-1-fulllto-part-2-thinlto-nolto-to-come/78732).
We add a feature that prevents the GlobalMerge pass from considering
data smaller than a minimum size in bytes for merging.
The MinSize is set in 3 ways:
1. If global-merge-min-data-size is explicitly set, then it uses that
value.
2. If SmallDataLimit is set and non-zero, then SmallDataLimit + 1 is
used.
3. Otherwise, 0 is used, which means all sizes are considered for
merging.
We found that this feature allowed us to see the benefit of the
GlobalMerge pass while eliminating some merging that was not beneficial.
This feature allowed us to enable the GlobalMerge pass on RISC-V in our
downstream by default because it led to improvements on multiple
benchmark suites.
I plan to post a separate patch to propose enabling this by default on
RISC-V. But I do not want that discussion to be part of the discussion
of adding this feature, so I am keeping the patches separate.
Currently we only match GEPs with a scalar base pointer, but a common
pattern that's emitted from the loop vectorizer is a strided vector base
plus some sort of scalar offset:
%base = getelementptr i64, ptr %p, <vscale x 1 x i64> %step
%gep = getelementptr i64, <vscale x 1 x ptr> %base, i64 %offset
This is common for accesses into a struct e.g. f[i].b below:
struct F { int a; char b; };
void foo(struct F *f) {
for (int i = 0; i < 1024; i += 2) {
f[i].a++;
f[i].b++;
}
}
This patch handles this case in RISCVGatherScatterLowering by recursing
on the base pointer if it's a vector.
With this we can convert roughly 80% of the indexed loads and stores
emitted to strided loads and stores on SPEC CPU 2017, -O3
-march=rva22u64_v
The structurizer required regions to be SESE: single entry, single exit.
This new pass transforms multiple-exit regions into single-exit regions.
```
+---+
| A |
+---+
/ \
+---+ +---+
| B | | C | A, B & C belongs to the same convergence region.
+---+ +---+
| |
+---+ +---+
| D | | E | C & D belongs to the parent convergence region.
+---+ +---+ This means B & C are the exit blocks of the region.
\ / And D & E the targets of those exits.
\ /
|
+---+
| F |
+---+
```
This pass would assign one value per exit target:
B = 0
C = 1
Then, create one variable per exit block (B, C), and assign it to the
correct value: in B, the variable will have the value 0, and in C, the
value 1.
Then, we'd create a new block H, with a PHI node to gather those 2
variables, and a switch, to route to the correct target.
Finally, the branches in B and C are updated to exit to this new block.
```
+---+
| A |
+---+
/ \
+---+ +---+
| B | | C |
+---+ +---+
\ /
+---+
| H |
+---+
/ \
+---+ +---+
| D | | E |
+---+ +---+
\ /
\ /
|
+---+
| F |
+---+
```
Note: the variable is set depending on the condition used to branch. If
B's terminator was conditional, the variable would be set using a
SELECT.
All internal edges of a region are left intact, only exiting edges are
updated.
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
This PR fixes invalid OpGroupWaitEvents emission to ensure that SPIR-V
Backend inserts a bitcast before OpGroupWaitEvents if the last argument
is a pointer that doesn't point to OpTypeEvent.
Instcombine will convert fdiv by a power-2 to fmul, this converts the
PerformVDIVCombine that converts fdiv+fcvt to fixed-point fcvt to fmul+fcvt.
The fdiv tests will look worse, but won't appear in practice (and should be
improved again by #93882).
For streaming-compatible functions with only +sme, we can't use
a NEON ORR (aliased as 'mov') for copies of Q-registers, so
we need to use a spill/fill instead.
This also fixes the fill, which should use the post-incrementing
addressing mode.
We were missing the PoisonOnly argument (so Depth + 1 was being used instead and the default Depth = 0 argument then being silently used)
Fixes#94145 and serves as the test case for 9e22c7a0ea87228dffcdfd7ab62724f72e0b3e30
This reverts commit de37c06f01772e02465ccc9f538894c76d89a7a1 to
de37c06f01772e02465ccc9f538894c76d89a7a1
It still breaks EXPENSIVE_CHECKS build. Sorry.
Port selection dag isel to new pass manager.
Only `AMDGPU` and `X86` support new pass version. `-verify-machineinstrs` in new pass manager belongs to verify instrumentation, it is enabled by default.
The getValidShiftAmountConstant/getValidMinimumShiftAmountConstant/getValidMaximumShiftAmountConstant helpers only worked with constant shift amounts, which could be problematic after type legalization (e.g. v2i64 might be partially scalarized or split into v4i32 on some targets such as 32-bit x86, Thumb2 MVE).
This patch proposes we generalize these helpers to work with ConstantRange+KnownBits if a scalar/buildvector constant isn't available.
Most restrictions are the same - the helper fails if any shift amount is out of bounds, getValidShiftConstant must be a specific constant uniform etc.
However, getValidMinimumShiftAmount/getValidMaximumShiftAmount now can return bounds values that aren't values in the actual data, as they are based off the common KnownBits of every vector element.
This addresses feedback on #92096
This adds codegen support for the "ptrauth" operand bundles, which can
be used to augment indirect calls with the equivalent of an
`@llvm.ptrauth.auth` intrinsic call on the call target (possibly
preceded by an `@llvm.ptrauth.blend` on the auth discriminator if
applicable.)
This allows the generation of combined authenticating calls
on AArch64 (in the BLRA* PAuth instructions), while avoiding
the raw just-authenticated function pointer from being
exposed to attackers.
This is done by threading a PtrAuthInfo descriptor through
the call lowering infrastructure, eventually selecting a BLRA
pseudo. The pseudo encapsulates the safe discriminator
computation, which together with the real BLRA* call get emitted
in late pseudo expansion in AsmPrinter.
Note that this also applies to the other forms of indirect calls,
notably invokes, rvmarker, and tail calls. Tail-calls in particular
bring some additional complexity, with the intersecting register
constraints of BTI and PAC discriminator computation.
However this doesn't currently support PAuth_LR tail-call variants.
This also adopts an x8+ allocation order for GPR64noip, matching
GPR64.
