This PR reworks implementation of OpSpecConstantOp with ptr-cast
operation (PtrCastToGeneric, GenericCastToPtr). Previous implementation
didn't take into account a lot of use cases, including multiple
inclusion of pointers, reference to a pointer from OpName, etc. A
reproducer is attached as a new test case.
This PR also fixes wrong type inference for IR patterns which generate
new virtual registers without SPIRV type. Previous implementation
assumed always that result has the same address space as a source that
is not the fact, and, for example, led to impossibility to emit a
ptr-cast operation in the reproducer, because wrong type inference
rendered source and destination with the same address space, eliminating
translation of G_ADDRSPACE_CAST.
This PR improves type inference and fixes inconsistency between
previously deduced element type of a pointer and function's return type.
It fixes https://github.com/llvm/llvm-project/issues/109401 by ensuring
that OpPhi is consistent with respect to operand types.
For register constraints that require specific register ranges, the
width of the range should match the type of the associated
parameter/return value. With this PR, we error out when that is not the
case. Previously, these cases would hit assertions or llvm_unreachables.
The handling of register constraints that require only a single register
remains more lenient to allow narrower non-vector types for the
associated IR values. For example, constraining an i16 or i8 value to a
32-bit register is still allowed.
Fixes#101190.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
After #110144, we can finish off #110129 and fold f16 vfmv.f.s into a
flh.
vfmv.f.s is only available for f16 with zvfh, which in turn requires
zfhmin so we can use flh.
bf16 has no vfmv.f.s so the extract_vector_elt is lowered as an integer
in #110144, and gets the existing integer vmv.x.s fold.
```
define internal void @foo() {
ret void
}
define i64 @main() {
ret i64 add (i64 ptrtoint (ptr @foo to i64), i64 -2147483626)
}
```
When `foo` is a local symbol, `foo` and `main` are in the same section,
and `offset` is near INT32_MIN, referencing `foo+offset` in `main` with
RIP-relative addressing needs `leaq .text+offset1(%rip), %rax` where
`offset1 < offset`, and `offset1` might underflow.
(https://discourse.llvm.org/t/arithmetic-referencing-dso-local-function-causes-compilation-error-on-linux-x64/80033):
Don't use RIP-relative addressing if the negative offset is near
INT32_MIN. Arbitrarily reuse the magic number in isOffsetSuitableForCodeModel to
guard against the edge case when `address(current_instruction)-foo < 4GiB-16MiB`.
If the difference is larger than 4GiB-16MiB, `ret i64 add (i64 ptrtoint
(ptr @foo to i64), i64 -2**32+256MiB)` would still cause the assembly
issue, such cases are unrealistic.
Pull Request: https://github.com/llvm/llvm-project/pull/98438
Currently, many attempts to lower loads and stores on buffer fat
pointers lower directly to intrinsic calls that will be unsupported by
or crash codegen (ex, storing a [2 x i32], a <6 x half>, or an i160).
Record the current behavior to make the effects of the fix more visible
in an upcoming PR.
Alongside something like:
vpternlogq zmm0, zmm2, zmm1, 64
We will now have a comment on the right like:
# zmm0 = zmm0 & zmm2 & ~zmm1
This makes it easy to tell at a glance what sort of truth table the
instruction will provide.
If we know vlen is a multiple of 16, we don't need any alignment
padding.
I wrote the code so that it would generate the minimum amount of padding
if the stack align was 32 or larger or if RVVBitsPerBlock was smaller
than half the stack alignment.
This fixes issue #109250
The issue happens during the `MachineBlockPlacement` pass. The block,
whose address was previously not taken, is deemed redundant by the pass
and subsequently replaced using
`MachineBasicBlock::ReplaceUsesOfBlockWith` in `BranchFolding`.
ReplaceUsesOfBlockWith only replaces uses in the terminator. However,
`expandPostRAPseudo` introduces new block uses when expanding catchrets.
These uses do not get replaced, which results in undefined label errors
later on.
Marking the block addresss as taken prevents the replacement of the
block, without also replacing non-terminator uses.
Allocating wwm-registers and per-thread VGPR operands
together imposes many challenges in the way the
registers are reused during allocation. There are
times when regalloc reuses the registers of regular
VGPRs operations for wwm-operations in a small range
leading to unwantedly clobbering their inactive lanes
causing correctness issues that are hard to trace.
This patch splits the VGPR allocation pipeline further
to allocate wwm-registers first and the regular VGPR
operands in a separate pipeline. The splitting would
ensure that the physical registers used for wwm
allocations won't take part in the next allocation
pipeline to avoid any such clobbering.
This patch was reverted because of a failing C test.
It now has being solved and can be merged into main again
This patch adds these intrinsics:
// Variants are also available for: _s8
svuint8x4_t svluti4_zt_u8_x4(uint64_t zt0, svuint8x2_t zn)
__arm_streaming __arm_in("zt0");
according to PR#324[1]
[1]ARM-software/acle#324
Converts AMDGPUResourceUsageAnalysis pass from Module to MachineFunction
pass. Moves function resource info propagation to to MC layer (through
helpers in AMDGPUMCResourceInfo) by generating MCExprs for every
function resource which the emitters have been prepped for.
Fixes https://github.com/llvm/llvm-project/issues/64863
We do not expect to see live carry out outputs on these adds,
so add a dead flag. Split the test for the degenerate case. This
makes it more apparent a regression in a future commit does not matter.
…me lowering
SME instructions can only be used in streaming mode. PTRUE for
predicated counter and the ld/st pair can be used when:
sve2.1 is available or
sme2 available in function in streaming mode.
Previously the frame lowering only checking if sme2 available when
building the machine instruction.
This fix checks if sme2 is available and is subtarget in streaming mode
Windows have different separators for paths than Unix based OS. One of
the tests in debug-compilation-unit.ll didn't have Win supported '\\'
variant which broken test suite on that OS.
Make print_btf.py correctly detect endianness of the BTF input. Input
endianness is inferred from BTF magic word [2], which is a 2-byte
integer at offset 0 of the input:
- sequence `EB 9F` signals big-endian input;
- sequence `9F EB` signals little-endian input.
Before this commit the magic sequence was read using "H" format for
`unpack` method of python's `struct` module:
- if magic is `0xEB9F` assume little-endian;
- if magic is `0x9FEB` assume big-endian.
However, format `H` reads data in native endianness.
Thus the above logic would only be correct on little endian hosts:
- byte sequence `9F EB` read as `0xEB9F` -> little-endian input;
- byte sequence `EB 9F` read as `0x9FEB` -> big-endian input.
On the big-endian host the relation should be inverse.
Fix this by always reading magic in big-endian (format `>H`).
This fixes CI error reported for a BPF test using print_btf.py script in
[1]. The error happens on a s390 host, which is big-endian.
[1] https://lab.llvm.org/buildbot/#/builders/42/builds/1192
[2]
https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-type-and-string-encoding
Remove the following intrinsics which can be trivially replaced with an
`addrspacecast`
* llvm.nvvm.ptr.gen.to.global
* llvm.nvvm.ptr.gen.to.shared
* llvm.nvvm.ptr.gen.to.constant
* llvm.nvvm.ptr.gen.to.local
* llvm.nvvm.ptr.global.to.gen
* llvm.nvvm.ptr.shared.to.gen
* llvm.nvvm.ptr.constant.to.gen
* llvm.nvvm.ptr.local.to.gen
Also, cleanup the NVPTX lowering of `addrspacecast` making it more
concise.
This was reverted to avoid conflicts while reverting #107655. Re-landing
unchanged.
Previously we used an empty MachinePointerInfo. I checked a few other
targets like X86, ARM, and AArch64 and they all appear to use correct
MachinePointerInfo.
This change deprecates the following intrinsics which can be trivially
converted to llvm funnel-shift intrinsics:
- @llvm.nvvm.rotate.b32
- @llvm.nvvm.rotate.right.b64
- @llvm.nvvm.rotate.b64
This fixes a bug in the previous version (#107655) which flipped the
order of the operands to the PTX funnel shift instruction. In LLVM IR
the high bits are the first arg and the low bits are the second arg,
while in PTX this is reversed.
KnownBits::srem does not correctly set the leader zero-bits, omitting
the fact that LHS may be known-negative or known-non-negative. Fix this.
Alive2 proof: https://alive2.llvm.org/ce/z/Ugh-Dq
We already disallow accessing the callee's allocas from a tail-called
function, because their stack memory will have been de-allocated before
the tail call. I think this should apply to byval arguments too, as they
also occupy space in the caller's stack frame.
This was originally part of #109943, spilt out for separate review.
This changes the existing promote logic to lower, so that it can use
normal integer operations. A minor change was needed to fneg lower code
to handle vectors.
This patches adds a 16 bit register class for use with Zhinx
instructions. This makes them more similar to Zfh instructions and
allows us to only spill 16 bits.
I've added CodeGenOnly instructions for load/store using GPRF16 as that
gave better results than insert_subreg/extract_subreg. I'm using FSGNJ
for GPRF16 copy with Zhinx as that gave better results. Zhinxmin will
use ADDI+subreg operations.
Function arguments use this new GPRF16 register class for f16 arguments
with Zhinxmin. Eliminating the need to use RISCVISD::FMV* nodes.
I plan to extend this idea to Zfinx next.
Support for large code model are added recently, and sementically
direct calls are lowered to an indirect branch with a constant pool target.
By default it does not use the x7 register and this is suboptimal with
Zicfilp because it introduces landing pad check, which is unnecessary
since the constant pool is read-only and unlikely to be tampered.
Change direct calls and tail calls to use x7 as the scratch
register (a.k.a. software guarded branch in the CFI spec)
This is the f64/f32 version of #109774.
I've left out f16 and bf16 for now because there's a separate issue
where we can't select extract_vector_elt when f16/bf16 is a legal type,
see #110126.
When the scalar type is illegal, it gets softened during type
legalization and gets lowered as an integer.
However with zfhmin/zfbfmin the type is now legal and it passes through
type legalization where it crashes because we didn't have any custom
lowering or patterns for it.
This handles said case via the existing custom lowering to a vslidedown
and vfmv.f.s.
It also handles the case where we only have zvfhmin/zvfbfmin and don't
have vfmv.f.s, in which case we need to extract it to a GPR and then use
fmv.h.x.
Fixes#110126
This change adds a pass to scalarize vectors in global scope into
arrays.
There are three distinct parts
1. find the globals that need to be updated and define what the new type
should be
2. initialize that new type and copy over all the right attributes over
from the old type.
3. Use the instruction visitor pattern to update the loads, stores, and
geps for the layout of the new data structure.
resolves https://github.com/llvm/llvm-project/issues/107920
Two main goals of this PR are:
* to support "Arithmetic with Overflow" intrinsics, including the
special case when those intrinsics are being generated by the
CodeGenPrepare pass during translations with optimization;
* to redirect intrinsics with aggregate return type to be lowered via
GlobalISel operations instead of SPIRV-specific unfolding/lowering (see
https://github.com/llvm/llvm-project/pull/95012).
There is a new test case
`llvm/test/CodeGen/SPIRV/passes/translate-aggregate-uaddo.ll` that
describes and checks the general logics of the translation.
This PR continues a series of PRs aimed to identify and fix flaws in
code emission, to improve pass rates for the mode with expensive checks
set on (see https://github.com/llvm/llvm-project/pull/101732,
https://github.com/llvm/llvm-project/pull/104104,
https://github.com/llvm/llvm-project/pull/106966), having in mind the
ultimate goal of proceeding towards the non-experimental status of
SPIR-V Backend.
The reproducers are:
1) consider `llc -O3 -mtriple=spirv64-unknown-unknown ...` with:
```
define spir_func i32 @foo(i32 %a, ptr addrspace(4) %p) {
entry:
br label %l1
l1:
%e = phi i32 [ %a, %entry ], [ %i, %body ]
%i = add nsw i32 %e, 1
%fl = icmp eq i32 %i, 0
br i1 %fl, label %exit, label %body
body:
store i8 42, ptr addrspace(4) %p
br label %l1
exit:
ret i32 %i
}
```
2) consider `llc -O0 -mtriple=spirv64-unknown-unknown ...` with:
```
define spir_func i32 @foo(i32 %a, ptr addrspace(4) %p) {
entry:
br label %l1
l1: ; preds = %body, %entry
%e = phi i32 [ %a, %entry ], [ %math, %body ]
%0 = call { i32, i1 } @llvm.uadd.with.overflow.i32(i32 %e, i32 1)
%math = extractvalue { i32, i1 } %0, 0
%ov = extractvalue { i32, i1 } %0, 1
br i1 %ov, label %exit, label %body
body: ; preds = %l1
store i8 42, ptr addrspace(4) %p, align 1
br label %l1
exit: ; preds = %l1
ret i32 %math
}
```
