We already had getOperandInfo support, so this marks the instructions as
supported in isCandidate. It also adds support for vfwmaccbf16.v{v,f}
from zvfbfwma
When we first began (2015) to lower f32/f64 selects to
X86ISD::BLENDV(scalar_to_vector(),scalar_to_vector(),scalar_to_vector()),
we limited it to AVX targets to avoid issues with SSE41's xmm0
constraint for the condition mask.
Since then we've seen general improvements in TwoAddressInstruction and
better handling of condition commutation for X86ISD::BLENDV nodes, which
should address many of the original concerns of using SSE41 BLENDVPD/S.
In most cases we will replace 3 logic instruction with the BLENDV node
and (up to 3) additional moves. Although the BLENDV is often more
expensive on original SSE41 targets, this should still be an improvement
in a majority of cases.
We also have no equivalent restrictions for SSE41 for v2f64/v4f32 vector
selection.
Fixes#105807
Move AArch64 non auto-generated test code into a static file, since the
script `./llvm/test/CodeGen/AArch64/Atomics/generate-tests.py` will overwrite
these tests when re-run. (Test code was originally added in change
465bc5e729fd755880b9a288de42a37ad1206301)
Somtimes when we're breaking up a large vector copy into several smaller
ones, not every single smaller source registers are initialized at the
time when the original COPY happens, and the verifier will not be
pleased when seeing the smaller copies reading from an undef register.
This patch is a workaround for the said issue by attaching an implicit
read of the source operand on the newly generated copies.
This is tested by llvm/test/CodeGen/RISCV/rvv/vector-interleave.ll which
would have crashed the compiler without this fix when
LLVM_EXPENSIVE_CHECK is enabled. Original context:
https://github.com/llvm/llvm-project/pull/124825#issuecomment-2639097531
---------
Co-authored-by: Craig Topper <craig.topper@sifive.com>
This patch adds intrinsics for tcgen05 wait,
fence and commit PTX instructions.
lit tests are added and verified with a
ptxas-12.8 executable.
Docs are updated in the NVPTXUsage.rst file.
Signed-off-by: Durgadoss R <durgadossr@nvidia.com>
Given a zext from an extract vector, with a shuffle mask like <4, 0, 0, 4> we
would previously recognize the top half as a deinterleave. In order to convert
into a uzp we should have been checking that the bottom half is also poison.
Fixes#125989
When a def in a block A reaches another block B that is in A's iterated
dominance frontier, a phi node is added to B for the def register.
A clobbering def can be created at a call instruction, for a register
clobbered by a call.
However, phi nodes are not created for a register, when one of the
reaching defs of the register is a clobbering def.
This patch adds phi nodes for registers that have a clobbering reaching
def. These additional phis help in checking reaching defs for an
instruction in RDF based copy propagation and addressing mode
optimizations.
The type inference for `spv_resource_getpointer` must look at the uses
of the intrinsic. The type of the handle will be a scalar type, but it
can still be read or written as a vector of that type.
Fixes https://github.com/llvm/llvm-project/issues/124551.
As added in #124274, CPUs in this range can suffer from performance
issues with ldapur. As the gain from ldar->ldapr is expected to be
greater than the minor gain from ldapr->ldapur, this opts to avoid the
instruction under the default -mcpu=generic when the -march is less that
armv8.8 / armv9.3.
I renamed AArch64Subtarget::Others to AArch64Subtarget::Generic to be
clearer what it means.
This adds the `llvm.modf` intrinsic, legalization, and lowering (mostly
reusing the lowering for sincos and frexp).
The `llvm.modf` intrinsic takes a floating-point value and returns both
the integral and fractional parts (as a struct).
```
declare { float, float } @llvm.modf.f32(float %Val)
declare { double, double } @llvm.modf.f64(double %Val)
declare { x86_fp80, x86_fp80 } @llvm.modf.f80(x86_fp80 %Val)
declare { fp128, fp128 } @llvm.modf.f128(fp128 %Val)
declare { ppc_fp128, ppc_fp128 } @llvm.modf.ppcf128(ppc_fp128 %Val)
declare { <4 x float>, <4 x float> } @llvm.modf.v4f32(<4 x float> %Val)
```
This corresponds to the libm `modf` function but returns multiple values
in a struct (rather than take output pointers), which makes it easier to
vectorize.
With this patch, we are enabling comparisons for Long double (80) types
on X87 stack. It lowers G_FCMP.
---------
Co-authored-by: mattarde <mattarde@intel.com>
This was ultimately working around bugs in subregister handling
in peephole-opt. In the common case, it would give up on folding
anything into a subregister extract copy.
This is a tiny change that can save up to 16 bytes of stack allocation,
which is more beneficial on RV32 than RV64.
cm.push allocates multiples of 16 bytes, but only uses a subset of those
bytes for pushing callee-saved registers. Up to 12 (rv32) or 8 (rv64)
bytes are left unused, depending on how many registers are pushed.
Before this change, we told LLVM that the entire allocation was used, by
creating a fixed stack object which covered the whole allocation.
This change instead gives an accurate extent to the fixed stack object,
to only cover the registers that have been pushed. This allows the
PrologEpilogInserter to use any unused bytes for spills. Potentially
this saves an extra move of the stack pointer after the push, because
the push can allocate up to 48 more bytes than it needs for registers.
We cannot do the same change for save/restore, because the restore
routines restore in batches of `stackalign/(xlen/8)` registers, and we
don't want to clobber the saved values of registers that we didn't tell
the compiler we were saving/restoring - for instance `__riscv_restore_0`
is used by the compiler when it only wants to save `ra`, but will end up
restoring `ra` and `s0`.
These tests are for frame handling code with push/pop. To increase
coverage of CFI/Unwind info, this removes the `nounwind` annotations and
regenerates the checks for this test.
With the test changes.
Original message:
The Zicond version of this requires an li instruction and an
additional register.
Without Zicond we match this in a DAGCombine on RISCVISD::SELECT_CC.
This PR has 2 commits. I'll pre-commit the test change if this looks
good.
Remove the restriction that scheduling rematerialization candidates
cannot have virtual reg uses.
Currently, this only allows for virtual reg uses which are already live
at the rematerialization point, so bring in allUsesAvailableAt to check
for this condition. Because of this condition, the uses of the remats
will already be live in to the region, so the remat won't increase
live-in pressure.
Add an expensive check to check this condition.
Ported hasAllNBitUsers to RISCV GISel side. Add GISelPredicate code to
each of the 16,32, and 64 bit words. It allows for generation of
optimized packw sequences along with other transparent narrowing
operations. Included a few new .ll files to expand testing and limited
the OptW pass Optimization to fewer options until GISel is ready for
more code generation paths
---------
Signed-off-by: Luke Quinn <quic_lquinn@quicinc.com>
Fix the case where the vector element type of the loaded extractelement
input does not match the result type of the extract.
This fixes a regression reported after
c55a7659b38946350315ac4a18d9805deb1f0a54
The setmaxnreg PTX instruction is supported on all arch-conditionals,
known up-to cuda-12.8, from sm90 onwards. This patch
updates the predicate checks to handle this. The feature is additionally
tested in setmaxnreg-sm100a.ll
If we're lowering a shuffle to a vrgather (or vcompress), and we know
that a prefix of the operation can be done while producing the same
(defined) lanes, do the operation with a narrower LMUL.
Currently, each expansion of `FILL_PPR_FROM_ZPR_SLOT_PSEUDO` creates a
new ptrue instruction. This patch adds a simple method to reuse a
previous ptrue instruction when expanding back-to-back fill pseudos.
This patch improves SVE codegen for the vector extract last active
intrinsic, using either the lastb instruction (if the passthru value was
poison or undef), or the clastb instruction.
@Artem-B, I believe you said we don't intend to support `sm_1x`.
Assuming that's correct, this PR will remove all remaining support I
could find for `sm_1x`.
Kernel Control Flow Integrity (kCFI) is a feature that hardens indirect
calls by comparing a 32-bit hash of the function pointer's type against
a hash of the target function's type. If the hashes do not match, the
kernel may panic (or log the hash check failure, depending on the
kernel's configuration). These hashes are computed at compile time by
applying the xxHash64 algorithm to each mangled canonical function (or
function pointer) type, then truncating the result to 32 bits. This hash
is written into each indirect-callable function header by encoding it as
the 32-bit immediate operand to a `MOVri` instruction, e.g.:
```
__cfi_foo:
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
nop
movl $199571451, %eax # hash of foo's type = 0xBE537FB
foo:
...
```
This PR extends x86-based kCFI with a 3-bit arity indicator encoded in
the `MOVri` instruction's register (reg) field as follows:
| Arity Indicator | Description | Encoding in reg field |
| --------------- | --------------- | --------------- |
| 0 | 0 parameters | EAX |
| 1 | 1 parameter in RDI | ECX |
| 2 | 2 parameters in RDI and RSI | EDX |
| 3 | 3 parameters in RDI, RSI, and RDX | EBX |
| 4 | 4 parameters in RDI, RSI, RDX, and RCX | ESP |
| 5 | 5 parameters in RDI, RSI, RDX, RCX, and R8 | EBP |
| 6 | 6 parameters in RDI, RSI, RDX, RCX, R8, and R9 | ESI |
| 7 | At least one parameter may be passed on the stack | EDI |
For example, if `foo` takes 3 register arguments and no stack arguments
then the `MOVri` instruction in its kCFI header would instead be written
as:
```
movl $199571451, %ebx # hash of foo's type = 0xBE537FB
```
This PR will benefit other CFI approaches that build on kCFI, such as
FineIBT. For example, this proposed enhancement to FineIBT must be able
to infer (at kernel init time) which registers are live at an indirect
call target: https://lkml.org/lkml/2024/9/27/982. If the arity bits are
available in the kCFI function header, then this information is trivial
to infer.
Note that there is another existing PR proposal that includes the 3-bit
arity within the existing 32-bit immediate field, which introduces
different security properties:
https://github.com/llvm/llvm-project/pull/117121.
This enables DAG combines to form this mask. Reverse is generally linear
in LMUL so this is reasonable, and results in better codegen for the 2
source variants.
For <= m1, the change is only slightly profitable if at all. We trade
some mask creation and an extract vrsub for a vslideup.vi. This is
likely roughly neutral. At >= m2, this is distinctly profitable as
generic DAG pushes the reverse into the two operands. We effectively
already did this for one operand, but the other was hitting a full
O(LMUL^2) shuffle. Moving that to be O(LMUL/2) operation is a big win.
In some situations the first operand to an instruction might not be a register
(for example with intrinsics). We are only interested in extend operations, so
make sure the instruction is one we expect before we attempt to access the
first reg.
