SETULT can be an unsigned less than integer compare or a unordered less
than FP compare. We need to check the VT to distinguish them.
Fixes on of the issues from #128237.
Remove the restriction when reusing wider BROADCAST_LOAD nodes that both nodes couldn't have uses of their load chains - use makeEquivalentMemoryOrdering to merge the chains instead.
`N` may get merged with existing nodes inside the loop. Early exit when
it is deleted to avoid the crash.
Alternative solution: use `DAGNodeDeletedListener` to refresh the value
of N.
Closes https://github.com/llvm/llvm-project/issues/128143.
This reverts commit 34167f99668ce4d4d6a1fb88453a8d5b56d16ed5.
Different set of verifier errors appears after other regalloc failure
tests with EXPENSIVE_CHECKS.
Similar to what we already do for VPERMV3 nodes - if we're trying to create a new unary variable shuffle and we started with a VPERMV node then always create a new one if it reduces the shuffle chain depth
In some cases after reporting an allocation failure, this would fail
the verifier. It picks the first allocatable register and assigns it,
but didn't update the liveness appropriately. When VirtRegRewriter
relied on the liveness to set kill flags, it would incorrectly add
kill flags if there was another overlapping kill of the virtual
register.
We can't properly assign the register to an overlapping range, so
break the liveness of the failing register (and any other interfering
registers) instead. Give the virtual register dummy liveness by
effectively deleting all the uses by setting them to undef.
The edge case not tested here which I'm worried about is if the read
of the register is a def of a subregister. I've been unable to come up
with a test where this occurs.
https://reviews.llvm.org/D122616
There's no need for a AVX1-only 32/64-bit scalar size limit - if the X86ISD::VBROADCAST node type is supported, X86ISD::VBROADCAST_LOAD will be as well.
If we're splatting a subvector using a insert_subvector(insert_subvector(undef,sub,0),sub,c) pattern then permit multiuse of the sub as long as the insert_subvector nodes are the only users.
Despite them being more expensive than other variable mask shuffles, we
were combining shuffle chains to VPERM2W/VPERM2B if any shuffle in the
chain was a variable shuffle - including very cheap shuffles like PSHUFB
or AND mask patterns.
This patch adjusts the BWI VPERMV3 threshold - it still always permits
the merge if the chain (of 2 or more shuffles) contains any
X86ISD::VPERMV/VPERMV3 shuffles (including DQ variants), but otherwise
only reduces the depth threshold based off the number of other variable
shuffles we'd fold away.
The standard libcalls for half to float and float to half conversion are
__extendhfsf2 and __truncsfhf2. However, LLVM currently uses
__gnu_h2f_ieee and __gnu_f2h_ieee instead. As far as I can tell, these
libcalls are an ARM-ism and only provided by libgcc on that platform.
compiler-rt always provides both libcalls.
Use the standard libcalls by default, and only use the __gnu libcalls on
ARM.
Handles both BWI and non-BWI cases (skips PMOV*XBW without BWI).
The vector-interleaved-store-i8-stride-8.ll VPTERNLOG diffs are due to
better value tracking now recognizing the zero-extension patterns where
before it was any-extension
Lack of these entries caused clang to crash on the following code:
```c
__m256bh fun(__m256bh arg) {
return arg;
}
__m256bh run() {
__m256bh arg= {0};
fun(arg);
}
```
It caused the FastISel to fail since it handled the call lowering basing
on the X86CallingConv table.
Curiously, if FastISel fails somewhere down the line and
selectionDAGISel fallbacks, the crash does not occur. Following code
_does not_ crash:
```c
__m256bh fun(__m256bh arg) {
return arg;
}
__m256bh run() {
__m256bh arg= {0};
return fun(arg);
}
```
This is puzzling to me. Obviously, if FastISel fails then compiler
fallbacks to something else to lower these calls -- but since the
X86callingConv table _doesn't_ have entries for vNbf16 how does this
other thing manage not to crash? It has to use some other mechanism, one
which doesn't use the table. This rises following questions:
- how is this lowering accomplished without, presumably, using the
CallingConv entries?
- why is the table not used? I mean this points to some logic
duplication (fastISel way vs. the other bug-free way)
- How to properly test this? There is a test for vNbf16 values, but it
also must not be using the FastISel path? This duplication of logic
makes it hard to test this, since we don't have direct control whether
the FastISel path or the other one is used.
Nonetheless, this PR fixes the crash, though I didn't create a test for
it, since I am unsure yet how it should look like. I would like to learn
how the working non-FastISel mechanism works; I tried looking for it,
but didn't yet manage to find anything
Shift amounts in SelectionDAG don't have to match the result type
of the shift. SelectionDAGBuilder will aggressively truncate shift
amounts to the target's preferred type. This may result in a zero extend
that existed in IR being removed.
If we look through a truncate here, we can't guarantee the upper bits
of the truncate input are zero. There may have been a zext that was
removed. Unfortunately, this regresses tests where no truncate was
involved. The only way I can think to fix this is to add an assertzext
when SelectionDAGBuilder truncates a shift amount or remove the
early truncation of shift amounts from SelectionDAGBuilder all together.
Fixes#126889.
Concat/unpack the src subvectors together in the bottom 128-bit vector and then extend with a single EXTEND/EXTEND_VECTOR_INREG instruction
Required the getEXTEND_VECTOR_INREG helper to be tweaked to accept EXTEND_VECTOR_INREG opcodes as well to avoid us having to remap the opcode between both types.
The 512-bit filter was to prevent AVX1/2 regressions, but most of that is now handled by canonicalizeShuffleWithOp
Ideally we need to support smaller element widths as well.
Noticed while triaging #116931
We already handle the simpler VPERMV3(LO,MASK,HI) fold which can reuse
the (widened) mask, this attempts to match the flipped concatenation,
and commutes the mask to handle the flip.
I've limited this to cases where we can extract the constant mask for
commutation, a more general solution would XOR the MSB of the shuffle
mask indices to commute, but this almost never constant folds away after
lowering so the benefit was minimal.
As per title.
This is not a huge deal at the moment, but becomes one when matching
node in topological order in the DAGCombiner, and is generally more
stable than the existing.
Similar to what we already do for VPERMV3, we can fold the non-zero-idx EXTRACT_SUBVECTOR into the shuffle mask and use a free EXTRACT_SUBVECTOR(V,0) instead
lowerShuffleAsBroadcast only matches a known-splat shuffle mask, but we
can use the isShuffleEquivalent/IsElementEquivalent helpers to attempt
to find a hidden broadcast-able shuffle pattern.
This requires an extension to IsElementEquivalent to peek through
bitcasts to match against wider shuffles - these typically appear during
shuffle lowering where we've widened a preceding shuffle, often to a
vector concatenation etc.
Amazingly I hit this while yak shaving #126033 .......
This is typically handled by SimplifyDemandedVectorElts, but this will
fail when there are multiple uses of the subv_broadcast_load node, but
if there's just one use of the load result (and the rest are uses of the
memory chain), we can still replace with a load and update the chain
accordingly.
Noticed on #126517
Before this patch, the stack argument token factor was used if any
outgoing stack slots needed to be written, but not when writing the
return address to a stack slot. Writing the return address stack slot
can also alias a stack slot for an input argument.
Always use StackArgumentTokenFactor, to ensure the store of the return
address will always use it, i.e. happen after all input arguments are
loaded.
PR: https://github.com/llvm/llvm-project/pull/126244
When we first began (2015) to lower f32/f64 selects to
X86ISD::BLENDV(scalar_to_vector(),scalar_to_vector(),scalar_to_vector()),
we limited it to AVX targets to avoid issues with SSE41's xmm0
constraint for the condition mask.
Since then we've seen general improvements in TwoAddressInstruction and
better handling of condition commutation for X86ISD::BLENDV nodes, which
should address many of the original concerns of using SSE41 BLENDVPD/S.
In most cases we will replace 3 logic instruction with the BLENDV node
and (up to 3) additional moves. Although the BLENDV is often more
expensive on original SSE41 targets, this should still be an improvement
in a majority of cases.
We also have no equivalent restrictions for SSE41 for v2f64/v4f32 vector
selection.
Fixes#105807
