The latest asics support v_cvt_pk_f16_f32 instruction. However current
implementation of vector fptrunc lowering fully scalarizes the vectors,
and the scalar conversions may not always be combined to generate the
packed one.
We made v2f32 -> v2f16 legal in
https://github.com/llvm/llvm-project/pull/139956. This work is an
extension to handle wider vectors. Instead of fully scalarization, we
split the vector to packs (v2f32 -> v2f16) to ensure the packed
conversion can always been generated.
This folds away subs(csel(1, 0, cc)) from brcond, that can be produced
in certain places from compares that are not already subs (like adc/sbc
generated from i128 add_with_overflow intrinsics).
This is the GISel equivalent of scalar_to_vector, making sure that when
we insert into undef we use a fmov that avoids the artificial dependency
on the previous register. This adds v2i32 and v2i64 patterns too for
similar reasons.
Transform `select_cc t1, t2, -1, 0` for floats into a vector comparison
which generates a mask, which is later on combined with potential
vectorized DUPs.
We can always concatenate v2f64/v4f64 per-lane shuffles into a single vshufpd instruction, assuming we can profitably concatenate at least one of its operands (or its an unary shuffle).
I was really hoping to get this into combineX86ShufflesRecursively but it still can't handle concatenation/length changing as well as combineConcatVectorOps.
With EVL tail folding we can end up with vsetvlis where the output vl
and the input AVL are the same register. When we try to coalesce it we
crashed because we tried to move the def's live interval before the
kill's live interval, e.g. in this example:
(vn0 def)
dead $x0 = PseudoVSETIVLI 1, 192, implicit-def $vl, implicit-def $vtype
renamable $v9 = COPY killed renamable $v8
(vn1 def) %23:gprnox0 = PseudoVSETVLI killed (vn0) %23:gprnox0, 197,
implicit-def $vl, implicit-def $vtype
We would try to move the vn1 def VNInfo up to the previous VSETVLI, in
the middle of vn0's segment.
However separately, we were also assuming that the vl would only have
one definition and thus were just taking the VNInfo from beginIndex(),
so we ended up with a backwards segment and got the error "Cannot create
empty or backwards segment".
This fixes these two issues, the first one by moving the AVL operand +
live interval up first, and the second by taking the VNInfo from
NextMI's slot index.
Fixes#141907
push2/pop2 requires 16-byte stack alignment. If the stack alignment is
less than that, push2/pop2 should not be emitted. It triggers general
protection exception if the data being pushed/popped by push2/pop2 is
not 16-byte aligned on the stack.
performActiveLaneMaskCombine already tries to combine a single
get.active.lane.mask where the low and high halves of the result are
extracted into a single whilelo which operates on a predicate pair.
If the get.active.lane.mask node requires splitting, multiple nodes are
created with saturating adds to increment the starting index. We cannot
combine these into a single whilelo_x2 at this point unless we know
the add will not overflow.
This patch adds custom lowering for the node if the return type is
nxv32xi1, as this can be replaced with a whilelo_x2 using legal types.
Anything wider than nxv32i1 will still require splitting first.
There is a DAG combine, that folds:
```
t1: v1i1 = setcc x:v1f16, y:v1f16, setogt:ch
t2: v1i64 = zero_extend t1
```
->
```
t1: v1i16 = setcc x:v1f16, y:v1f16, setogt:ch
t2: v1i64 = any_extend t1
```
This creates an issue on AArch64 when attempting to widen the result to
`v4i16`. The operand types (`v1f16`) are set to be scalarized, so the
"by hand" widening with `DAG.WidenVector` is used for them, however,
this only widens to the next power-of-2, so returns `v2f16`, which does
not match the result VF. The fix is to manually construct the widened
inputs using `INSERT_SUBVECTOR`.
Fixes#136540
If the lower 32-bits of a i64 value are known to be zero, then icmp
lowering will shift+truncate down to a i32 allowing the immediate to be
embedded.
There's a lot more that could be done here to match icmp lowering, but
this PR just focuses on known regressions.
Fixes#142513Fixes#62145
This issue starts in the selection DAG and causes the backend to emit
the following for a trivial tail call:
```
ldr w8, [sp]
str w8, [sp]
b func
```
I'm not too sure that checking for immutability of a specific stack
object is a good enough of a gurantee, because as soon a tail-call is
done lowering,`setHasTailCall()` is called and in that case perhaps a
pass is allowed to change the value of the object in-memory?
This can be extended to the ARM backend as well.
Removed the `tailcall` keyword from a few other test assets, I'm
assuming their original intent was left intact.
When -triple is windows, SelectionDAGLegalize process Legalizing br_jt,
would generate ISD::JUMP_TABLE_DEBUG_INFO.
Then Mips process emitInstruction, would think JUMP_TABLE_DEBUG_INFO is
a pseudo instruction and generate an error `Pseudo opcode found in
emitInstruction()`.
This instruction `TargetOpcode::JUMP_TABLE_DEBUG_INFO` is only used to
note jump table debug info, so we can ignore it when Mips emit
instruction.
Fix#134916.
The XAndesPerf extension includes unsigned bitfield extraction
instruction `NDS.BFOZ`, which can extract the bits from 0 to Len -1,
place them starting at bit Msb, and zero-fills the remaining bits.
This patch handles the cases where Msb < Lsb for `NDS.BFOZ`.
Instruction Sytax:
nds.bfoz Rd, Rs1, Msb, Lsb
The operation is:
if Msb < Lsb:
Lenm1 = Lsb - Msb;
Rd[Lsb:Msb] = Rs1[Lenm1:0];
if (Lsb < (XLen -1)) Rd[XLen-1:Lsb+1]=0;
Rd[Msb-1:0]=0;
When Len == 1, it is a special case where the Msb is set to 0 instead of
being equal to the Lsb.
This PR addresses a specific edge case when deciding whether or not to
produce a bitcast instruction.
Specifically, when the given instruction is a global array, the element
type of the array wasn't correctly compared to the return type. In this
specific case, if the types are equal, a bitcast shouldn't be created,
but it was.
This PR checks to see if the element type of the array is the same as
the return type, and if it is, it doesn't create a bitcast instruction.
Fixes https://github.com/llvm/llvm-project/issues/139013
This patch adds a DAG combine rule for BITCAST nodes converting from
vector `i1` masks generated by `setcc` into integer vector types. It
recognizes common select mask patterns and lowers them into efficient
LoongArch LSX/LASX mask instructions such as:
- [X]VMSKLTZ.{B,H,W,D}
- [X]VMSKGEZ.B
- [X]VMSKNEZ.B
When the vector comparison matches specific patterns (e.g., x < 0, x >=
0, x != 0, etc.), the transformation is performed pre-legalization. This
avoids scalarization and unnecessary operations, improving both
performance and code size.
We have multiple different attributes in clang representing device
kernels for specific targets/languages. Refactor them into one attribute
with different spellings to make it more easily scalable for new
languages/targets.
---------
Signed-off-by: Sarnie, Nick <nick.sarnie@intel.com>
Solely relying on top‑down scheduling can underutilize hardware, since
long‑latency instructions often end up scheduled too late and their
latency isn’t well hidden. Adding bottom‑up post‑RA scheduling lets us
move those instructions earlier, which improves latency hiding and
yields roughly a 2% performance gain on key benchmarks.
Summary:
The patch fixes the issue [[PowerPC] missing VSX FMA Mutation optimize
in some case for option -schedule-ppc-vsx-fma-mutation-early
#111906](https://github.com/llvm/llvm-project/issues/111906)
In certain cases, the Register Coalescer pass—which eliminates COPY
instructions—can interfere with the PowerPC VSX FMA Mutation pass.
Specifically, it can prevent the mutation of a COPY adjacent to an
XSMADDADP into a single XSMADDMDP instruction. As a result, the xxspltiw
instruction is not hoisted out of the loop as expected, leading to
missed optimization opportunities.
To address this, the patch ensures that the `VSX FMA Mutation` pass runs
before the `Register Coalescer` pass when the
-schedule-ppc-vsx-fma-mutation-early option is enabled.
In MachinePipeliner, loop-carried memory dependencies are represented by
DAG, which makes things complicated and causes some necessary
dependencies to be missing. This patch introduces a new class to manage
loop-carried memory dependencies to simplify the logic. The ultimate
goal is to add currently missing dependencies, but this is a first step
of that, and this patch doesn't intend to change current behavior. This
patch also adds new tests that show the missed dependencies, which
should be fixed in the future.
Split off from #135148
The existing way of managing clustered nodes was done through adding
weak edges between the neighbouring cluster nodes, which is a sort of
ordered queue. And this will be later recorded as `NextClusterPred` or
`NextClusterSucc` in `ScheduleDAGMI`.
But actually the instruction may be picked not in the exact order of the
queue. For example, we have a queue of cluster nodes A B C. But during
scheduling, node B might be picked first, then it will be very likely
that we only cluster B and C for Top-Down scheduling (leaving A alone).
Another issue is:
```
if (!ReorderWhileClustering && SUa->NodeNum > SUb->NodeNum)
std::swap(SUa, SUb);
if (!DAG->addEdge(SUb, SDep(SUa, SDep::Cluster)))
```
may break the cluster queue.
For example, we want to cluster nodes (order as in `MemOpRecords`): 1 3
2. 1(SUa) will be pred of 3(SUb) normally. But when it comes to (3, 2),
As 3(SUa) > 2(SUb), we would reorder the two nodes, which makes 2 be
pred of 3. This makes both 1 and 2 become preds of 3, but there is no
edge between 1 and 2. Thus we get a broken cluster chain.
To fix both issues, we introduce an unordered set in the change. This
could help improve clustering in some hard case.
One key reason the change causes so many test check changes is: As the
cluster candidates are not ordered now, the candidates might be picked
in different order from before.
The most affected targets are: AMDGPU, AArch64, RISCV.
For RISCV, it seems to me most are just minor instruction reorder, don't
see obvious regression.
For AArch64, there were some combining of ldr into ldp being affected.
With two cases being regressed and two being improved. This has more
deeper reason that machine scheduler cannot cluster them well both
before and after the change, and the load combine algorithm later is
also not smart enough.
For AMDGPU, some cases have more v_dual instructions used while some are
regressed. It seems less critical. Seems like test `v_vselect_v32bf16`
gets more buffer_load being claused.
This is a copy of the fneg.f16.ll, just with type replaced.
The final logic shall be the same as with f16 as these are
just bit operations.
---------
Co-authored-by: Matt Arsenault <Matthew.Arsenault@amd.com>
`freeze` on `fmul` (without `nnan`) followed by `fadd` or `fsub` into a
single `fma` is supported.
This patch adds lit tests to verify the optimization behavior for both
nnan and non-nnan variants.
