Previously they were legal by default, so the truncstore/extload test
cases would get combined and crash during selection.
These are set to expand for f16 so do the same for bf16.
This adds VL patterns for vfwmaccbf16.vv so that we can handle fixed
length vectors.
It does this by teaching combineOp_VLToVWOp_VL to emit
RISCVISD::VFWMADD_VL for bf16. The change in getOrCreateExtendedOp is
needed because getNarrowType is based off of the bitwidth so returns
f16. We need to explicitly check for bf16.
Note that the .vf patterns don't work yet, since the build_vector splat
gets lowered to a (vmv_v_x_vl (fmv_x_anyexth x)) instead of a vfmv.v.f,
which SplatFP doesn't pick up, see #106637.
On newer GPUs, where `cvta.param` instruction is available we can avoid
making byval arguments when their pointers are used in a few more cases,
even when `__grid_constant__` is not specified.
- phi
- select
- memcpy from the parameter.
Switched pointer traversal from a DIY implementation to PtrUseVisitor.
GNU ld will error when encountering a pcrel_lo whose corresponding
pcrel_hi is in a different section. [1] introduced a check to help
prevent this issue by preventing outlining in a few circumstances.
However, we can also hit this same issue when outlining from functions
with prefixes ("hot"/"unlikely"/"unknown" from profile information, for
example) as the outlined function might not have the same prefix,
possibly resulting in a "paired" pcrel_lo and pcrel_hi ending up in
different sections.
To prevent this issue, take a similar approach as [1] and additionally
prevent outlining when we see a pcrel_lo and the function has a prefix.
[1]
96c85f80f0Fixes#107520
Always generate v_cndmask_b32 instead of modifying exec around
v_mov_b32. This is expected to be faster because
modifying exec generally causes pipeline stalls.
This is a large patch includes the MC level support for V_CVT_F16_F32,
V_CVT_F32_F16 and V_LDEXP_F16 in true16 format.
This patch includes the asm/disasm changes to encode/decode the 16bit
vsrc, vdst and src modifieres for vop and dpp format. This patch is a
dependency for many 16 bit instructions while only three instructions
are updated to make it easier to review.
There will be another patch to support these three instructions in the
codeGen level, this patch just replaces these two instructions with its
fake16 format.
This is a three deep expression which is deeper than we've otherwise
gone for multiple expansions, but I think it's reasonable to do so. This
covers mul by 50, 100, and 200 which are reasonably common naturally
arising numbers.
Add tests for "sub(select(icmp(a,b),a,b),select(icmp(a,b),b,a)) -> abd(a,b)" patterns that still fail to match to abd nodes
This will hopefully be helped by #108218
trunc (binop X, C) --> binop (trunc X, trunc C) --> binop (trunc X, C`)
Try to narrow the width of math or bitwise logic instructions by pulling
a truncate ahead of binary operators.
Vx and Nx cores consider 32-bit and 64-bit basic arithmetic equal in
costs.
combineSubABS already handles the "(sub Y, cmovns X, -X) -> (add Y, cmovns -X, X)" fold by flipping the cmov operands.
We can do something similar for the negation of ABDS/U patterns which have been expanded to a CMOVL/CMOVB with a pair of commuted subtractions: "NEG(ABD(X,Y)) -> NEG(CMOV(SUB(X,Y),SUB(Y,X))) -> CMOV(SUB(Y,X),SUB(X,Y))"
When running SPIR-V Backend with optimization levels higher than 0, we
observe GEP Operator's as a new factor, massively used to convey the
semantics of the original LLVM IR. Previously, an issue related to GEP
Operator was mentioned and fixed on the consumer side of toolchains
(see, for example, Khronos Trandslator Issue
https://github.com/KhronosGroup/SPIRV-LLVM-Translator/issues/2486 and PR
https://github.com/KhronosGroup/SPIRV-LLVM-Translator/pull/2487).
However, there is a case when GenCode creates G_PTR_ADD to convey the
original semantics under optimization levels higher than 0 where it's
SPIR-V Backend that fails to translate source LLVM IR correctly.
Consider the following reproducer:
```
%struct = type { i32, [257 x i8], [257 x i8], [129 x i8], i32, i64, i64, i64, i64, i64, i64 }
@Mem = linkonce_odr dso_local addrspace(1) global %struct zeroinitializer, align 8
define weak dso_local spir_func void @__devicelib_assert_fail(ptr addrspace(4) noundef %expr, i32 noundef %line, i1 %fl) {
entry:
%cmp = icmp eq i32 %line, 0
br i1 %cmp, label %lbl, label %exit
lbl:
store i32 %line, ptr addrspace(1) getelementptr inbounds (i8, ptr addrspace(1) @Mem, i64 648), align 8
br i1 %fl, label %lbl, label %exit
exit:
ret void
}
```
converted to the following machine instructions by SPIR-V Backend:
```
%4:type(s64) = OpTypeInt 32, 0
%22:type(s64) = OpTypePointer 5, %4:type(s64)
%2:type(s64) = OpTypeInt 8, 0
%28:type(s64) = OpTypePointer 5, %2:type(s64)
%10:pid(p1) = G_GLOBAL_VALUE @Mem
%36:type(s64) = OpTypeStruct %4:type(s64), %32:type(s64), %32:type(s64), %34:type(s64), %4:type(s64), %35:type(s64), %35:type(s64), %35:type(s64), %35:type(s64), %35:type(s64), %35:type(s64)
%37:iid(s32) = G_INTRINSIC_W_SIDE_EFFECTS intrinsic(@llvm.spv.const.composite)
%8:iid(s32) = ASSIGN_TYPE %37:iid(s32), %36:type(s64)
G_INTRINSIC_W_SIDE_EFFECTS intrinsic(@llvm.spv.init.global), %10:pid(p1), %8:iid(s32)
%29:pid(p1) = nuw G_PTR_ADD %10:pid, %16:iid(s64)
%15:pid(p1) = nuw ASSIGN_TYPE %29:pid(p1), %28:type(s64)
%27:pid(p2) = G_BITCAST %15:pid(p1)
%17:pid(p2) = ASSIGN_TYPE %27:pid(p2), %22:type(s64)
G_STORE %1:iid(s32), %17:pid(p2) :: (store (s32) into %ir.3, align 8, addrspace 1)
```
On the next stage of instruction selection this `G_PTR_ADD`-related
pattern would be interpreted as an initialization of a global variable
and converted to an invalid constant GEP pattern that, in its turn,
would fail to be verified by LLVM during back translation from SPIR-V to
LLVM IR.
This PR introduces a fix for the problem by adding one more case of
`G_PTR_ADD` translation, when we use a non-const GEP to convey the
meaning. The reproducer is attached as a new test case.
The InitUndef pass currently uses the getLargestSuperClass() hook (which
is only used by that pass) to chose the register to initialize. This was done
to reduce the number of undef init pseudos needed, e.g. so that the vrnov0
regclass would use the same pseudo as v0. After #106744 we use a single
generic pseudo, so this is no longer necessary.
This adds the basic assembly generation support for the final EH
proposal, which was newly adopted in Sep 2023 and advanced into Phase 4
in Jul 2024:
https://github.com/WebAssembly/exception-handling/blob/main/proposals/exception-handling/Exceptions.md
This adds support for the generation of new `try_table` and `throw_ref`
instruction in .s asesmbly format. This does NOT yet include
- Block annotation comment generation for .s format
- .o object file generation
- .s assembly parsing
- Type checking (AsmTypeCheck)
- Disassembler
- Fixing unwind mismatches in CFGStackify
These will be added as follow-up PRs.
---
The format for `TRY_TABLE`, both for `MachineInstr` and `MCInst`, is as
follows:
```
TRY_TABLE type number_of_catches catch_clauses*
```
where `catch_clause` is
```
catch_opcode tag+ destination
```
`catch_opcode` should be one of 0/1/2/3, which denotes
`CATCH`/`CATCH_REF`/`CATCH_ALL`/`CATCH_ALL_REF` respectively. (See
`BinaryFormat/Wasm.h`) `tag` exists when the catch is one of `CATCH` or
`CATCH_REF`.
The MIR format is printed as just the list of raw operands. The
(stack-based) assembly instruction supports pretty-printing, including
printing `catch` clauses by name, in InstPrinter.
In addition to the new instructions `TRY_TABLE` and `THROW_REF`, this
adds four pseudo instructions: `CATCH`, `CATCH_REF`, `CATCH_ALL`, and
`CATCH_ALL_REF`. These are pseudo instructions to simulate block return
values of `catch`, `catch_ref`, `catch_all`, `catch_all_ref` clauses in
`try_table` respectively, given that we don't support block return
values except for one case (`fixEndsAtEndOfFunction` in CFGStackify).
These will be omitted when we lower the instructions to `MCInst` at the
end.
LateEHPrepare now will have one more stage to covert
`CATCH`/`CATCH_ALL`s to `CATCH_REF`/`CATCH_ALL_REF`s when there is a
`RETHROW` to rethrow its exception. The pass also converts `RETHROW`s
into `THROW_REF`. Note that we still use `RETHROW` as an interim pseudo
instruction until we convert them to `THROW_REF` in LateEHPrepare.
CFGStackify has a new `placeTryTableMarker` function, which places
`try_table`/`end_try_table` markers with a necessary `catch` clause and
also `block`/`end_block` markers for the destination of the `catch`
clause.
In MCInstLower, now we need to support one more case for the multivalue
block signature (`catch_ref`'s destination's `(i32, exnref)` return
type).
InstPrinter has a new routine to print the `catch_list` type, which is
used to print `try_table` instructions.
The new test, `exception.ll`'s source is the same as
`exception-legacy.ll`, with the FileCheck expectations changed. One
difference is the commands in this file have `-wasm-enable-exnref` to
test the new format, and don't have `-wasm-disable-explicit-locals
-wasm-keep-registers`, because the new custom InstPrinter routine to
print `catch_list` only works for the stack-based instructions (`_S`),
and we can't use `-wasm-keep-registers` for them.
As in `exception-legacy.ll`, the FileCheck lines for the new tests do
not contain the whole program; they mostly contain only the control flow
instructions for readability.
This allows us to reduce VLs feeding reduction instructions. In
particular, this means that <3 x Ty> reduce(load) like sequences no
longer require a VL toggle.
This was waiting on 3d72957; now that the latent correctness issue is
fixed, we can expand this transform.
Continuing with #107993 and #108007, this handles the last of the main
rematerializable vector instructions.
There's an extra spill in one of the test cases, but it's likely noise
from the spill weights and isn't an issue in practice.
Even though vmv.v.x has a non constant scalar operand, we can still
rematerialize it because we have split register allocation between
vectors and scalars.
InlineSpiller will check to make sure that the scalar operand is live at
the point where the rematerialization occurs, so this won't extend any
scalar live ranges. However this also means we may not be able to
rematerialize in some cases, as shown in @vmv.v.x_needs_extended.
It might be worthwhile teaching InlineSpiller to extend scalar live
ranges in a future patch. I experimented with this locally and it
reduced spills on 531.deepsjeng_r by a further 3%.
The previous `attempt to fix [CGData][MachineOutliner] Global Outlining
(#90074) #108037` was incomplete because the
`ImmutableModuleSummaryIndexWrapperPass` is now optional for the
MachineOutliner pass.
With this fix, the test file `CodeGen/AArch64/O3-pipeline.ll` shows no
changes compared to its state before `[CGData][MachineOutliner] Global
Outlining (#90074)`.
Co-authored-by: Kyungwoo Lee <kyulee@fb.com>
This commit introduces support for outlining functions across modules
using codegen data generated from previous codegen. The codegen data
currently manages the outlined hash tree, which records outlining
instances that occurred locally in the past.
The machine outliner now operates in one of three modes:
1. CGDataMode::None: This is the default outliner mode that uses the
suffix tree to identify (local) outlining candidates within a module.
This mode is also used by (full)LTO to maintain optimal behavior with
the combined module.
2. CGDataMode::Write (`-codegen-data-generate`): This mode is identical
to the default mode, but it also publishes the stable hash sequences of
instructions in the outlined functions into a local outlined hash tree.
It then encodes this into the `__llvm_outline` section, which will be
dead-stripped at link time.
3. CGDataMode::Read (`-codegen-data-use-path={.cgdata}`): This mode
reads a codegen data file (.cgdata) and initializes a global outlined
hash tree. This tree is used to generate global outlining candidates.
Note that the codegen data file has been post-processed with the raw
`__llvm_outline` sections from all native objects using the
`llvm-cgdata` tool (or a linker, `LLD`, or a new ThinLTO pipeline
later).
This depends on https://github.com/llvm/llvm-project/pull/105398. After
this PR, LLD (https://github.com/llvm/llvm-project/pull/90166) and Clang
(https://github.com/llvm/llvm-project/pull/90304) will follow for each
client side support.
This is a patch for
https://discourse.llvm.org/t/rfc-enhanced-machine-outliner-part-2-thinlto-nolto/78753.
When AArch64LoadStoreOptimizer pass merges an SP update with a
load/store instruction and needs to adjust unwind information either:
* create the merged instruction at the location of the SP update
(so no CFI instructions are moved), or
* only move a CFI instruction if the move would not reorder it across
other CFI instructions
If neither of the above is possible, don't perform the optimisation.
This intrinsic is meant to be used in functions that have a "tail" that
needs to be run with all the lanes enabled. The "tail" may contain
complex control flow that makes it unsuitable for the use of the
existing WWM intrinsics. Instead, we will pretend that the function
starts with all the lanes enabled, then branches into the actual body of
the function for the lanes that were meant to run it, and then finally
all the lanes will rejoin and run the tail.
As such, the intrinsic will return the EXEC mask for the body of the
function, and is meant to be used only as part of a very limited pattern
(for now only in amdgpu_cs_chain functions):
```
entry:
%func_exec = call i1 @llvm.amdgcn.init.whole.wave()
br i1 %func_exec, label %func, label %tail
func:
; ... stuff that should run with the actual EXEC mask
br label %tail
tail:
; ... stuff that runs with all the lanes enabled;
; can contain more than one basic block
```
It's an error to use the result of this intrinsic for anything
other than a branch (but unfortunately checking that in the verifier is
non-trivial because SIAnnotateControlFlow will introduce an amdgcn.if
between the intrinsic and the branch).
The intrinsic is lowered to a SI_INIT_WHOLE_WAVE pseudo, which for now
is expanded in si-wqm (which is where SI_INIT_EXEC is handled too);
however the information that the function was conceptually started in
whole wave mode is stored in the machine function info
(hasInitWholeWave). This will be useful in prolog epilog insertion,
where we can skip saving the inactive lanes for CSRs (since if the
function started with all the lanes active, then there are no inactive
lanes to preserve).
GCC compiles the built-in function `__builtin_bswap16`, to the ARM
instruction rev16, which reverses the byte order of 16-bit data. On the
other Clang compiles the same built-in function to e.g.
```
rev w8, w0
lsr w0, w8, #16
```
i.e. it performs a byte reversal of a 32-bit register, (which moves the
lower half, which contains the 16-bit data, to the upper half) and then
right shifts the reversed 16-bit data back to the lower half of the
register.
We can improve Clang codegen by generating `rev16` instead of `rev` and
`lsr`, like GCC.
This fixes#107950 and adds a test case for it. The issue was due to
us incorrectly assuming that we stored a V0Defs entry for every single
instruction.
We actually only store them for instructions that use V0, so when we
updated the V0Def after moving we sometimes ended up copying nullptr
over from an instruction that doesn't use V0 and clearing the V0Def
entry inadvertently.
Because we don't have V0Defs on instructions that don't use V0, the
FIXME was never actually needed in the first place since the
bookkeeping wasn't out of sync to begin with.
That commit also mentioned that a future unmasked to masked pseudo
peephole might need unmasked pseudos to have V0Defs entries, but after
working on this locally it turns out we don't.
This reverts commit ce3648094d44e8c098396a353b215acecb363cda.
In #107827 we now set true's passthru to the false operand if it was
undef. We need to remember to also constrain the regclass in case true
is a masked pseudo which needs its passthrus to be in VR[M*]NoV0
shifts are the same as sub where rhs == 0 is identity.
and is the inverted case where:
`SELECT (AND(X,1) == 0), (AND Y, Z), Y`
-> `(AND Y, (OR NEG(AND(X, 1)), Z))`
With -1 as the identity.
Closes#107910
Fixes generation of invalid loads leading to misaligned access errors.
The bug got exposed by SLP vectorizer change ec360d6 which allowed SLP
to produce `v16i8` vectors.
Also updated the tests to use automatic check generator.
