Moving these into the middle-end pipeline will allow for additional
optimization of the expansion result, such as CSE of redundant loads
(c.f. https://godbolt.org/z/bEna4Md9r). For now, we conservatively place
the passes at the end of the middle-end pipeline, so we mostly don't
benefit from additional optimizations yet. The pipeline position will be
moved in a future change.
This builds on work done by legrosbuffle in
https://reviews.llvm.org/D60318.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This PR enables `G_ADD` and immediate dependencies (relavent ext and
trunc related ops) to be fully legalized and selected.
The most important change made is getting the boilerplate for
RegBankSelect working.
Split from #157161.
Split the input vector with an extend_low and high and then split the
results again with extend_low and high for a total of 6 instructions.
This is removes 3 shuffles and a couple of extends.
74e4694 moved ObjCARCContract from running before the codegen pipeline
into addISelPrepare(), which runs after PreISelIntrinsicLowering.
This broke ObjCARCContract's retainRV-to-claimRV optimization because
ObjCARCContract identifies ARC calls via intrinsics, not their lowered
counterparts.
This patch restores the pre-74e4694 ordering by moving ObjCARCContract
to addISelPasses.
The IntrinsicInst.cpp change looks extraneous but is required here:
ObjCARCContract may now rewrite the bundle operand from retainRV to
claimRV. When PreISelIntrinsicLowering then encounters this new
intrinsic use, lowerObjCCall asserts mayLowerToFunctionCall.
Assisted-by: claude
rdar://137997453
If Subtarget is null during global emission, it is now retrieved
from TargetMachine to prevent crashes caused by empty VTs in
wasmSymbolSetType.
Fixed: https://github.com/llvm/llvm-project/issues/181527
DAGCombiner canonicalizes `and(vnot(x), sext(cmp))` into `vselect(cmp,
vnot(x), zero)`. Without a dedicated ISel pattern, this falls through to
the generic vselect-to-bitselect lowering, producing `v128.not` +
`v128.bitselect`. This patch adds a higher-priority pattern that emits a
single `v128.andnot` instead, saving two instructions.
The direct `and(vnot(x), y)` pattern is already handled by the existing
`andnot` PatFrag. This patch covers the indirect path through
DAGCombiner's `and(x, sext(cmp))` → `vselect(cmp, x, zero)`
canonicalization, which obscures the `andnot` opportunity when `x` is
`vnot(y)`.
Assisted-by: Claude (Anthropic)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FastISel can already fold some sign- and zero-extending loads, but a
number of i64 extension patterns still leave redundant instructions
behind.
This patch series extends load folding to recognize several such cases,
including:
- promoted sign-extension chains
- copy + i64.extend_i32_{u,s} chains
- AND-based zero-extension chains
- shift-based sign-extension chains
When these patterns originate from narrow integer loads, fold them
directly into widened i64 loads instead of materializing intermediate
i32 loads followed by redundant i64 extends.
Fixes#179672
fixes https://github.com/llvm/llvm-project/issues/129441
cc @lukel97 @badumbatish
https://github.com/llvm/llvm-project/pull/145108
I've been learning a bit about LLVM, trying to make progress on some of
these issues. The code below is based on
https://github.com/llvm/llvm-project/pull/145108#issuecomment-3004561085,
by implementing `shouldExpandReduction`.
The implementation works for the test cases I added, but (obviously)
fails for any existing cases. `ISD::VECREDUCE_AND` and
`ISD::VECREDUCE_OR` are now marked as legal, which is required for the
`Pat`s to fire, but when they don't that causes a selection failure.
So, I'm wondering, what is the right approach here. Should I mark these
intrinsics as `Custom` instead and manually perform the transformation
in C++? Or is there some trick to still get the default lowering (a
series of loads and scalar bitwise operations) when the patterns don't
fire?
Selects of the form `cond ? 1 : 0` are created during unrolling of
setcc+vselect. Currently these are not optimized away post-legalization
even if fully redundant. Having these extra selects sitting between
things can prevent other folds from applying.
Enabling this requires some mitigations in the ARM backend, in
particular in the interaction with MVE support. There's two changes
here:
* Form CSINV/CSNEG/CSINC from CMOV, rather than only creating it during
SELECT_CC lowering. (After this change, the lowering in SELECT_CC can be
dropped without test changes, let me know if I should do that.)
* Support pushing negations through CMOV in more cases, in particular if
the operands are constant or the negation can be handled by flipping
lshr/ashr.
Additionally, in the X86 backend, try to simplify CMOV to SETCC if only the
low bit is demanded.
WebAssembly FastISel currently fails to fold sign-extension patterns
composed of zero-extending loads followed by shift operations. This
results in redundant shift and constant instructions in the output.
Before:
i32.load8_u $push3=, 0($0)
i32.const $push0=, 24
i32.shl $push1=, $pop3, $pop0
i32.const $push4=, 24
i32.shr_s $push2=, $pop1, $pop4
The matched shift instruction sequence is removed and safely folded into
a single sign-extending load, erasing the dead code via the
MachineBasicBlock iterator.
After:
i32.load8_s $push0=, 0($0)
Fixed: #184302
The rust `simd_bitmask` intrinsic is UB when the lanes of its input are
not either `0` or `!0`, presumably so that the implementation can be
more efficient because it could look at any bit. To get the "mask of
MSB" behavior of webassembly's `bitmask`, we would like to simply first
compare with a zero vector.
```llvm
define i32 @example(<2 x i64> noundef %v) {
entry:
%1 = icmp slt <16 x i8> %v, zeroinitializer
%2 = bitcast <16 x i1> %1 to i16
%3 = zext i16 %2 to i32
ret i32 %3
}
```
On x86_64, this additional comparison optimizes away, but for wasm it
does not.
https://godbolt.org/z/T5sPejocs
This PR adds a new combine, so that instead of emitting
```asm
example:
local.get 0
v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
i8x16.lt_s
i8x16.bitmask
end_function
```
we just emit
```
example:
local.get 0
i8x16.bitmask
end_function
```
Set the correct memory ordering for relaxed atomics after ISel. This
allows
SelectionDAG to keep the simple generic selection for target-independent
AtomicLoad nodes, but keeps the ordering immediate correct in the MIR.
Notably, the MachineMemOperand still has the original memory ordering
and MIR passes would use that rather than the ordering immedate to make
their code motion decisions (if we had any for Wasm, which we don't).
Wide vector multiplications by power-of-2 constants were
canonicalized to v8i32 shl nodes. Generic legalizers then split these
into separate 128-bit extend and shift operations, bypassing
WebAssembly's native extended multiplication patterns.
Before:
mul v8i32:t1, <4096, ...>
=> shl v8i32:t1, <12, ...>
=> split into independent 128-bit extend + shift sequences
WebAssembly SIMD has no native wide vector shifts, but it does
support 128-bit extended multiplications. Lowering these nodes
directly to extmul_low/extmul_high pairs keeps them in native 128-bit
form and improves DAG matching.
After:
mul v8i32:t1, <4096, ...>
=> concat_vectors (extmul_low t1, c), (extmul_high t1, c)
This preserves the original vector width while utilizing the native
128-bit SIMD pipeline.
Fixed: https://github.com/llvm/llvm-project/issues/179143
FastISel emits separate load and AND instructions for bitmasking.
(before) %1:i32 = LOAD_I32 %addr; %2:i32 = AND_I32 %1, 255
Fold AND masks into ZExt loads by verifying operands with
maskTrailingOnes. A getFoldedLoadOpcode wrapper is implemented
to manage dispatching logic for better extensibility.
(after) %1:i32 = LOAD8_U_I32 %addr
Fixed: https://github.com/llvm/llvm-project/issues/180783
When folding loads with extensions, the extension operand can be a freeze node
in addition to a load. We can look through it to do the desirability check.
Fixes#184676
Existing ISel patterns match setne/seteq following SIMD boolean reductions
any_true and all_true, and drop the ones that are redundant (because the
reductions always return 1 or 0). This adds patterns to also produce eqz
instructions instead of a comparison with a const.
Current tooling for the WebAssembly component model uses import modules
and names such as `$root` and `[thread-index]`. Importing these from
assembly files requires support for non-valid identifiers in
`.import_name` and `.import_module` directives. This PR adds support for
specifying those as strings, e.g.:
```asm
.import_module __wasm_component_model_builtin_thread_index, "$root"
.import_name __wasm_component_model_builtin_thread_index, "[thread-index]"
```
For wasm, forming minnum/maxnum style ISD nodes is non-profitable,
because (in cases where any float min/max support exists at all), it has
pmin/pmax instructions that correspond to the fcmp+select semantics, or
relaxed_fmin/relaxed_fmax (for the nnan+nsz case) with even loser
semantics.
As such, return false from isProfitableToCombineMinNumMaxNum(), and also
respect that hook in the SDAGBuilder.
Initial support for acquire-release atomics, specified as part of
https://github.com/WebAssembly/shared-everything-threads
This adds an ordering operand to atomic loads, stores, RMWs,
wait/notify,
and fences. It currently defaults to 0 and ISel is not updated yet, so
atomics produced by the compiler will still always be seqcst.
Asm parsing and printing, binary emission and disassembly are all
updated. Binary emission will always use the old encoding because the
encoding is smaller, and to get backwards compatibility for free.
The CombineSetCC helpers and performAnyAllCombine generate MVT::i1
results.
However MVT::i1 is an illegal type in WebAssembly, and this combiner can
run either before or after legalization. Directly creating the intrinsic
and negating its result using XOR instead of i1 and a NOT operation
avoids this problem.
Fixes#183842
The `tryToFoldLoadIntoMI` function omitted materializing base registers
for addresses before folding sign-extend instructions into loads. This
left `$noreg` as the base register, crashing subsequent passes.
WebAssembly memory instructions structurally require a valid base
register. Calling the existing `materializeLoadStoreOperands` function
ensures that a `CONST 0` virtual register is generated when addressing
global variables directly without a pre-existing base register.
(before) %1:i32 = LOAD8_S_I32_A32 0, @ch, $noreg ... -> CRASH (after)
%3:i32 = CONST_I32 0
%1:i32 = LOAD8_S_I32_A32 0, @ch, %3:i32 ... -> Folded safely
FastISel currently defaults to unsigned loads for i8/i16/i32 types,
leaving any sign-extension to be handled by a separate instruction. This
patch optimizes this by folding the SExtInst into the LoadInst, directly
emitting a signed load (e.g., i32.load8_s).
When a load has a single SExtInst use, selectLoad emits a signed load
and safely removes the redundantly emitted SExtInst.
Fixed: #180783
cc https://github.com/llvm/llvm-project/issues/179143
This adds a second pattern: we already recognize "shuffle + extend +
add" as `addext`, this adds another pattern for "extend + shuffle +
add", which can come up when programs are optimized.
SELECT_CC nodes with externref or funcref return types were not being
expanded, causing "Cannot select" errors during instruction selection.
This adds SELECT_CC to the list of operations that should be expanded
for reference types, similar to how it's already handled for scalar
types (i32, i64, f32, f64). This allows the SELECT_CC to be lowered to a
SELECT node, which already has instruction patterns defined in
WebAssemblyInstrRef.td.
This reverts commit 8d29d09309654541fb2861524276ada6a3ebf84c.
The underlying issue causing the revert has been fixed independently
as 301fa24671256734df6b7ee65f23ad885400108e.
Original message:
Move narrowInterleaveGroups to to general VPlan optimization stage.
To do so, narrowInterleaveGroups now has to find a suitable VF where all
interleave groups are consecutive and saturate the full vector width.
If such a VF is found, the original VPlan is split into 2:
a) a new clone which contains all VFs of Plan, except VFToOptimize, and
b) the original Plan with VFToOptimize as single VF.
The original Plan is then optimized. If a new copy for the other VFs has
been created, it is returned and the caller has to add it to the list of
candidate plans.
Together with https://github.com/llvm/llvm-project/pull/149702, this
allows to take the narrowed interleave groups into account when
computing costs to choose the best VF and interleave count.
One example where we currently miss interleaving/unrolling when
narrowing interleave groups is https://godbolt.org/z/Yz77zbacz
PR: https://github.com/llvm/llvm-project/pull/149706
This checks every user function of `setjmp` or `longjmp` and if any of
them does not have `+exception-handling` target feature, errors out.
Hopefully this gives a clearer error message to the users in case they
do not provide consistent SjLj flags at compile time vs. link time.
Closes#178135 and closes
https://github.com/emscripten-core/emscripten/issues/26165.
This lowers the splitting as:
```
any_active(hi_mask)
? (find_last_active(hi_mask) + lo_mask.getVectorElementCount())
: find_last_active(lo_mask)
```
And trivially lowers `<1 x i1>` scalarization to returning zero. Which
is a natural result of the splitting (and the lack of a sentinel
"none-active" result value).
The lowerings likely can be improved. This patch is for completeness.
Should fix:
https://github.com/llvm/llvm-project/pull/178862#issuecomment-3862310334Fixes#180212
We don't currently support common symbols for Wasm, and we currently
emit a generic error with a backtrace. Instead, don't crash, and report
the names of the offending symbols.
Before, Wasm FastISel treated all indirect calls the same, causing
miscompilations at O0 when trying to call a funcref (`call ptr
addrspace(20)`), as it would treat the funcref as a normal `ptr`
This adds a check so it falls back to ISelDAG when encountering calls
outside addrspace 0 (which covers direct calls and indirect calls
through normal function pointers).
Related: #140933
Fixes a crash during type legalization by allowing
ISD::ANY_EXTEND_VECTOR_INREG to fall back to default expansion instead
of hitting llvm_unreachable.
Fixed: #177209
Custom lower FMINNUM, FMINIMUMNUM, FMAXNUM and FMAXIMUMNUM to generate
relaxed_min and relaxed_max when the inputs cannot be NaN or signed
zero.
Tablegen patterns have also been modified to check the above conditions
when trying to match relaxed min/max using the pmin/pmax pattern.
This restriction was originally added in
https://reviews.llvm.org/D143256, with the given justification:
> Currently, in TargetLowering, if the target does not support fminnum,
we lower to fminimum if neither operand could be a NaN. But this isn't
quite correct because fminnum and fminimum treat +/-0 differently; so,
we need to prove that one of the operands isn't a zero.
As far as I can tell, this was never correct. Before
https://github.com/llvm/llvm-project/pull/172012, `minnum` and `maxnum`
were nondeterministic with regards to signed zero, so it's always been
perfectly legal to lower them to operations that order signed zeroes.
These are long overdue for removal. These were originally a hack
to support loading half values before there was any / decent support
for the half type through the backend. There's no reason to continue
supporting these, they're equivalent to fpext/fptrunc with a bitcast.
SelectionDAG stopped translating these directly, and used the
bitcast + fp cast since f7a02c17628e825, so there's been no reason
to use these since 2014.
