Block numbers are updated too frequently, which makes it difficult to
keep analyses up to date. Therefore, introduce a second number per basic
block that is used for analyses and is renumbered less often. This frees
analyses from providing somewhat efficient facilities for dealing with
changed block numbers, making it simpler to implement in e.g. LoopInfo
or CycleInfo.
(Currently, "less often" means not at all, but we might want to renumber
after certain passes if the numbering gets too sparse and no analyses
are preserved anyway.)
When we introduced a more general use of block numbers some time ago,
using the existing numbers seemed to be a somewhat obvious choice, but I
now think that this was a bad decision, as it conflates a number that is
used for ordering with a number that should be more stable.
MachineBasicBlock isn't particularly size-optimized and there's a fair
amount of padding where we can add another number.
There should be no performance impact,
WebAssembly FastISel currently fails to fold sign-extension patterns
composed of zero-extending loads followed by shift operations. This
results in redundant shift and constant instructions in the output.
Before:
i32.load8_u $push3=, 0($0)
i32.const $push0=, 24
i32.shl $push1=, $pop3, $pop0
i32.const $push4=, 24
i32.shr_s $push2=, $pop1, $pop4
The matched shift instruction sequence is removed and safely folded into
a single sign-extending load, erasing the dead code via the
MachineBasicBlock iterator.
After:
i32.load8_s $push0=, 0($0)
Fixed: #184302
The rust `simd_bitmask` intrinsic is UB when the lanes of its input are
not either `0` or `!0`, presumably so that the implementation can be
more efficient because it could look at any bit. To get the "mask of
MSB" behavior of webassembly's `bitmask`, we would like to simply first
compare with a zero vector.
```llvm
define i32 @example(<2 x i64> noundef %v) {
entry:
%1 = icmp slt <16 x i8> %v, zeroinitializer
%2 = bitcast <16 x i1> %1 to i16
%3 = zext i16 %2 to i32
ret i32 %3
}
```
On x86_64, this additional comparison optimizes away, but for wasm it
does not.
https://godbolt.org/z/T5sPejocs
This PR adds a new combine, so that instead of emitting
```asm
example:
local.get 0
v128.const 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
i8x16.lt_s
i8x16.bitmask
end_function
```
we just emit
```
example:
local.get 0
i8x16.bitmask
end_function
```
Set the correct memory ordering for relaxed atomics after ISel. This
allows
SelectionDAG to keep the simple generic selection for target-independent
AtomicLoad nodes, but keeps the ordering immediate correct in the MIR.
Notably, the MachineMemOperand still has the original memory ordering
and MIR passes would use that rather than the ordering immedate to make
their code motion decisions (if we had any for Wasm, which we don't).
Wide vector multiplications by power-of-2 constants were
canonicalized to v8i32 shl nodes. Generic legalizers then split these
into separate 128-bit extend and shift operations, bypassing
WebAssembly's native extended multiplication patterns.
Before:
mul v8i32:t1, <4096, ...>
=> shl v8i32:t1, <12, ...>
=> split into independent 128-bit extend + shift sequences
WebAssembly SIMD has no native wide vector shifts, but it does
support 128-bit extended multiplications. Lowering these nodes
directly to extmul_low/extmul_high pairs keeps them in native 128-bit
form and improves DAG matching.
After:
mul v8i32:t1, <4096, ...>
=> concat_vectors (extmul_low t1, c), (extmul_high t1, c)
This preserves the original vector width while utilizing the native
128-bit SIMD pipeline.
Fixed: https://github.com/llvm/llvm-project/issues/179143
Largely a straight-forward replacement with occasional simplifcations.
For AMDGPU, I assumed that unconditional branches are always uniform and
therefore "simplified"/changed AMDGPUAnnotateUniformValues to only
annotate conditional branches.
Target-specific FastISel only selects conditional branches,
unconditional branches are already handled by the non-target-specific
code.
BranchInst currently represents both unconditional and conditional
branches. However, these are quite different operations that are often
handled separately. Therefore, split them into separate opcodes and
classes to allow distinguishing these operations in the type system.
Additionally, this also slightly improves compile-time performance.
FastISel emits separate load and AND instructions for bitmasking.
(before) %1:i32 = LOAD_I32 %addr; %2:i32 = AND_I32 %1, 255
Fold AND masks into ZExt loads by verifying operands with
maskTrailingOnes. A getFoldedLoadOpcode wrapper is implemented
to manage dispatching logic for better extensibility.
(after) %1:i32 = LOAD8_U_I32 %addr
Fixed: https://github.com/llvm/llvm-project/issues/180783
When folding loads with extensions, the extension operand can be a freeze node
in addition to a load. We can look through it to do the desirability check.
Fixes#184676
Existing ISel patterns match setne/seteq following SIMD boolean reductions
any_true and all_true, and drop the ones that are redundant (because the
reductions always return 1 or 0). This adds patterns to also produce eqz
instructions instead of a comparison with a const.
This removes the `wasm-disable-fix-irreducible-control-flow-pass`
switch.
It was originally added in #67715 as a way to avoid the potentially
absurd compile times the pass used to bring. However with the successful
merge of #184441, the pass itself has been fixed to avoid this issue.
Given that, it is no longer necessary nor desirable to keep this switch.
Current tooling for the WebAssembly component model uses import modules
and names such as `$root` and `[thread-index]`. Importing these from
assembly files requires support for non-valid identifiers in
`.import_name` and `.import_module` directives. This PR adds support for
specifying those as strings, e.g.:
```asm
.import_module __wasm_component_model_builtin_thread_index, "$root"
.import_name __wasm_component_model_builtin_thread_index, "[thread-index]"
```
For wasm, forming minnum/maxnum style ISD nodes is non-profitable,
because (in cases where any float min/max support exists at all), it has
pmin/pmax instructions that correspond to the fcmp+select semantics, or
relaxed_fmin/relaxed_fmax (for the nnan+nsz case) with even loser
semantics.
As such, return false from isProfitableToCombineMinNumMaxNum(), and also
respect that hook in the SDAGBuilder.
Initial support for acquire-release atomics, specified as part of
https://github.com/WebAssembly/shared-everything-threads
This adds an ordering operand to atomic loads, stores, RMWs,
wait/notify,
and fences. It currently defaults to 0 and ISel is not updated yet, so
atomics produced by the compiler will still always be seqcst.
Asm parsing and printing, binary emission and disassembly are all
updated. Binary emission will always use the old encoding because the
encoding is smaller, and to get backwards compatibility for free.
The CombineSetCC helpers and performAnyAllCombine generate MVT::i1
results.
However MVT::i1 is an illegal type in WebAssembly, and this combiner can
run either before or after legalization. Directly creating the intrinsic
and negating its result using XOR instead of i1 and a NOT operation
avoids this problem.
Fixes#183842
The `tryToFoldLoadIntoMI` function omitted materializing base registers
for addresses before folding sign-extend instructions into loads. This
left `$noreg` as the base register, crashing subsequent passes.
WebAssembly memory instructions structurally require a valid base
register. Calling the existing `materializeLoadStoreOperands` function
ensures that a `CONST 0` virtual register is generated when addressing
global variables directly without a pre-existing base register.
(before) %1:i32 = LOAD8_S_I32_A32 0, @ch, $noreg ... -> CRASH (after)
%3:i32 = CONST_I32 0
%1:i32 = LOAD8_S_I32_A32 0, @ch, %3:i32 ... -> Folded safely
Rather than mapping out full "reachability" between blocks in a region
to find loops and using `LoopBlocks` to find the bodies of said loops,
use SCCs (strongly-connected components) to provide this information.
This brings in LLVM's generic `SCCIterator` (which uses Tarjan's
algorithm) as the implementation for sorting the basic blocks of the CFG
into their SCCs.
This PR greatly reduces the compile-time footprint of the pass, making
memory use and time taken negliable where it might have previously
caused stalls and OOM before (e.g. #47793,
usagi-coffee/tree-sitter-abl#114)
------
Supersedes #179722Fixes#47793Fixes#165041 (probably)
Thanks to @jkbz64 for the initial investigations (w/ AI; see #179722)
into why this pass was slow and memory consuming and showing that SCCs
were the key.
Also thanks to the Cheerp compiler project for bringing `SCCIterator` to
light in this context ([blog
post](https://cheerp.io/blog/control-flow#fix-the-irreducible-control-flow),
[implementation](https://github.com/leaningtech/cheerp-compiler/blob/master/llvm/lib/CheerpUtils/FixIrreducibleControlFlow.cpp)).
FastISel currently defaults to unsigned loads for i8/i16/i32 types,
leaving any sign-extension to be handled by a separate instruction. This
patch optimizes this by folding the SExtInst into the LoadInst, directly
emitting a signed load (e.g., i32.load8_s).
When a load has a single SExtInst use, selectLoad emits a signed load
and safely removes the redundantly emitted SExtInst.
Fixed: #180783
cc https://github.com/llvm/llvm-project/issues/179143
This adds a second pattern: we already recognize "shuffle + extend +
add" as `addext`, this adds another pattern for "extend + shuffle +
add", which can come up when programs are optimized.
Currently, the post-sort verifications `WebAssemblyCFGSort` does in
assertion builds is quite expensive. Given the example of
[usagi-coffee/tree-sitter-abl](https://github.com/usagi-coffee/tree-sitter-abl),
we have one MASSIVE function with nearly 120K blocks. Sorting this is
fast, but the verifications after take some 40+ seconds (on my machine).
The culprit was the manipulation of the region `OnStack`. I've tried to
replace that with interval based verifications to make sure the regions
are well formed, well nested, and that the numbers of the blocks within
a region are within the interval of the numbers of the header and bottom
of the region.
This brings the aforementioned example down to well under a second spent
in the pass in an assertions build.
SELECT_CC nodes with externref or funcref return types were not being
expanded, causing "Cannot select" errors during instruction selection.
This adds SELECT_CC to the list of operations that should be expanded
for reference types, similar to how it's already handled for scalar
types (i32, i64, f32, f64). This allows the SELECT_CC to be lowered to a
SELECT node, which already has instruction patterns defined in
WebAssemblyInstrRef.td.
Since #182059 there is only one case in which these functions return -1,
so callers no longer need to distinguish between (int64_t)-1 and
(uint32_t)-1, so we can go back to a 32-bit return value like it was
before #180954.
The failures should have been resolved with #180290 (which also added
WebAssembly tests).
This reverts commit 811fb223af2b3e2d68c99b346f4b75dcf3de3417.
This checks every user function of `setjmp` or `longjmp` and if any of
them does not have `+exception-handling` target feature, errors out.
Hopefully this gives a clearer error message to the users in case they
do not provide consistent SjLj flags at compile time vs. link time.
Closes#178135 and closes
https://github.com/emscripten-core/emscripten/issues/26165.
This change adds initial support to libObject for reading compact
imports and support for writing compact imports in the linker.
There is minimal testing here since to tools like lllvm-readobj, and
obj2yaml don't currently report compact imports any differently.
See https://github.com/WebAssembly/compact-import-section
We don't currently support common symbols for Wasm, and we currently
emit a generic error with a backtrace. Instead, don't crash, and report
the names of the offending symbols.
Before, Wasm FastISel treated all indirect calls the same, causing
miscompilations at O0 when trying to call a funcref (`call ptr
addrspace(20)`), as it would treat the funcref as a normal `ptr`
This adds a check so it falls back to ISelDAG when encountering calls
outside addrspace 0 (which covers direct calls and indirect calls
through normal function pointers).
Related: #140933
Fixes a crash during type legalization by allowing
ISD::ANY_EXTEND_VECTOR_INREG to fall back to default expansion instead
of hitting llvm_unreachable.
Fixed: #177209
Custom lower FMINNUM, FMINIMUMNUM, FMAXNUM and FMAXIMUMNUM to generate
relaxed_min and relaxed_max when the inputs cannot be NaN or signed
zero.
Tablegen patterns have also been modified to check the above conditions
when trying to match relaxed min/max using the pmin/pmax pattern.
This commit introduces the VectorInstrContext (VIC) infrastructure to
improve cost estimates for insert/extracts based on the context
instruction in which the insert/extract is used.
This is similar to CastContextHint, and allows providing context on how
the insert/extract is going to be used before creating IR. This is
useful in the LoopVectorizer, where costs need to estimated before
creating IR.
The new hint currently only replaces an existing check in AArch64,
but new uses will be introduced in follow-ups, including
https://github.com/llvm/llvm-project/pull/177201.
PR: https://github.com/llvm/llvm-project/pull/175982
The WebAssemblyTargetMachine constructor had an ordering issue where
initAsmInfo() was called before basicCheckForEHAndSjLj(). This caused
problems in incremental compilation scenarios where:
1. `initAsmInfo()` sets `MCAsmInfo` exception type based on
`Options.ExceptionModel`
2. But `Options.ExceptionModel` might still be None at this point
3. `basicCheckForEHAndSjLj()` runs later and updates
`Options.ExceptionModel`
based on command-line flags like `-wasm-enable-eh`
4. `MCAsmInfo` retains the incorrect exception type (`None` instead of
`Wasm`)
5. This prevents WebAssembly exception handling passes from running
The fix swaps the order so basicCheckForEHAndSjLj() runs first to
establish the correct exception model before initAsmInfo() configures
MCAsmInfo based on that model.
This enables WebAssembly exception handling to work correctly in
clang-repl and other incremental compilation scenarios.
Returns uint64_t to simplify callers. The goal is eventually replace
getValueType with this query, which should return the known minimum
reference-able size, as provided (instead of a Type) during create.
Additionally the common isSized query would be replaced with an
isExactKnownSize query to test if that size is an exact definition.
