Rather than mapping out full "reachability" between blocks in a region
to find loops and using `LoopBlocks` to find the bodies of said loops,
use SCCs (strongly-connected components) to provide this information.
This brings in LLVM's generic `SCCIterator` (which uses Tarjan's
algorithm) as the implementation for sorting the basic blocks of the CFG
into their SCCs.
This PR greatly reduces the compile-time footprint of the pass, making
memory use and time taken negliable where it might have previously
caused stalls and OOM before (e.g. #47793,
usagi-coffee/tree-sitter-abl#114)
------
Supersedes #179722Fixes#47793Fixes#165041 (probably)
Thanks to @jkbz64 for the initial investigations (w/ AI; see #179722)
into why this pass was slow and memory consuming and showing that SCCs
were the key.
Also thanks to the Cheerp compiler project for bringing `SCCIterator` to
light in this context ([blog
post](https://cheerp.io/blog/control-flow#fix-the-irreducible-control-flow),
[implementation](https://github.com/leaningtech/cheerp-compiler/blob/master/llvm/lib/CheerpUtils/FixIrreducibleControlFlow.cpp)).
FastISel currently defaults to unsigned loads for i8/i16/i32 types,
leaving any sign-extension to be handled by a separate instruction. This
patch optimizes this by folding the SExtInst into the LoadInst, directly
emitting a signed load (e.g., i32.load8_s).
When a load has a single SExtInst use, selectLoad emits a signed load
and safely removes the redundantly emitted SExtInst.
Fixed: #180783
cc https://github.com/llvm/llvm-project/issues/179143
This adds a second pattern: we already recognize "shuffle + extend +
add" as `addext`, this adds another pattern for "extend + shuffle +
add", which can come up when programs are optimized.
Currently, the post-sort verifications `WebAssemblyCFGSort` does in
assertion builds is quite expensive. Given the example of
[usagi-coffee/tree-sitter-abl](https://github.com/usagi-coffee/tree-sitter-abl),
we have one MASSIVE function with nearly 120K blocks. Sorting this is
fast, but the verifications after take some 40+ seconds (on my machine).
The culprit was the manipulation of the region `OnStack`. I've tried to
replace that with interval based verifications to make sure the regions
are well formed, well nested, and that the numbers of the blocks within
a region are within the interval of the numbers of the header and bottom
of the region.
This brings the aforementioned example down to well under a second spent
in the pass in an assertions build.
SELECT_CC nodes with externref or funcref return types were not being
expanded, causing "Cannot select" errors during instruction selection.
This adds SELECT_CC to the list of operations that should be expanded
for reference types, similar to how it's already handled for scalar
types (i32, i64, f32, f64). This allows the SELECT_CC to be lowered to a
SELECT node, which already has instruction patterns defined in
WebAssemblyInstrRef.td.
Since #182059 there is only one case in which these functions return -1,
so callers no longer need to distinguish between (int64_t)-1 and
(uint32_t)-1, so we can go back to a 32-bit return value like it was
before #180954.
The failures should have been resolved with #180290 (which also added
WebAssembly tests).
This reverts commit 811fb223af2b3e2d68c99b346f4b75dcf3de3417.
This checks every user function of `setjmp` or `longjmp` and if any of
them does not have `+exception-handling` target feature, errors out.
Hopefully this gives a clearer error message to the users in case they
do not provide consistent SjLj flags at compile time vs. link time.
Closes#178135 and closes
https://github.com/emscripten-core/emscripten/issues/26165.
This change adds initial support to libObject for reading compact
imports and support for writing compact imports in the linker.
There is minimal testing here since to tools like lllvm-readobj, and
obj2yaml don't currently report compact imports any differently.
See https://github.com/WebAssembly/compact-import-section
We don't currently support common symbols for Wasm, and we currently
emit a generic error with a backtrace. Instead, don't crash, and report
the names of the offending symbols.
Before, Wasm FastISel treated all indirect calls the same, causing
miscompilations at O0 when trying to call a funcref (`call ptr
addrspace(20)`), as it would treat the funcref as a normal `ptr`
This adds a check so it falls back to ISelDAG when encountering calls
outside addrspace 0 (which covers direct calls and indirect calls
through normal function pointers).
Related: #140933
Fixes a crash during type legalization by allowing
ISD::ANY_EXTEND_VECTOR_INREG to fall back to default expansion instead
of hitting llvm_unreachable.
Fixed: #177209
Custom lower FMINNUM, FMINIMUMNUM, FMAXNUM and FMAXIMUMNUM to generate
relaxed_min and relaxed_max when the inputs cannot be NaN or signed
zero.
Tablegen patterns have also been modified to check the above conditions
when trying to match relaxed min/max using the pmin/pmax pattern.
This commit introduces the VectorInstrContext (VIC) infrastructure to
improve cost estimates for insert/extracts based on the context
instruction in which the insert/extract is used.
This is similar to CastContextHint, and allows providing context on how
the insert/extract is going to be used before creating IR. This is
useful in the LoopVectorizer, where costs need to estimated before
creating IR.
The new hint currently only replaces an existing check in AArch64,
but new uses will be introduced in follow-ups, including
https://github.com/llvm/llvm-project/pull/177201.
PR: https://github.com/llvm/llvm-project/pull/175982
The WebAssemblyTargetMachine constructor had an ordering issue where
initAsmInfo() was called before basicCheckForEHAndSjLj(). This caused
problems in incremental compilation scenarios where:
1. `initAsmInfo()` sets `MCAsmInfo` exception type based on
`Options.ExceptionModel`
2. But `Options.ExceptionModel` might still be None at this point
3. `basicCheckForEHAndSjLj()` runs later and updates
`Options.ExceptionModel`
based on command-line flags like `-wasm-enable-eh`
4. `MCAsmInfo` retains the incorrect exception type (`None` instead of
`Wasm`)
5. This prevents WebAssembly exception handling passes from running
The fix swaps the order so basicCheckForEHAndSjLj() runs first to
establish the correct exception model before initAsmInfo() configures
MCAsmInfo based on that model.
This enables WebAssembly exception handling to work correctly in
clang-repl and other incremental compilation scenarios.
Returns uint64_t to simplify callers. The goal is eventually replace
getValueType with this query, which should return the known minimum
reference-able size, as provided (instead of a Type) during create.
Additionally the common isSized query would be replaced with an
isExactKnownSize query to test if that size is an exact definition.
Query libcalls through analysis instead of the TargetLowering
copy. This could be nicer by parsing the libcall name and checking
against the LibcallImpl, but that's probably slower for such a specific
case (alternatively, ExternalSymbol should support encoding as a
LibcallImpl).
Currently the WebAssembly backend crashes when trying to lower some
extract.last.active intrinsic calls. Mark their cost as invalid
temporarily, to avoid them being introduced by the loop
vectorizer after 2abd6d6d7ac (#158088).
This Change is to prepare to make RegState into an enum class. It:
- Updates documentation to match the order in the code.
- Brings the `get<>RegState` functions together and makes them
`constexpr`.
- Adopts the `get<>RegState` where RegStates were being chosen with
ternary operators in backend code.
- Introduces `hasRegState` to make querying RegState easier once it is
an enum class.
- Adopts `hasRegState` where equivalent was done with bitwise
arithmetic.
- Introduces `RegState::NoFlags`, which will be used for the lack of
flags.
- Documents that `0x1` is a reserved flag value used to detect if
someone is passing `true` instead of flags (due to implicit bool to
unsigned conversions).
- Updates two calls to `MachineInstrBuilder::addReg` which were passing
`false` to the flags operand, to no longer pass a value.
- Documents that `getRegState` seems to have forgotten a call to
`getEarlyClobberRegState`.
This PR relands llvm/llvm-project#176091 (commit
1d616cdca3aba9d22f120888bb6b09b75ca90b92) which was reverted in
llvm/llvm-project#176190 (commit
6309cd8668fc2ae589f156b23f86821f4ce5b7ea).
Splits out change from https://github.com/llvm/llvm-project/pull/176015
Changes shouldExpandAtomicRMWInIR to take a constant argument: This is
to allow some other TargetLowering constant-argument functions to call
it. This change touches several backends. An alternative solution
exists, but to me, this seems the "right" way.
Reverts llvm/llvm-project#176091
Reverting because some compilers were erroring on the call to
`Reg.isReg()` (which is not `constexpr`) in a `constexpr` function.
This Change is to prepare to make RegState into an enum class. It:
- Updates documentation to match the order in the code.
- Brings the `get<>RegState` functions together and makes them
`constexpr`.
- Adopts the `get<>RegState` where RegStates were being chosen with
ternary operators in backend code.
- Introduces `hasRegState` to make querying RegState easier once it is
an enum class.
- Adopts `hasRegState` where equivalent was done with bitwise
arithmetic.
- Introduces `RegState::NoFlags`, which will be used for the lack of
flags.
- Documents that `0x1` is a reserved flag value used to detect if
someone is passing `true` instead of flags (due to implicit bool to
unsigned conversions).
- Updates two calls to `MachineInstrBuilder::addReg` which were passing
`false` to the flags operand, to no longer pass a value.
- Documents that `getRegState` seems to have forgotten a call to
`getEarlyClobberRegState`.
Corrected various spelling mistakes such as 'occurred', 'receiver',
'initialized', 'length', and others in comments, variable names,
function names, and documentation throughout the project. These
changes improve code readability and maintain consistency in naming
and documentation.
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
We've already optimised these, so update the cost model to reflect it.
And skip the isBeforeLegalize check when lowering i8 muls, because it
then misses the cases where, say v32i8, has been type legalised into 2x
v16i8.
Also explicitly disable memory interleaving for any factor other than
two or four.
The default `f16` lowering has some issues that result in incorrect
float behavior, so over time most targets have switched to use
`softPromoteHalfType`. Swap to soft promotion by default and add
overrides for SystemZ and AMDGPU, which are the two remaining backends
that still depend on this behavior.
All basic `f16` op tests now pass on all remaining experimental arches.
Fixes: https://github.com/llvm/llvm-project/issues/97981
Fixes: https://github.com/llvm/llvm-project/issues/97975
Commit
6ad41bcc49
changed how frem is expanded during legalization and it
broke WebAssembly but we were missing test coverage. We want to maintain
our previous behavior of unrolling vectors and using a libcall to
implement scalar frem. I'm not sure why this now has to be different
(in ISelLowering) from other libcalls like fsin which work the same way
in the end, but this code does accurately describe what we want.
Fixes: https://github.com/emscripten-core/emscripten/issues/25991
The default `half` legalization, which Wasm currently uses, does not
respect IEEE conventions: for example, casting to bits may invoke a lossy
libcall, meaning soft float operations cannot be correctly implemented.
Change to the soft promotion legalization which passes `f16` as an `i16`
and treats each `half` operation as an individual
f16->f32->libcall->f32->f16 sequence.
Of note in the test updates are that `from_bits` and `to_bits` are now
libcall-free, and that chained operations now round back to `f16` after
each step.
Fixes the wasm portion of
https://github.com/llvm/llvm-project/issues/97981
Fixes the wasm portion of
https://github.com/llvm/llvm-project/issues/97975
Fixes: https://github.com/llvm/llvm-project/issues/96437
Fixes: https://github.com/llvm/llvm-project/issues/96438
Resolves#169058.
This adds ~~an InstCombine pass~~ a TTI hook to the WebAssembly backend
that folds `i8x16.swizzle` and `i8x16.relaxed.swizzle` operations to
`shufflevector` operations if their mask operands are constant.
This is mainly useful for abstractions over the raw intrinsics--for
instance, in architecture-generic SIMD code that may not be able to
expose the constant shuffles due to type system limitations.
I took most of this from the x86 backend (in particular,
`simplifyX86vpermilvar` in `X86InstCombineIntrinsic`), and adapted it
for the WebAssembly backend. There wasn't any previous
`instCombineIntrinsic` method on the WebAssembly `TargetTransformInfo`,
so I added it. Right now, this swizzle optimization is the only one it
performs.
As I noted in the transform itself, the "relaxed" swizzle actually has
stricter preconditions than the non-relaxed one. If a non-negative but
still out-of-bounds index is provided, the "relaxed" swizzle can choose
between returning 0 and the lane at the index modulo 16. However, it
must make the same choice every time, and we don't know which choice the
runtime will make, so we can't constant-fold it.
The regression tests were mostly generated by Claude and adapted a bit
by me (I tried to follow the [InstCombine contributor
guide](https://llvm.org/docs/InstCombineContributorGuide.html#tests)).
There was previously no WebAssembly subdirectory within the InstCombine
tests, so I created that too; as of now, the swizzle fold test is the
only file in it. Everything else was written by myself (well, partly
copy-pasted from the x86 backend).
I'm not sure how to write an Alive2 test for this; I can't find any
examples where the input is an arbitrary constant.
This patch extends the `load_zero` pattern matching to
support floating-point vector types (`v4f32` and `v2f64`).
Previously, the optimization to generate `v128.load32_zero` and
`v128.load64_zero` was only enabled for integer types
(`v4i32` and `v2i64`). This change adds the necessary TableGen
patterns to correctly match scalar floating-point loads inserted
into zero-initialized vectors.
Fixes https://github.com/llvm/llvm-project/issues/172647
Currently, MC assumes that all `target-feature` flag attributes are well
formed and will crash otherwise. This change handles those cases more
gracefully.
The existing condition for checking whether or not to expand an frem
instruction in expand-fp is not sufficiently precise.
The expansion on other targets than AMDGPU - which is the only intended
user right now - is only prevented due to the interaction with the
MaxLegalFpConvertBitWidth check. Relying on this is conceptually wrong
and limits the use of the pass for other targets and further expansions
(e.g. merging with the similar ExpandLargeDivRem pass).
Change the expansion criterion to always expand frem of a given type
for targets that use "Expand" as the legalization action for the
underlying scalar type and use this to exit the pass early for targets
which do not require any expansions. This requires to change the
frem legalization action for all targets which do not want frem to
be expanded in this pass from "Expand" to "LibCall".
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
