The test case generates a extract_subvector(index) leading into a mul.
Make sure we don't try and treat the scalable vector extract as a 128bit
vector in the mull combine.
Fixes#168912
A funnel shift with the first two operands is a rotate. AArch64 has
scalar instructions for ror so can handle a ROTR by a constant or
variant amount cheaply. A ROTL is a ROTR by the opposite amount, which
for constants is cheap and for variable shifts requires an extra neg.
Removes DependenceInfo::getRuntimeAssumptions(), DependenceInfo::Assumptions,
and the print of "Runtime Assumptions:". The runtime assumptions are still
properly attached to each Dependence result and printed as part of the
per-dependence output.
Update the logic in narrowToSingleScalar to allow narrowing even if not
all users use scalars, if at least one of the operands already needs
broadcasting.
In that case, there won't be any additional broadcasts introduced. This
should allow removing the special handling for stores, which can
introduce additional broadcasts currently.
Fixes https://github.com/llvm/llvm-project/issues/169668.
PR: https://github.com/llvm/llvm-project/pull/168246
This patch adds SVE support for the `masked.compressstore` intrinsic via
the existing `VECTOR_COMPRESS` lowering and compressing the store mask
via `VECREDUCE_ADD`.
Currently, only `nxv4[i32|f32]` and `nxv2[i64|f64]` are directly
supported, with other types promoted to these, where possible.
This is done in preparation for LV support of this intrinsic, which is
currently being worked on in #140723.
clang-riscv-gauntlet buildbot was previously failing. Conservatively
bail out if not handling a branch when looking for floating-point
recurrences, we may wish to support select and branch in the future.
Fixes: https://lab.llvm.org/buildbot/#/builders/210/builds/5908.
`handleFloatingPointIV` is now abstracted out into different routines,
particularly:
- `maybeFloatingPointRecurrence` which establishes whether we handle a
floating-point iv recurrence;
- `tryConvertToIntegerIV` which attempts to convert the fp start, step
and exit values into integer ones;
- `canonicalizeToIntegerIV` which rewrites the recurrence.
Minor opportunity to modernize the code where possible.
This commit improves the lowering of vectors of fp16 when truncating and
(previously) extending. Truncating has to be handled in a specific way
to avoid double rounding.
When SIMULATED_TRAP is at the end of a block with no successors,
insertSimulatedTrap incorrectly returns the original MBB despite adding
HaltLoopBB to the CFG.
EmitInstrWithCustomInserter detects CFG changes by comparing the
returned MBB with the original. When they match, it assumes no
modification occurred and skips MachineLoopInfo invalidation. This
causes stale loop information in subsequent passes, particularly when
using the NPM which relies on accurate invalidation signals.
Fix: Return HaltLoopBB to properly signal the CFG modification.
Most predicable Thumb instructions do not encode the predicate operand,
but rather take it from an enclosing IT block.
Add `bits<0> p` to the encoding of these instructions to make the
predicate operand decodable by the generated code.
The previous approach was to analyze an instruction after it has been
decoded and add missing predicate operand if necessary. The
post-decoding pass is still required to check predicate applicability
and advance IT block state, but it no longer modifies a decoded
instruction.
Some of the custom decoder methods have become redundant and can be
removed in the future, delegating the decoding task to TableGen-erated
decoder.
Pull Request: https://github.com/llvm/llvm-project/pull/156540
In #160470, there is a discussion about the possibility to explored a
general approach for handling memory intrinsics.
API changes:
- Remove getMaskedMemoryOpCost, getGatherScatterOpCost,
getExpandCompressMemoryOpCost, getStridedMemoryOpCost from
Analysis/TargetTransformInfo.
- Add getMemIntrinsicInstrCost.
In BasicTTIImpl, map intrinsic IDs to existing target implementation
until the legacy TTI hooks are retired.
- masked_load/store → getMaskedMemoryOpCost
- masked_/vp_gather/scatter → getGatherScatterOpCost
- masked_expandload/compressstore → getExpandCompressMemoryOpCost
- experimental_vp_strided_{load,store} → getStridedMemoryOpCost
TODO: add support for vp_load_ff.
No functional change intended; costs continue to route to the same
target-specific hooks.
- Following #168029. This is a step toward a unified interface for
masked/gather-scatter/strided/expand-compress cost modeling.
- Replace the ad-hoc parameter list with a single attributes object.
API change:
```
- InstructionCost getExpandCompressMemoryOpCost(Opcode, DataTy,
- VariableMask, Alignment,
- CostKind, Inst);
+ InstructionCost getExpandCompressMemoryOpCost(MemIntrinsicCostAttributes,
+ CostKind);
```
Notes:
- NFCI intended: callers populate MemIntrinsicCostAttributes with same
information as before.
In some case, VPWidenPointerInductions become only used by scalars after
legalizeAndOptimizationInducftions was already run, for example due to
some VPlan optimizations.
Move the code to scalarize VPWidenPointerInductions to a helper and use
it if needed.
This fixes a crash after #148274 in the added test case.
Fixes https://github.com/llvm/llvm-project/issues/169780
Currently, the following two snippets get treated very differently from
each other (https://godbolt.org/z/rYGj9TGz6):
```LLVM
define <8 x i8> @foo(<8 x i8> %x, <8 x i8> %y) local_unnamed_addr #0 {
entry:
%0 = shufflevector <8 x i8> %x, <8 x i8> %y, <8 x i32>
<i32 0, i32 8, i32 1, i32 9, i32 2, i32 10, i32 3, i32 11>
ret <8 x i8> %0
}
define <8 x i8> @bar(<8 x i8> %x, <8 x i8> %y) local_unnamed_addr #0 {
entry:
%0 = shufflevector <8 x i8> %x, <8 x i8> %y, <8 x i32>
<i32 8, i32 0, i32 9, i32 1, i32 10, i32 2, i32 11, i32 3>
ret <8 x i8> %0
}
```
```
foo: // @foo
zip1 v0.8b, v0.8b, v1.8b
ret
.LCPI1_0:
.byte 8 // 0x8
.byte 0 // 0x0
.byte 9 // 0x9
.byte 1 // 0x1
.byte 10 // 0xa
.byte 2 // 0x2
.byte 11 // 0xb
.byte 3 // 0x3
bar: // @bar
adrp x8, .LCPI1_0
mov v0.d[1], v1.d[0]
ldr d1, [x8, :lo12:.LCPI1_0]
tbl v0.8b, { v0.16b }, v1.8b
ret
```
The reason is that `isZIPMask` does not recognise the pattern when the
operands are flipped.
This PR fixes `isZIPMask` so that both `foo` and `bar` get compiled as
expected:
```
foo: // @foo
zip1 v0.8b, v0.8b, v1.8b
ret
bar: // @bar
zip1 v0.8b, v1.8b, v0.8b
ret
```
I intend to open a similar follow-up PR for `isTRNMask`, which seems to
have the same problem.
I noticed this while working on
https://github.com/llvm/llvm-project/issues/137447, though the change
does not on itself fix that issue.
AMD uses the translator to recover LLVM-IR from SPIRV.
Currently, the translator doesn't implement the
`SPV_KHR_float_controls2` extension (I'm working on it).
If this extension is used by the SPIRV module, we cannot translate it
back to LLVM-IR.
I'm working on the extension, but in the meantime, lets just disable it
when the target triple's vendor is `amd`.
This patch moves the validation logic of delinearization results from DA
to Delinearization. Also call it in `printDelinearization` to test its
behavior. The motivation is as follows:
- Almost the same code exists in `tryDelinearizeFixedSize` and
`tryDelinearizeParametricSize`. Consolidating it in Delinearization
avoids code duplication.
- Currently this validation logic is not well tested. Moving it to
Delinearization allows us to write regression tests easily.
This patch changes the test outputs and debug messages, but otherwise
NFCI.
Remove Subtarget uses from ARMAsmPrinter, making use of TargetMachine
where applicable and getting the Subtarget from the MF where not. Some
of the `if() llvm_unreachable` have been replaced by `asserts`.
Compiler-rt internal feature table is synced with the one in libgcc
(common/config/i386/i386-cpuinfo.h).
LLVM internal feature table is refactored to include a field ABI_VALUE,
so we won't be relying on ordering to keep the values correct. The table
is also synced to the one in compiler-rt.
`dist_schedule` was previously supported in Flang/Clang but was not
implemented in MLIR, instead a user would get a "not yet implemented"
error. This patch adds support for the `dist_schedule` clause to be
lowered to LLVM IR when used in an `omp.distribute` or `omp.wsloop`
section.
There has needed to be some rework required to ensure that MLIR/LLVM
emits the correct Schedule Type for the clause, as it uses a different
schedule type to other OpenMP directives/clauses in the runtime library.
This patch also ensures that when using dist_schedule or a chunked
schedule clause, the correct llvm loop parallel accesses details are
added.
A barrier will pause execution until all threads reach it. If some go to
a different barrier then we deadlock. This manifests in that the
finalization callback must only be run once. Fix by ensuring we always
go through the same finalization block whether the thread in cancelled
or not and no matter which cancellation point causes the cancellation.
The old callback only affected PARALLEL, so it has been moved into the
code generating PARALLEL. For this reason, we don't need similar changes
for other cancellable constructs. We need to create the barrier on the
shared exit from the outlined function instead of only on the cancelled
branch to make sure that threads exiting normally (without cancellation)
meet the same barriers as those which were cancelled. For example,
previously we might have generated code like
```
...
%ret = call i32 @__kmpc_cancel(...)
%cond = icmp eq i32 %ret, 0
br i1 %cond, label %continue, label %cancel
continue:
// do the rest of the callback, eventually branching to %fini
br label %fini
cancel:
// Populated by the callback:
// unsafe: if any thread makes it to the end without being cancelled
// it won't reach this barrier and then the program will deadlock
%unused = call i32 @__kmpc_cancel_barrier(...)
br label %fini
fini:
// run destructors etc
ret
```
In the new version the barrier is moved into fini. I generate it *after*
the destructors because the standard describes the barrier as occurring
after the end of the parallel region.
```
...
%ret = call i32 @__kmpc_cancel(...)
%cond = icmp eq i32 %ret, 0
br i1 %cond, label %continue, label %cancel
continue:
// do the rest of the callback, eventually branching to %fini
br label %fini
cancel:
br label %fini
fini:
// run destructors etc
// safe so long as every exit from the function happens via this block:
%unused = call i32 @__kmpc_cancel_barrier(...)
ret
```
To achieve this, the barrier is now generated alongside the finalization
code instead of in the callback. This is the reason for the changes to
the unit test.
I'm unsure if I should keep the incorrect barrier generation callback
only on the cancellation branch in clang with the OMPIRBuilder backend
because that would match clang's ordinary codegen. Right now I have
opted to remove it entirely because it is a deadlock waiting to happen.
Adding support for serializing the ada entry flags helps with mir based
test cases. Without this change, the flags are simple displayed as being
"unkmown".
Remove explicit VT numbers from ValueTypes.td so that patches that add a
new VT do not have to renumber the entire file.
In TableGen VTs are now identified by ValueType.LLVMName instead of
ValueType.Value. This is important for target-defined types (typically
based on PtrValueType) which are not mentioned in ValueTypes.td itself.
At present, the shrink wrapping pass misses opportunities to shrink wrap
in the presence of machine basic blocks which exit the function without
returning. Such cases arise from C++ functions like the following:
```cxx
int foo(int err, void* ptr) {
if (err == -1) {
if (ptr == nullptr) {
throw MyException("Received `nullptr`!", __FILE__, __LINE__);
}
handle(ptr);
}
return STATUS_OK;
}
```
In particular, assuming `MyException`'s constructor is not marked
`noexcept`, the above code will generate a trivial EH landing pad
calling `__cxa_free_exception()` and rethrowing the unhandled internal
exception, exiting the function without returning. As such, the shrink
wrapping pass refuses to touch the above function, spilling to the stack
on every call, even though no CSRs are clobbered on the hot path. This
patch tweaks the shrink wrapping logic to enable the pass to fire in
this and similar cases.
This patch adds recognition of high-half multiply by parts into a single
larger multiply.
Considering a multiply made up of high and low parts, we can split the
multiply into:
x * y == (xh*T + xl) * (yh*T + yl)
where `xh == x>>32` and `xl == x & 0xffffffff`. `T = 2^32`.
This expands to
xh*yh*T*T + xh*yl*T + xl*yh*T + xl*yl
which I find it helpful to be drawn as
[ xh*yh ]
[ xh*yl ]
[ xl*yh ]
[ xl*yl ]
We are looking for the "high" half, which is xh*yh + xh*yl>>32 + xl*yh>>32 +
carrys. The carry makes this difficult and there are multiple ways of
representing it. The ones we attempt to support here are:
Carry: xh*yh + carry + lowsum
carry = lowsum < xh*yl ? 0x1000000 : 0
lowsum = xh*yl + xl*yh + (xl*yl>>32)
Ladder: xh*yh + c2>>32 + c3>>32
c2 = xh*yl + (xl*yl >> 32); c3 = c2&0xffffffff + xl*yh
Carry4: xh*yh + carry + crosssum>>32 + (xl*yl + crosssum&0xffffffff) >> 32
crosssum = xh*yl + xl*yh
carry = crosssum < xh*yl ? 0x1000000 : 0
Ladder4: xh*yh + (xl*yh)>>32 + (xh*yl)>>32 + low>>32;
low = (xl*yl)>>32 + (xl*yh)&0xffffffff + (xh*yl)&0xfffffff
They all start by matching `xh*yh` + 2 or 3 other operands. The bottom of the
tree is `xh*yh`, `xh*yl`, `xl*yh` and `xl*yl`.
Based on #156879 by @c-rhodes
On RISC-V, some loops that the loop vectorizer vectorizes pre-LTO may
turn out to have the exact trip count exposed after LTO, see #164762.
If the trip count is small enough we can fold away the
@llvm.experimental.get.vector.length intrinsic based on this corollary
from the LangRef:
> If %cnt is less than or equal to %max_lanes, the return value is equal
to %cnt.
This on its own doesn't remove the @llvm.experimental.get.vector.length
in #164762 since we also need to teach computeKnownBits about
@llvm.experimental.get.vector.length and the sub recurrence, but this PR
is a starting point.
I've added this in InstCombine rather than InstSimplify since we may
need to insert a truncation (@llvm.experimental.get.vector.length can
take an i64 %cnt argument, the result is always i32).
Note that there was something similar done in VPlan in #167647 for when
the loop vectorizer knows the trip count.
Further to
https://github.com/llvm/llvm-project/pull/147134#discussion_r2337246489,
switch to use the madvise() api to page in mmap'd files and
1) All new code compiled in #if LLVM_ENABLE_THREADS is set so it can be
seen where the changes were from this PR.
2) The new PR moves to use madvise() instead of the ad-hoc page
referencing code I wrote which should avoid SIGSEGVs if the buffer is
deallocated.
3) A new property SerialBackgroundQueue().stopAllWork to be used to stop
background workers when there is no further call for them. Usually the
background "page-in" threads have completed first but it seems with this
troublesome test this is not always the case and buffers stored in the
static input file cache are being deallocated while being referenced.
---------
Co-authored-by: James Henderson <James.Henderson@sony.com>
The Language Environment (LE) reserves 128 byte for the argument area
when the optional field is not present. If the argument area is larger,
then the field must be present to guarantee that the space is reserved
on stack extension. Creating this field when alloca() is used may reduce
the needed stack space in case alloca() causes a stack extension.
Deactivation symbol operands are supported in the code generator by
building on the previously added support for IRELATIVE relocations.
Reviewers: ojhunt, fmayer, ahmedbougacha, nikic, efriedma-quic
Reviewed By: fmayer
Pull Request: https://github.com/llvm/llvm-project/pull/133537
