These are the last remaining "trivial" changes to passes that use
Instruction pointers for insertion. All of this should be NFC, it's just
changing the spelling of how we identify a position.
In one or two locations, I'm also switching uses of getNextNode etc to
using std::next with iterators. This too should be NFC.
---------
Merged by: Stephen Tozer <stephen.tozer@sony.com>
this implements part 1 of 2 for #83626
- `CGBuiltin.cpp` - modified to have seperate cases for signed and
unsigned integers.
- `SemaChecking.cpp` - modified to prevent the generation of a double
dot product intrinsic if the builtin were to be called directly.
- `IntrinsicsDirectX.td` creation of the signed and unsigned dot
intrinsics needed for instruction expansion.
- `DXILIntrinsicExpansion.cpp` - handle instruction expansion cases for
integer dot product.
Because the RemoveDIs work is putting a debug-info bit into
BasicBlock::iterator and iterators are needed for insertion, the
getAsInstruction method declaration would need to use a fully defined
instruction-iterator, which leads to a complicated
header-inclusion-order problem. Much simpler to instead just not insert,
and make it the callers problem to insert.
This is proportionate because there are only four call-sites to
getAsInstruction -- it would suck if we did this everywhere.
---------
Merged by: Stephen Tozer <stephen.tozer@sony.com>
We use the VSCBIQ/VSBIQ/VSBCBIQ family of instructions to implement
USUBO/USUBO_CARRY for the i128 data type. However, these instructions
use an inverted sense of the borrow indication flag (a value of 1
indicates *no* borrow, while a value of 0 indicated borrow). This
does not match the semantics of the boolean "overflow" flag of the
USUBO/USUBO_CARRY ISD nodes.
Fix this by generating code to explicitly invert the flag. These
cancel out of the result of USUBO feeds into an USUBO_CARRY.
To avoid unnecessary zero-extend operations, also improve the
DAGCombine handling of ZERO_EXTEND to optimize (zext (xor (trunc)))
sequences where appropriate.
Fixes: https://github.com/llvm/llvm-project/issues/83268
This reverts commit c59129a7c79448837d665de8f2743ad4b14666f6.
This causes regressions in some x264 workloads like pixel_var_8x8 due to it
interfering with the strided load combine. Reverting so I can try to rework
it as a lowering instead.
Update PromoteAllocaToVector so it considers the whole function before promoting allocas.
Allocas are scored & sorted so the highest value ones are seen first. The budget is now per function instead of per alloca.
Passed internal performance testing.
We use the VSCBIQ/VSBIQ/VSBCBIQ family of instructions to implement
USUBO/USUBO_CARRY for the i128 data type. However, these instructions
use an inverted sense of the borrow indication flag (a value of 1
indicates *no* borrow, while a value of 0 indicated borrow). This
does not match the semantics of the boolean "overflow" flag of the
USUBO/USUBO_CARRY ISD nodes.
Fix this by generating code to explicitly invert the flag. These
cancel out of the result of USUBO feeds into an USUBO_CARRY.
To avoid unnecessary zero-extend operations, also improve the
DAGCombine handling of ZERO_EXTEND to optimize (zext (xor (trunc)))
sequences where appropriate.
Fixes: https://github.com/llvm/llvm-project/issues/83268
- Instead of lowering float/double ISD::ATOMIC_LOAD / ISD::ATOMIC_STORE
nodes to regular LOAD/STORE nodes, make them legal and select those nodes
properly instead. This avoids exposing them to the DAGCombiner.
- AtomicExpand pass no longer casts float/double atomic load/stores to integer
(FP128 is still casted).
CI checks were passing in #84962 (c48d8182f172ac24244d5fb038b7ab983f67def4) but
that commit caused failures once merged due to ships passing since the
PR was not rebased on #85131. This commit fixes this problem by adding
sched resources for integer min max instructions from Zbb in P600 model.
This PR includes an initial scheduler model shows improvement on
multiple workloads over NoSchedModel and SiFive7Model for sifive-p670.
We plan on making significant changes to this model in the future so
that it is more accurate. This patch would close
https://github.com/llvm/llvm-project/pull/80612.
This is an alternative to #85610, that moreElement's small G_TRUNC
vectors to widen the vectors. It needs to disable one of the existing
Unmerge(Trunc(..)) combines, and some of the code is not as optimal as
it could be. I believe with some extra optimizations it could look
better (I was thinking combining trunc(buildvector) -> buildvector and
possibly improving buildvector lowering by generating
insert_vector_element earlier).
Cast Offset variable to int64_t type directly inside a multiplication
and function call to utilize 64-bit arithmetic.
Ensure that the multiplication will not overflow.
A static analyzer warned about this since the function expects a 64-bit
argument, but the multiplication is evaluated inside a 32-bit context.
- Use computeMaxCallFrameSize() in PEI::calculateCallFrameInfo() instead of duplicating the code.
- Set AdjustsStack in FinalizeISel instead of in computeMaxCallFrameSize().
These predicates should be copied from the corresponding Pseudo
instruction. Previously that did not work because of a problem with
setting the right predicates on the Pseudos, but #85442 fixed that.
This changs the way the assembly matcher works for Aarch32 parsing.
Previously there was a pile of hacks which dictated whether the CC,
CCOut, and VCC operands should be present which de-facto chose if the
wide/narrow (or thumb1/thumb2/arm) instruction version were chosen.
This meant much of the TableGen machinery present for the assembly
matching was effectively being bypassed and worked around.
This patch makes the CC and CCOut operands optional which allows the ASM
matcher operate as it was designed and means we can avoid doing some of
the hacks done previously. This also adds the option for the target to
allow the prioritizing the smaller instruction encodings as is required
for Aarch32.
This PR:
* adds Lifetime intrinsics/instructions
* fixes how the binary header is emitted (correct version and better
approximation of Bound)
* add validation into more test cases
For each call that changes the streaming-mode ISel inserts a
COALESCER_BARRIER node for the FP and (non-scalable) vector arguments to
the callee.
When calling a non-streaming function from a streaming-compatible
function, it's not required to have +sme (in case the SME code-path is
not actually executed at runtime). The patterns to match the
COALESCER_BARRIER however were still predicated with `HasSME`, which is
incorrect. This patch tries to fix that.
This change allows us to use `--lto-partitions` in some cases (not at
all guaranteed it works perfectly), as LDS is lowered before the module
is split for parallel codegen.
We must run LowerLDS before splitting modules as it needs to see all
callers of functions with LDS to properly lower them.
rldimi is 64-bit instruction, so the corresponding builtin should not
be available in 32-bit mode. Rotate amount should be in range and
cases when mask is zero needs special handling.
This change also swaps the first and second operands of rldimi/rlwimi
to match previous behavior. For masks not ending at bit 63-SH,
rotation will be inserted before rldimi.
[GlobalISel] Implement convergence control tokens and intrinsics in GMIR
In the IR translator, convert the LLVM token type to LLT::token(), which is an
alias for the s0 type. These show up as implicit uses on convergent operations.
Differential Revision: https://reviews.llvm.org/D158147
This patch fixes an issue with the legalization of G_LOAD where the
presence of .lowerIfMemSizeNotByteSizePow2 before .clampMaxNumElements
was causing issues for vectors which matched that condition.
Such vectors would be lowered into per-element loads instead of being
split up into 128-bit chunks.
Add DXIL module flag bit offset for SHADER_FEATURE_FLAG.
Added DXIL_MODULE_FLAG for DXIL module flag which does not have feature flag.
Use DXILModuleFlags for ComputedShaderFlags instead of
ShaderFeatureFlags.
ComputedShaderFlags::getFeatureFlags() was added to get FeatureFlags.
Rename DXContainerGlobals::getShaderFlags to DXContainerGlobals::getFeatureFlags.
Fixes#57925
This is part of #70452 that changes the type used for the external
interface of MMO to LocationSize as opposed to uint64_t. This means the
constructors take LocationSize, and convert ~UINT64_C(0) to
LocationSize::beforeOrAfter(). The getSize methods return a
LocationSize.
This allows us to be more precise with unknown sizes, not accidentally
treating them as unsigned values, and in the future should allow us to
add proper scalable vector support but none of that is included in this
patch. It should mostly be an NFC.
Global ISel is still expected to use the underlying LLT as it needs, and
are not expected to see unknown sizes for generic operations. Most of
the changes are hopefully fairly mechanical, adding a lot of getValue()
calls and protecting them with hasValue() where needed.
Definitions like this did not work as intended:
let is_flat_scratch = 1 in {
let SubtargetPredicate = HasFlatScratchSVSMode in
def _SVS : FLAT_Scratch_Load_Pseudo<opName, regClass, HasTiedOutput, 1,
1>,
FlatScratchInst<opName, "SVS">;
let SubtargetPredicate = HasFlatScratchSTMode in
def _ST : FLAT_Scratch_Load_Pseudo<opName, regClass, HasTiedOutput, 0,
0, 0>,
FlatScratchInst<opName, "ST">;
}
They tried to override SubtargetPredicate, but then it was overridden
again (back to its default value) by setting is_flat_scratch, which
caused SubtargetPredicate to be recalculated in the base class. (This
patch also removes some overrides of SubtargetPredicate that are
redundant due to being recalculated in the base class.)
Fix this by pushing overrides of is_flat_scratch and is_flat_global "in"
as far as possible. This has the added benefit that there is no need to
override them around groups of Pseudo definitions like this:
let is_flat_global = 1 in {
defm GLOBAL_ATOMIC_CMPSWAP : FLAT_Global_Atomic_Pseudo
<"global_atomic_cmpswap",
VGPR_32, i32, v2i32, VReg_64>;
...
}
which are plainly Global instructions anyway.
Verified by inspecting the output of TableGen. It seems to be NFC in
practice.
This patch makes NEON-enabled AArch64 backend expand the `sin, cos, pow,
log, log2, log10, exp, exp2, exp10` intrinsics for `v1f64` data type,
all of which caused selection failure before this patch.
Fixes https://github.com/llvm/llvm-project/issues/83729
Exploit the per global code model attribute on AIX. On AIX we need to
update both the code sequence used to access the global (either 1 or 2
instructions for small and large code model respectively) and the
storage mapping class that we emit the toc entry.
---------
Co-authored-by: Amy Kwan <akwan0907@gmail.com>
On GFX10 only, saddr = EXEC_HI (instead of NULL) is use to distinguish
ST mode from other SCRATCH addressing modes. Handle this when defining
the saddr field instead of overriding it in subclasses.
