This adds a reduced test case for the regression seen in x264 with #83035.
If the intermediate concatenating shuffles are large enough then the
splitting combine will prevent the strided load combine which is
preferable.
this implements part 1 of 2 for #83626
- `CGBuiltin.cpp` - modified to have seperate cases for signed and
unsigned integers.
- `SemaChecking.cpp` - modified to prevent the generation of a double
dot product intrinsic if the builtin were to be called directly.
- `IntrinsicsDirectX.td` creation of the signed and unsigned dot
intrinsics needed for instruction expansion.
- `DXILIntrinsicExpansion.cpp` - handle instruction expansion cases for
integer dot product.
This is re-working of #74460, which adds a soft-float ABI for AArch64.
That was reverted because it causes errors when building the linux and
fuchsia kernels.
The problem is that GCC's implementation of the ABI compatibility checks
when using the hard-float ABI on a target without FP registers does it's
checks after optimisation. The previous version of this patch reported
errors for all uses of floating-point types, which is stricter than what
GCC does in practice.
This changes two things compared to the first version:
* Only check the types of function arguments and returns, not the types
of other values. This is more relaxed than GCC, while still guaranteeing
ABI compatibility.
* Move the check from Sema to CodeGen, so that inline functions are only
checked if they are actually used. There are some cases in the linux
kernel which depend on this behaviour of GCC.
We use the VSCBIQ/VSBIQ/VSBCBIQ family of instructions to implement
USUBO/USUBO_CARRY for the i128 data type. However, these instructions
use an inverted sense of the borrow indication flag (a value of 1
indicates *no* borrow, while a value of 0 indicated borrow). This
does not match the semantics of the boolean "overflow" flag of the
USUBO/USUBO_CARRY ISD nodes.
Fix this by generating code to explicitly invert the flag. These
cancel out of the result of USUBO feeds into an USUBO_CARRY.
To avoid unnecessary zero-extend operations, also improve the
DAGCombine handling of ZERO_EXTEND to optimize (zext (xor (trunc)))
sequences where appropriate.
Fixes: https://github.com/llvm/llvm-project/issues/83268
This reverts commit c59129a7c79448837d665de8f2743ad4b14666f6.
This causes regressions in some x264 workloads like pixel_var_8x8 due to it
interfering with the strided load combine. Reverting so I can try to rework
it as a lowering instead.
Added common check for DPP and Iterative strategies for uniform value
case since optimization applied is same.
Authored-by: Pravin Jagtap <Pravin.Jagtap@amd.com>
Update PromoteAllocaToVector so it considers the whole function before promoting allocas.
Allocas are scored & sorted so the highest value ones are seen first. The budget is now per function instead of per alloca.
Passed internal performance testing.
We use the VSCBIQ/VSBIQ/VSBCBIQ family of instructions to implement
USUBO/USUBO_CARRY for the i128 data type. However, these instructions
use an inverted sense of the borrow indication flag (a value of 1
indicates *no* borrow, while a value of 0 indicated borrow). This
does not match the semantics of the boolean "overflow" flag of the
USUBO/USUBO_CARRY ISD nodes.
Fix this by generating code to explicitly invert the flag. These
cancel out of the result of USUBO feeds into an USUBO_CARRY.
To avoid unnecessary zero-extend operations, also improve the
DAGCombine handling of ZERO_EXTEND to optimize (zext (xor (trunc)))
sequences where appropriate.
Fixes: https://github.com/llvm/llvm-project/issues/83268
- Instead of lowering float/double ISD::ATOMIC_LOAD / ISD::ATOMIC_STORE
nodes to regular LOAD/STORE nodes, make them legal and select those nodes
properly instead. This avoids exposing them to the DAGCombiner.
- AtomicExpand pass no longer casts float/double atomic load/stores to integer
(FP128 is still casted).
This is an alternative to #85610, that moreElement's small G_TRUNC
vectors to widen the vectors. It needs to disable one of the existing
Unmerge(Trunc(..)) combines, and some of the code is not as optimal as
it could be. I believe with some extra optimizations it could look
better (I was thinking combining trunc(buildvector) -> buildvector and
possibly improving buildvector lowering by generating
insert_vector_element earlier).
- Use computeMaxCallFrameSize() in PEI::calculateCallFrameInfo() instead of duplicating the code.
- Set AdjustsStack in FinalizeISel instead of in computeMaxCallFrameSize().
This PR:
* adds Lifetime intrinsics/instructions
* fixes how the binary header is emitted (correct version and better
approximation of Bound)
* add validation into more test cases
For each call that changes the streaming-mode ISel inserts a
COALESCER_BARRIER node for the FP and (non-scalable) vector arguments to
the callee.
When calling a non-streaming function from a streaming-compatible
function, it's not required to have +sme (in case the SME code-path is
not actually executed at runtime). The patterns to match the
COALESCER_BARRIER however were still predicated with `HasSME`, which is
incorrect. This patch tries to fix that.
This change allows us to use `--lto-partitions` in some cases (not at
all guaranteed it works perfectly), as LDS is lowered before the module
is split for parallel codegen.
We must run LowerLDS before splitting modules as it needs to see all
callers of functions with LDS to properly lower them.
rldimi is 64-bit instruction, so the corresponding builtin should not
be available in 32-bit mode. Rotate amount should be in range and
cases when mask is zero needs special handling.
This change also swaps the first and second operands of rldimi/rlwimi
to match previous behavior. For masks not ending at bit 63-SH,
rotation will be inserted before rldimi.
[GlobalISel] Implement convergence control tokens and intrinsics in GMIR
In the IR translator, convert the LLVM token type to LLT::token(), which is an
alias for the s0 type. These show up as implicit uses on convergent operations.
Differential Revision: https://reviews.llvm.org/D158147
This patch fixes an issue with the legalization of G_LOAD where the
presence of .lowerIfMemSizeNotByteSizePow2 before .clampMaxNumElements
was causing issues for vectors which matched that condition.
Such vectors would be lowered into per-element loads instead of being
split up into 128-bit chunks.
Add DXIL module flag bit offset for SHADER_FEATURE_FLAG.
Added DXIL_MODULE_FLAG for DXIL module flag which does not have feature flag.
Use DXILModuleFlags for ComputedShaderFlags instead of
ShaderFeatureFlags.
ComputedShaderFlags::getFeatureFlags() was added to get FeatureFlags.
Rename DXContainerGlobals::getShaderFlags to DXContainerGlobals::getFeatureFlags.
Fixes#57925
This patch makes NEON-enabled AArch64 backend expand the `sin, cos, pow,
log, log2, log10, exp, exp2, exp10` intrinsics for `v1f64` data type,
all of which caused selection failure before this patch.
Fixes https://github.com/llvm/llvm-project/issues/83729
Exploit the per global code model attribute on AIX. On AIX we need to
update both the code sequence used to access the global (either 1 or 2
instructions for small and large code model respectively) and the
storage mapping class that we emit the toc entry.
---------
Co-authored-by: Amy Kwan <akwan0907@gmail.com>
`clang -c -masm=intel` compiling a source file with file scope basic asm
incorrectly uses the AT&T dialect.
```
% cat a.c
asm("mov rax, rax");
% clang a.c -c -masm=intel
<inline asm>:1:1: error: unknown use of instruction mnemonic without a size suffix
mov rax, rax
^
```
Fix this by setting the assembler dialect from the MCAsmInfo object.
Note: `clang -c -flto -masm=intel a.c` still fails because of
https://reviews.llvm.org/D82862 for #34830: it tried to support AT&T
syntax for clang-cl, but the forced AT&T syntax is not compatible with
intended Intel syntax.
Pull Request: https://github.com/llvm/llvm-project/pull/85367
This patch adds support for the @llvm.experimental.vector.{interleave2,
deinterleave2} intrinsics to IRTranslator for fixed-width vector types.
They are lowered to vector shuffles, in roughly the same manner as
SelectionDAG.
Fixes casts between double/float/half and i128. The pass seems to be
broken for bfloat though. I also believe we could have a better implementation
which attempts to make use the native 32-bit conversion instructions like
the 64-bit expansion does.
At the moment MachineLoopInfo has a very simple way to determine if a
use of a physical register will be invariant: if it is not a constant
value or if it's not an ignorable use, then it's not considered
invariant.
From a compile-time performance perspective this makes a lot of sense,
but it limits code that uses implicit physical registers from being
hoisted until the later MachineLICM pass (after register allocation),
which has a lot fewer opportunities to hoist.
For AArch64 SME we use an implicit physical register ($vg) to avoid
rematerialization beyond certain instructions. Doing this led to
regressions because simple expressions were no longer hoisted by Early
MachineLICM.
This patch adds some extra checks to 'isLoopInvariant' to see if any of
the defs are found in the loop. If not, we can considered it loop
invariant.
We expect the impact on compile-time to be negligible because there is
an incentive for users to reduce the need for the smstart/smstop
instructions that define $vg. In either case, we've put the
functionality under a target interface to limit this only to specific
registers.
There are a few places where `arena` name is used for pointers in
non-zero address space in BPF backend, rename these to use a more
generic `address_space`:
- macro `__BPF_FEATURE_ARENA_CAST` -> `__BPF_FEATURE_ADDR_SPACE_CAST
- name for arena global variables section `.arena.N` ->
`.addr_space.N`
This change implements lowering for #70076, #70100, #70072, & #70102
`CGBuiltin.cpp` - - simplify `lerp` intrinsic
`IntrinsicsDirectX.td` - simplify `lerp` intrinsic
`SemaChecking.cpp` - remove unnecessary check
`DXILIntrinsicExpansion.*` - add intrinsic to instruction expansion
cases
`DXILOpLowering.cpp` - make sure `DXILIntrinsicExpansion` happens first
`DirectX.h` - changes to support new pass
`DirectXTargetMachine.cpp` - changes to support new pass
Why `any`, and `lerp` as instruction expansion just for DXIL?
- SPIR-V there is an
[OpAny](https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#OpAny)
- SPIR-V has a GLSL lerp extension via
[Fmix](https://registry.khronos.org/SPIR-V/specs/1.0/GLSL.std.450.html#FMix)
Why `exp` instruction expansion?
- We have an `exp2` opcode and `exp` reuses that opcode. So instruction
expansion is a convenient way to do preprocessing.
- Further SPIR-V has a GLSL exp extension via
[Exp](https://registry.khronos.org/SPIR-V/specs/1.0/GLSL.std.450.html#Exp)
and
[Exp2](https://registry.khronos.org/SPIR-V/specs/1.0/GLSL.std.450.html#Exp2)
Why `rcp` as instruction expansion?
This one is a bit of the odd man out and might have to move to
`cgbuiltins` when we better understand SPIRV requirements. However I
included it because it seems like [fast math mode has an AllowRecip
flag](https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#_fp_fast_math_mode)
which lets you compute the reciprocal without performing the division.
We don't have that in DXIL so thought to include it.
This change implements #70074
- `hlsl_intrinsics.h` - add the `rsqrt` api
- `DXIL.td` add the llvm intrinsic to DXIL op lowering map.
- `Builtins.td` - add an hlsl builtin for rsqrt.
- `CGBuiltin.cpp` add the ir generation for the rsqrt intrinsic.
- `SemaChecking.cpp` - reuse the one arg float only checks.
- `IntrinsicsDirectX.td` -add an `rsqrt` intrinsic.
