This patch improves the codegen for saddo on i32 and i64 in both 32-bit
and 64-bit modes by custom lowering. It implements signed-add overflow
detection using the `(x eqv y) & (sum xor x)`bit-level sequence.
The patch add 16 bytes load size for function
PPCTTIImpl::enableMemCmpExpansion and fold i128 equality/inequality
compares of two loads into a vectorized compare using vcmpequb.p when
Altivec is available.
Rationale:
A scalar i128 SETCC (eq/ne) normally lowers to multiple scalar ops. On
VSX-capable subtargets, we can instead reinterpret the i128 loads as
v16i8 vectors and use the Altive vcmpequb.p instruction to perform a
full 128-bit equality check in a single vector compare.
Example Result:
This transformation replaces memcmp(a, b, 16) with two vector loads and
one vector compare instruction.
the patch
[Add strictfp attribute to prevent unwanted optimizations of libm
calls](https://reviews.llvm.org/D34163)
add `I.isStrictFP()` into
```
if (!I.isNoBuiltin() && !I.isStrictFP() && !F->hasLocalLinkage() &&
F->hasName() && LibInfo->getLibFunc(*F, Func) &&
LibInfo->hasOptimizedCodeGen(Func))
```
it prevents the backend from optimizing even non-math libcalls such as
`strlen` and `memcmp` if a call has the strict floating-point attribute.
For example, it prevent converting strlen and memcmp to milicode call
__strlen and __memcmp.
sincospi/sincospif/sincospil does not appear to exist on common
targets. Darwin targets have __sincospi and __sincospif, so define
and use those implementations. I have no idea what version added
those calls, so I'm just guessing it's the same conditions as
__sincos_stret.
Most of this patch is working to preserve codegen when a vector
library is explicitly enabled. This only covers sleef and armpl,
as those are the only cases tested.
The multiple result libcalls have an aberrant process where the
legalizer looks for the scalar type's libcall in RuntimeLibcalls,
and then cross references TargetLibraryInfo to find a matching
vector call. This was unworkable in the sincospi case, since the
common case is there is no scalar call available. To preserve
codegen if the call is available, first try to match a libcall
with the vector type before falling back on the old scalar search.
Eventually all of this logic should be contained in RuntimeLibcalls,
without the link to TargetLibraryInfo. In principle we should perform
the same legalization logic as for an ordinary operation, trying
to find a matching subvector type with a libcall.
conversion of bl memmove call to milicode bl .___memmove64[PR] in
64--bit mode is broken , the patch fix the problem.
in the llvm/include/llvm/IR/RuntimeLibcalls.td, we do not need to define
the
`def ___memmove64 : RuntimeLibcallImpl<MEMCPY>` in PPC64AIXCallList
` def ___memmove32 : RuntimeLibcallImpl<MEMCPY>` in PPC32AIXCallList
since there is function
```
/// Return a function impl compatible with RTLIB::MEMCPY, or
/// RTLIB::Unsupported if fully unsupported.
RTLIB::LibcallImpl getMemcpyImpl() const {
RTLIB::LibcallImpl Memcpy = getLibcallImpl(RTLIB::MEMCPY);
if (Memcpy == RTLIB::Unsupported) {
// Fallback to memmove if memcpy isn't available.
return getLibcallImpl(RTLIB::MEMMOVE);
}
return Memcpy;
}
```
Global annotations metadata would trigger an assertion during code
emission on AIX. Filter out globals that are in the "llvm.metadata"
section before reaching the assert. Adds a test to verify the metadata
is not emitted on either ELF or XCOFF targets.
Update MMA tests to add run line for `cpu=future` to ensure MMA
functionality is not broken with the new `wacc` register classes
introduced. Previous commit have added def for using the new `wacc`
registers, this just add in testing and fixes a few patterns that was
missing .
Unused loop invariant loads were not sunk from the preheader to the exit
block, increasing live range.
This commit moves the sinkUnusedInvariant logic from indvarsimplify to
LICM also adds functionality to sink unused load that's not
clobbered by the loop body.
For an insertelt with a dynamic index, the default handling in
DAGTypeLegalizer and LegalizeDAG will reserve a stack slot for the
vector, lower the insertelt to a store, then load the modified vector
back into temporaries. The vector store and load may be legalized into a
sequence of smaller operations depending on the target.
Let V = the vector size and L = the length of a chain of insertelts with
dynamic indices. In the worse case, this chain will lower to O(VL)
operations, which can increase code size dramatically.
Instead, identify such chains, reserve one stack slot for the vector,
and lower all of the insertelts to stores at once. This requires only
O(V + L) operations. This change only affects the default lowering
behavior.
Currently it is considered suitable to lower to a bit test for a set of
switch case clusters when the the number of unique destinations
(`NumDests`) and the number of total comparisons (`NumCmps`) satisfy:
`(NumDests == 1 && NumCmps >= 3) || (NumDests == 2 && NumCmps >= 5) ||
(NumDests == 3 && NumCmps >= 6)`
However it is found for some cases on powerpc, for example, when
NumDests is 3, and the number of comparisons for each destination is all
2, it's not profitable to lower the switch to bit test. This is to add
an option to set the minimum of largest number of comparisons to use bit
test for switch lowering.
---------
Co-authored-by: Shimin Cui <scui@xlperflep9.rtp.raleigh.ibm.com>
The instruction `tlbie` changed in ISA3.0.
ISA V2.07: `tlbie RB,RS`
ISA V3.0: `tlbie RB,RS,RIC,PRS,R`, with `tlbie RB,RS` aliased to `tlbie
RB,RS,0,0,0`
The branches emitted for atomic operations after the store-conditional
are currently not hinted, even though they should be.
According to the Power10 Processor Chip User’s Manual:
` “Without static prediction, if the lock is not acquired in the first
iteration, the branch history mechanism works to update the prediction
to predict taken; that is, predict lock acquisition failure and cause
more lwarx traffic for the next iteration.”`
This patch addresses the issue by adding explicit branch hints for
atomic operations after the store-conditional.
Try to remove `UnsafeFPMath` uses in PowerPC backend. These global flags
block some improvements like
https://discourse.llvm.org/t/rfc-honor-pragmas-with-ffp-contract-fast/80797.
Remove them incrementally.
FP operations may raise exceptions are replaced by constrained
intrinsics. However, vector type is not supported by these intrinsics.
The previous [NFC
patch](https://github.com/llvm/llvm-project/pull/160476#top) addressed
only the vector type `v4i32`, this is a continuation for the previous
patch which adds the remaining 3 vector types which were left out.
This should include the following operands:
- `v2i64`: `A + vector {1, 1,}`
- `v8i16`: `A + vector {1, 1, 1, 1, 1, 1, 1, 1}`
- `v16i8`: `A + vector {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}`
---------
Co-authored-by: himadhith <himadhith.v@ibm.com>
### Optimize BUILD_VECTOR having special quadword patterns
This change optimizes `BUILD_VECTOR` operations by using the `lxvkq` or
`xxpltib + vsrq` instructions to inline constants matching specific
128-bit patterns:
- **MSB set pattern**: `0x8000_0000_0000_0000_0000_0000_0000_0000`
- **LSB set pattern**: `0x0000_0000_0000_0000_0000_0000_0000_0001`
### Implementation Details
The `lxvkq` instruction loads special quadword values into VSX
registers:
```asm
lxvkq XT, UIM
# When UIM=16: loads 0x8000_0000_0000_0000_0000_0000_0000_0000
```
The optimization reconstructs the 128-bit register pattern from
`BUILD_VECTOR` operands, accounting for target endianness. For example,
the MSB pattern can be represented as:
- **Big-Endian**: `<i64 -9223372036854775808, i64 0>`
- **Little-Endian**: `<i64 0, i64 -9223372036854775808>`
Both produce the same register value:
`0x8000_0000_0000_0000_0000_0000_0000_0000`
### MSB Pattern (`0x8000...0000`)
All vector types (`v2i64`, `v4i32`, `v8i16`, `v16i8`) generate:
```asm
lxvkq v2, 16
```
### LSB Pattern (`0x0000...0001`)
All vector types generate:
```asm
xxspltib v2, 255
vsrq v2, v2, v2
```
---------
Co-authored-by: Tony Varghese <tony.varghese@ibm.com>
This NFC patch adds a new function which aids in emitting machine
instructions for floating point vectors. This was previously not
included in the test file as it currently only checks for integer
vectors.
---------
Co-authored-by: himadhith <himadhith.v@ibm.com>
Previously if we had a subregister extract reading from a
full copy, the no-subregister incoming copy would overwrite
the DefSubReg index of the folding context.
There's one ugly rvv regression, but it's a downstream
issue of this; an unnecessary same class reg-to-reg full copy
was avoided.
This NFC patch looks to lock down the instruction generated for the
operation of `A + vector {1, 1, 1, 1}` in which the current code emits
`vspltisw`.
It can be made better with the use of a `2 cycle` instruction `xxleqv`
over the current `4 cycle vspltisw`.
---------
Co-authored-by: himadhith <himadhith.v@ibm.com>
This code was already creating HandleSDNodes to handle the case where a
node gets replaced with an equivalent node. However, the code before the
handles are created also performs RAUW operations, which can end up
CSEing and deleting nodes.
Fix this issue by moving the handle creation earlier.
Fixes https://github.com/llvm/llvm-project/issues/160040.
The result type of the vector extend intrinsics generated by the
BUILD_VECTOR lowering code should match how they are actually defined.
Currently the result type is defaulting to the operand type there. This
can conflict with calls to the same intrinsic from other paths.
AIX has "millicode" routines, which are functions loaded at boot time
into fixed addresses in kernel memory. This allows them to be customized
for the processor. The __strlen routine is a millicode implementation;
we use millicode for the strlen function instead of a library call to
improve performance.
The PowerPC changes are caused by shifts created by different IR
operations being CSEd now. This allows consecutive loads to be turned
into vectors earlier. This has effects on the ordering of other combines
and legalizations. This leads to some improvements and some regressions.
This was being used for 2 different purposes.
The TargetMachine constructor prepends +64bit based on isPPC64
triples as a mode switch. The same feature name was also explicitly
added to different processors, making it impossible to perform a pure
feature check for whether 64-bit mode is enabled ir not. i.e.,
checkFeatures("+64bit") would be true even for ppc32 triples.
The comment in tablegen suggests it's relevant to track which processors
support 64-bit mode independently of whether that's the active compile
target, so replace that with a new feature.
Pre-commit test case for exploitation of `xxsel` for ternary operations
of the pattern. This adds support for v4i32, v2i64, v16i8 and v8i16
operand types for the following patterns.
The following are the patterns involved in the change:
```
ternary(A, and(B,C), nor(B,C))
ternary(A, B, nor(B,C))
ternary(A, C, nor(B,C))
ternary(A, xor(B,C), nor(B,C))
ternary(A, not(C), nor(B,C))
ternary(A, not(B), nor(B,C))
ternary(A, nand(B,C), nor(B,C))
ternary(A, or(B,C), eqv(B,C))
ternary(A, nor(B,C), eqv(B,C))
ternary(A, not(C), eqv(B,C))
ternary(A, nand(B,C), eqv(B,C))
ternary(A, and(B,C), not(C))
ternary(A, B, not(C))
ternary(A, xor(B,C), not(C))
ternary(A, or(B,C), not(C))
ternary(A, not(B), not(C))
ternary(A, nand(B,C), not(C))
ternary(A, and(B,C), not(B))
ternary(A, xor(B,C), not(B))
ternary(A, or(B,C), not(B))
ternary(A, nand(B,C), not(B))
ternary(A, B, nand(B,C))
ternary(A, C, nand(B,C))
ternary(A, xor(B,C), nand(B,C))
ternary(A, or(B,C), nand(B,C))
ternary(A, eqv(B,C), nand(B,C))
```
Exploitation of `xxeval` for the above patterns to be added as a follow
up.
Co-authored-by: Tony Varghese <tony.varghese@ibm.com>
`f16` is more functional than just a storage type on the platform,
though it does have some codegen issues [1]. To prepare for future
changes, do the following nonfunctional updates to the existing `half`
test:
* Add tests for passing and returning the type directly.
* Add tests showing bitcast behavior, which is currently incorrect but
serves as a baseline.
* Add tests for `fabs` and `copysign` (trivial operations that shouldn't
require libcalls).
* Add invocations for big-endian and for PPC32.
* Rename the test to `half.ll` to reflect its status, which also matches
other backends.
[1]: https://github.com/llvm/llvm-project/issues/97975
This patch enables `-fpatchable-function-entry` on PPC64 little-endian
Linux. It is mutually exclusive with existing XRay instrumentation on
this target.
