The use of nested m_Reassociatable matchers by #169644 can result in
high compile times as the inner m_Reassociatable call is being repeated
a lot while the outer call is trying to match. Place the inner
m_ReassociatableAnd at the beginning of the pattern so it is not
repeatedly matched in recursion.
SelectionDAG uses the DAGCombiner to fold a load followed by a sext to a
load and sext instruction. For example, in x86 we will see that
```
%1 = load i32, ptr @GlobArr
#dbg_value(i32 %1, !43, !DIExpression(), !52)
%2 = sext i32 %1 to i64, !dbg !53
```
is converted to:
```
%0:gr64_nosp = MOVSX64rm32 $rip, 1, $noreg, @GlobArr, $noreg, debug-instr-number 1, debug-location !51
DBG_VALUE $noreg, $noreg, !"Idx", !DIExpression(), debug-location !52
```
The `DBG_VALUE` needs to be transferred correctly to the new combined
instruction, and it needs to be appended with a `DIExpression` which
contains a `DW_OP_LLVM_fragment`, describing that the lower bits of the
virtual register contain the value.
This patch fixes the above described problem.
Some floating-point optimization don't trigger because they can produce
incorrect results around signed zeros, and rely on the existence of the
nsz flag which commonly appears when fast-math is enabled.
However, this flag is not a hard requirement when all of the users of
the combined value are either guaranteed to overwrite the sign-bit or
simply ignore it (comparisons, etc.).
The optimizations affected:
- fadd x, +0.0 -> x
- fsub x, -0.0 -> x
- fsub +0.0, x -> fneg x
- fdiv(x, sqrt(x)) -> sqrt(x)
- frem lowering with power-of-2 divisors
The existing code for generating umulh/smulh was checking that that the
getTypeToTransformTo was a LegalOrCustom operation. This only takes a
single legalization step though, so if v4i32 was legal, a v8i32 would be
transformed but a v16i32 would not.
This patch introduces a getLegalTypeToTransformTo that performs
getTypeToTransformTo until a legal type is reached. The umulh/smulh code
can then use it to check if the final resultant type will be legal.
Type legalization can promote constant operands. The MULHU optimization
`mulhu x, (1 << c) -> x >> (bitwidth - c)` was failing when constants
were promoted because:
1. `isConstantOrConstantVector` check rejected promoted constants
2. `BuildLogBase2` -> `takeInexpensiveLog2` -> `matchUnaryPredicate`
rejected promoted constants
This fixes both by adding `AllowTruncation=true`, following the pattern
from the recent UDIV fix (#169491).
If we are force reconstructing a carry from a raw MVT::i1 type, make
sure we don't miss any cases while peeling through trunc/ext chains -
check for i1 types at the start of the while loop
Fixes#169691
This reverts commit 6d5f87fc4284c4c22512778afaf7f2ba9326ba7b.
Previously this failed due to treating the unknown MachineMemOperand
value as known uniform.
If the sign bit is a computed sign mask (i.e., we know it's
either +0 or -0), turn this into a disjoint or. This pattern
appears in the pow implementations.
We also need to know the sign bit of the magnitude is 0 for
the or to be disjoint. Unfortunately the DAG's FP tracking is
weak and we did not have a way to check if the sign bit is known
0, so add something for that. Ideally we would get a complete
computeKnownFPClass implementation.
This is intended to help avoid the regression which caused
d3e7c4ce7a3d7 to be reverted.
Tweak the existing (bitcast (load x)) -> (load (bitcast*)x) fold to handle oneuse freeze as well
Inspired by #163070 - this tries to avoid in place replacement of frozen nodes which has caused infinite loops in the past
Fixes https://github.com/llvm/llvm-project/issues/166596
We cannot use `APInt::isMask` if `numBits` exceeds the bitwidth of APInt
or `numBits` is zero. We avoid such a case by guaranteeing BitWidthDiff
< BitWidth.
When a load or store accesses N bytes starting from a pointer P, and we want to
compute an offset pointer within these N bytes after P, we know that the
arithmetic to add the offset must be inbounds. This is for example relevant
when legalizing too-wide memory accesses, when lowering memcpy&Co., or when
optimizing "vector-load -> extractelement" into an offset load.
For SWDEV-516125.
This PR preserves the InBounds flag (#162477) where possible in PTRADD-related
DAGCombines. We can't preserve them in all the cases that we could in the
analogous GISel change (#152495) because SDAG usually represents pointers as
integers, which means that pointer provenance is not preserved between PTRADD
operations (see the discussion at PR #162477 for more details). This PR marks
the places in the DAGCombiner where this is relevant explicitly.
For SWDEV-516125.
For an insertelt with a dynamic index, the default handling in
DAGTypeLegalizer and LegalizeDAG will reserve a stack slot for the
vector, lower the insertelt to a store, then load the modified vector
back into temporaries. The vector store and load may be legalized into a
sequence of smaller operations depending on the target.
Let V = the vector size and L = the length of a chain of insertelts with
dynamic indices. In the worse case, this chain will lower to O(VL)
operations, which can increase code size dramatically.
Instead, identify such chains, reserve one stack slot for the vector,
and lower all of the insertelts to stores at once. This requires only
O(V + L) operations. This change only affects the default lowering
behavior.
[DAG] Fold mismatched widened avg idioms to narrow form (fixes half of
[llvm#147946](https://github.com/llvm/llvm-project/issues/147946))
1. `trunc(avgceilu(sext(x), sext(y))) -> avgceils(x, y)`
2. `trunc(avgceils(zext(x), zext(y))) -> avgceilu(x, y)`
When inputs are sign-extended, unsigned and signed averaging operations
produce identical results after truncation, allowing us to use the
semantically correct narrow operation.
alive2: https://alive2.llvm.org/ce/z/ZRbfHT
We're very careful not to truncate binary arithmetic ops if it will
affect legality, or cause additional truncation instructions, hence we
currently limit this to cases where one operand is constant.
But if both ops are the same (i.e. for some add/mul cases) then we
wouldn't increase the number of truncations, so can be slightly more
aggressive at folding the truncation.
simplifyDivRem does not work as well after type legalisation because
splatted constants can have a size mismatch between the scalar to splat
and the element type of the splatted result. simplifyDivRem does not
seem to care about this mismatch so I've updated the "is one" check
for the divisor to allow truncation.
Reland #161355, after fixing up the cross-projects-tests for the wasm
simd intrinsics.
Original commit message:
Lower v4f32 and v2f64 fmuladd calls to relaxed_madd instructions.
If we have FP16, then lower v8f16 fmuladds to FMA.
I've introduced an ISD node for fmuladd to maintain the rounding
ambiguity through legalization / combine / isel.
Lower v4f32 and v2f64 fmuladd calls to relaxed_madd instructions.
If we have FP16, then lower v8f16 fmuladds to FMA.
I've introduced an ISD node for fmuladd to maintain the rounding
ambiguity through legalization / combine / isel.
This patch updates the FP-to-Int conversion handling:
- For signed integers: use `ftrunc` followed by clamping to the target
integer range.
- For unsigned integers: apply `fabs` + `ftrunc`, then clamp.
This removes the previous dependence on `nsz` and ensures correct
lowering for both signed and unsigned cases.
I've tested the code generation of -mtriple=amdgcn. It seems that the
assembly code is expected, but I'm not sure how to write a general
testcase for every target.
Fixes#160623.
Add several improvements to DAGCombine patterns for fmin/fmax:
* Fix incorrect results due to minimumnum not being marked as IsMin
- e.g. nnan minimumnum(x, +inf) returned +inf instead of x
* Fix incorrect results checking maximumnum for vecreduce patterns
* Make maxnum/minnum return QNaN if one input is SNaN instead of X
* Quiet SNaN inputs when propagating them e.g. maximum(x, SNaN) = QNaN
* Update comments to mark when SNaN propagation is being ignored
Remove the `NoSignedZerosFPMath` use in `visitFNEG`. Now the only use of
`NoSignedZerosFPMath` is in `foldFPToIntToFP`, but adding fast-math
flags support for `uitofp` may introduce breaking changes.
