Cover more cases in preparation for making greater use
of fcmp based lowerings. Also add more tests for the inverted
cases. Test iszero | isnan test masks. We should probably just
generate every combination of test masks.
This should be a typo in `emitConstantSizeRepmov`. Both its caller and
callee store the alignment in a 64-bit variables, no reason to truncate
it to 32-bit. It results in alignment turns into 0 when larger than
0x100000000.
Fixes#61348
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D145863
Each target's `TargetInstrInfo` is responsible for announcing which code
patterns it is able to transform during the MachineCombiner pass.
Currently, these patterns are applied without preserving the debug
instruction number required by the InstrRef implementation of
LiveDebugValues. As such, we've seen a number of examples where debug
information is dropped for variables in InstrRef mode that were
otherwise available in VarLoc mode. This has been observed both in X86
and AArch examples.
This commit is an initial attempt at preserving said numbers by changing
the general (target agnostic) implementation of TargetInstrInfo: the
reassociation pattern must keep the debug number of the "top level"
instruction, i.e., the instruction whose value represents the final
value of the arithmetic expression. Intermediate values must have their
debug number dropped, as they have no equivalent value in the
unoptimized code.
Future work is required to update each target's
`TargetInstrInfo::genAlternativeCodeSequence` method.
Differential Revision: https://reviews.llvm.org/D145759
[DAGCombiner] handle more store value forwarding
When lowering calls on target like PPC, some stack loads
will be generated for by value parameters. Node CALLSEQ_START
prevents such loads from being combined.
Suggested by @RolandF, this patch removes the unnecessary
loads for the byval parameter by extending ForwardStoreValueToDirectLoad
Reviewed By: nemanjai, RolandF
Differential Revision: https://reviews.llvm.org/D138899
Try to more aggressively narrow masks of extended values.
This is mainly for cases where the mask is trying to zero out any_extended upper bits, assuming we can zext/trunc the values for free.
This catches a few actual missed folds, as well as helps canonicalize a number of other cases which were being caught in isel etc.
Differential Revision: https://reviews.llvm.org/D145866
This came about while investigating ways to handle D145468 in a more generic manner, which involves trying harder to fold and(zext(x),c) -> zext(and(x,c))
Alive2: https://alive2.llvm.org/ce/z/7fXtDt (generic fold)
Differential Revision: https://reviews.llvm.org/D145855
This change improves FS discriminators in the following ways:
(1) use call-stack debug information in the the to generate
discriminators: the same (src/line) DILs can now have same
discriminator value if they come from different call-stacks.
This effectively increases the usable discriminator values
for each round of FS discriminator pass.
(2) don't generate the FS discriminator for meta instructions
(i.e. instructions not emitted). This reduces the number
discriminators conflicts (for the case we run out of discriminator
bits for that pass).
(3) use less expensive hashing of xxHash64.
These improvements should bring better performance for FSAFDO
and they should be used by default. But this change creates
incompatible FS discriminators. For the iterative profile users,
they might see a performance drop in the first release with
this change (due to the fact that the profiles have the old
discriminators and the compiler uses the new discriminator).
We have measured that this is not more than 1.5% on several
benchmarks. Note the degradation should be gone in the second
release and one should expect a performance gain over the binary
without this change.
One possible solution to the iterative profile issue would be
separating discriminators for profile-use and the ones emitted to
the binary. This would require a mechanism to allow two sets of
discriminators to be maintained and then phasing out the first
approach. This is too much churn in the compiler and the
performance implications do not seem to be worth the effort.
Instead, we put the changes under an option so iterative profile
users can do a gradual rollout of this change. We will make the
option default value to true in a later patch and eventually
purge this option from the code base.
Differential Revision: https://reviews.llvm.org/D145171
Ignoring freeze(undef) if it has multiple uses in LowerAVXCONCAT_VECTORS
causes the custom INSERT_SUBVECTOR for vector widening to be ignored.
Differential Revision: https://reviews.llvm.org/D144903
Ignoring freeze(undef) if it has multiple uses in LowerAVXCONCAT_VECTORS
causes the custom INSERT_SUBVECTOR for vector widening to be ignored.
Differential Revision: https://reviews.llvm.org/D144903
Masked variants of UNPCKLPD, UNPCKHPD, and PERMILPS were missing and
be transformed with the same logic as their non-masked counterparts.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D144763
Use this to handle new transform: `{v}unpck{l|h}pd` -> `{v}shufps`. We
need the sched information here as `{v}shufps` is 1 more byte of code
size, so we only want to make this transformation if `{v}shufps` is
actually faster.
Differential Revision: https://reviews.llvm.org/D144570
The merged store touches memory for other underlying objects, so mapping
the merged store to the first underlying object is not correct. For example
in https://github.com/llvm/llvm-project/issues/60744, the merged store is
not correctly analyzed as dependent with memory operations which are also
part of the merged store.
Fixes#60744
Reviewed By: foad
Differential Revision: https://reviews.llvm.org/D144711
This is generally done by the InstCombine, but can be emitted as an
intermediate step and is cheap to handle.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D145177
This is generally done by the InstCombine, but can be emitted as an
intermediate step and is cheap to handle.
Differential Revision: https://reviews.llvm.org/D145143
Without this patch:
%cond = call i32 @llvm.umax.i32(i32 %X, i32 1)
is compiled as:
83 ff 02 cmp $0x2,%edi
b8 01 00 00 00 mov $0x1,%eax
0f 43 c7 cmovae %edi,%eax
With this patch, the compiler generates:
89 f8 mov %edi,%eax
83 ff 01 cmp $0x1,%edi
83 d0 00 adc $0x0,%eax
saving 3 bytes. We should be able to save 5 bytes in larger functions
where the mov is unnecessary.
This patch converts the specific cmov pattern to cmp $1 followed by
adc $0.
This patch partially fixes:
https://github.com/llvm/llvm-project/issues/60374
The LLVM IR optimizer is yet to canonicalize max expressions to
actual @llvm.umax.
Differential Revision: https://reviews.llvm.org/D144451
It turns out that there are relatively trivial, albeit rare, cases that
require a MaxDepth of more than 16 (see added test). However, we want to
avoid having to rely on a large fixed MaxDepth.
Since these cases are relatively rare, apply the following strategy:
1. Start with a low MaxDepth of 16 - if the entry node was not
reached, we can return (the common case).
2. If the entry node was reached, exponentially increase MaxDepth up
to some large limit that should cover all cases and guard against
stack exhaustion.
This retains the better performance with a low MaxDepth in the common
case, and in complex cases backs off and retries. On a whole, this is
preferable vs. starting with a large MaxDepth which would unnecessarily
penalize the common case where a low MaxDepth is sufficient.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D145386
Shows the failure of combineBitcastvxi1 to sign-extend a vXi1 allones vselect operand
There's a number of other problems in Issue #61104 still to address, but this one has an easy quick fix
Similar to most of the other vector-shuffle-* test files
Avoids some codegen deltas due to upcoming changes for no-costs domain switching between shuffle types
During legalization of the SelectionDAG, some nodes are replaced with
arch-specific nodes. These may be complex nodes, where the root node no
longer corresponds to the node that should carry the extra info.
Fix the issue by copying extra info to the new node and all its new
transitive operands during RAUW. See code comments for more details.
This fixes the remaining pcsections-atomics.ll tests on X86.
v2: Optimize copyExtraInfo() deep copy. For now we assume that only
NodeExtraInfo that have PCSections set require deep copy. Furthermore,
limit the depth of graph search while pre-populating the visited set,
assuming the to-be-replaced subgraph 'From' has limited complexity. An
assertion catches if the maximum depth needs to be increased.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D144677
Move the opcode checks to after we have already verified we found a
valid shift instruction (`0 < ShiftAmt`) in `matchUnaryPermuteShuffle`
and `lowerShuffleAsShift`.
Reviewed By: pengfei, RKSimon
Differential Revision: https://reviews.llvm.org/D143786
