DemandedBits is forced to all ones if there are multiple users.
The changes X86 test cases looks like they were miscompiles before.
The value of eax/rax from the cmov is returned from the function in
addition to being used by the sar. That usage needs all bits even
though the sar doesn't.
We have an existing DAG combine for when an insert/extract subvector pair is entirely a nop, but we hadn't handled the case where the net result was either an insert or an extract (but not both). The transform is restricted to index = 0 to avoid having to adjust indices after the transform.
Reviews, a couple comments on the test changes:
* Mostly RISCV, mostly schedule reordering.
* One real regression in splats-with-mixed-vl.ll due to a different overly aggressive combine, fix in a follow up patch.
* The test/CodeGen/X86/vector-replicaton-i1-mask.ll diff looked concerning at first, but not the mask size at most 4 i1s. I think the type changes on the mask loads are correct, but would welcome a second opinion with someone more familiar with AVX512 codegen.
Differential Revision: https://reviews.llvm.org/D158201
This reverts commit 54d663d5896008c09c938f80357e2a056454bc65, which breaks the test CodeGen/SystemZ/ctpop-01.ll for stage2-ubsan check (see https://lab.llvm.org/buildbot/#/builders/85/builds/18410)
I manually confirmed that the test had been passing immediately prior to that commit
(BUILDBOT_REVISION=4772c66cfb00d60f8f687930e9dd3aa1b6872228 llvm-zorg/zorg/buildbot/builders/sanitizers/buildbot_bootstrap_ubsan.sh)
This modifies the G_UADDE legalizaton to a version that looks shorter
on Mips and RISC-V when feeding the equivalent IR to SelectionDAG.
This also removes the boolean select from G_USUBE.
Comments taken from LegalizeDAG and tweaked.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D158232
RISC-V found a case where the CombineTo caused N to be CSEd with
an existing node and then deleted. The top level DAGCombiner loop
was surprised to find a node was deleted, but SDValue() was returned
from the visit function.
We need to return SDValue(N, 0) to tell the top level loop that
a change was made, but the worklist updates were already handled.
Fixes#64772.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D158208
If carryin was 1, and RHS is 0xffffffff we were not giving a carry
out.
In that case Res would be equal to LHS, so Res <u LHS would be false.
But there should be a carry out since carryin+RHS wraps around to 0.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D157943
Simple function which scalarizes Ops then ExtOrTruncs them according to function parameters
Differential Revision: https://reviews.llvm.org/D157733
Change-Id: Ie5215069228f7bf530cd2dbb4bd17cbf409e046a
This extends the lowering of ceil, floor, nearbyint, rint, round, roundeven and
trunc. They are all very similar, so can reuse the same legalization info.
selectIntrinsicTrunc and selectIntrinsicRound can be removed as they can be
selected via tablegen patterns, and G_INTRINSIC_ROUNDEVEN is marked as a gisel
equivalent of froundeven. Otherwise this reuses the existing code, filling it
out to handle more types.
Differential Revision: https://reviews.llvm.org/D157679
When building a fatbinary, the driver invokes the compiler multiple
times with different "--target". (For example, with "-x cuda
--cuda-gpu-arch=sm_70" flags, clang will be invoded twice, once with
--target=x86_64_...., once with --target=sm_70) If we use
-fsplit-machine-functions or -fno-split-machine-functions for such
invocation, the driver reports an error.
This CL changes the behavior so:
- "-fsplit-machine-functions" is now passed to all targets, for non-X86
targets, the flag is a NOOP and causes a warning.
- "-fno-split-machine-functions" now negates -fsplit-machine-functions (if
-fno-split-machine-functions appears after any -fsplit-machine-functions)
for any target triple, previously, it causes an error.
- "-fsplit-machine-functions -Xarch_device -fno-split-machine-functions"
enables MFS on host but disables MFS for GPUS without warnings/errors.
- "-Xarch_host -fsplit-machine-functions" enables MFS on host but disables
MFS for GPUS without warnings/errors.
Reviewed by: xur, dhoekwater
Differential Revision: https://reviews.llvm.org/D157750
1. In X86LowerAMXType.cpp dyn_cast could lead to UserI be nullptr which coud be dref in IRBuilder constructor.
2. In AsmPrinter.cpp, doInitialization could make MMI be nullptr if MMIWP->getMMI() is false, then the deref after could be unexpected.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D157948
Fix verification that a PhysReg is live in to an MBB.
isLiveIn does not handle reg units, so cannot identify when a
register would be defined because its super register is partially
defined.
Additionally a PhysReg may be partial defined at block entry and
then fully defined before any use.
Reviewed By: foad, arsenm
Differential Revision: https://reviews.llvm.org/D157086
We should query the subtarget of the calling function, not of the
intrinsic.
This probably makes no functional difference (as libcalls are
unlikely to vary across subtargets), but fixes minor compile-time
regressions from unnecessary subtarget instantiations.
Followup to D157567.
Differential Revision: https://reviews.llvm.org/D157848
As far as I understand - When lowering a G_CONSTANT_FOLD_BARRIER we replace the
DstReg with SrcReg, and need to check that the register class is equivalent
when doing so for the replacement to be legal. During lowering we could end up
visiting nodes in an odd order, leaving a G_CONSTANT_FOLD_BARRIER with a known
regclass for the src, but only a regbank for the dst. Providing the Regbank
contains the regclass, the replacement should still be safe.
This fixes an assert seen in the llvm-test-suite when lowering hoisted
constants, relaxing canReplaceReg to account for the case when the regbank
covers the regclass, so it is better able to handle differences in visiting
order.
Differential Revision: https://reviews.llvm.org/D157202
On X86 for vec types `(X & Y) == Y` is generally preferable to
`(X & Y) != 0`. Creating zero requires an extra instruction and on
pre-avx512 targets there is no vector `pcmpne` so it requires two
additional instructions to invert the `pcmpeq`.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D157014
1) Handle casts a bit more cleanly just with a loop rather than with
recursion.
2) Add additional cases for smin/smax
3 ) For shifts we can also deduce non-zero if the maximum shift amount
on the known 1s is non-zero.
Differential Revision: https://reviews.llvm.org/D156777
Because the code layout is not known during compilation, the distance of
cross-section jumps is not knowable at compile-time. Because of this, we
should assume that any cross-sectional jumps are out of range. This
assumption is necessary for machine function splitting on AArch64, which
introduces cross-section branches in the middle of functions. The linker
relaxes out-of-range unconditional branches, but it clobbers X16 to do
so; it doesn't relax conditional branches, which must be manually
relaxed by the compiler.
Differential Revision: https://reviews.llvm.org/D145211
Move the sra(shl(x,c1),c1) -> sign_extend_inreg(x) fold inside SimplifyDemandedBits so we can recognize hidden splats with DemandedElts masks.
Because the c1 shift amount has multiple uses, hidden splats won't get simplified to a splat constant buildvector - meaning the existing fold in DAGCombiner::visitSRA can't fire as it won't see a uniform shift amount.
I also needed to add TLI preferSextInRegOfTruncate hook to help keep truncate(sign_extend_inreg(x)) vector patterns on X86 so we can use PACKSS more efficiently.
Differential Revision: https://reviews.llvm.org/D157972
We already do this in getNode, but the undef might appear during
another DAGCombine.
While here remove code for handling noop truncates. getNode checks
the types and won't a noop truncate.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D157910
As with D157399 we can reduce duplication by doing this. Unlike that
patch, I'm posting the refactoring before the functional changes I want
to make here.
Differential Revision: https://reviews.llvm.org/D157403
Hopefully a straightforward refactoring that reduces duplication that
was clearly causing repeated work in D157287.
Differential Revision: https://reviews.llvm.org/D157399
This is a lot of copy-pasting for the existing handling of
G_VECREDUCE_FMAX/G_VECREDUCE_FMIN to add handling for
G_VECREDUCE_FMAXIMUM/G_VECREDUCE_FMINIMUM in the same way.
Differential Revision: https://reviews.llvm.org/D156615
Currently when widening operands for insert_subvector nodes, we check
first that the indices are valid by seeing if the subvector is
statically known to be smaller than or equal to the in-place vector.
However if we're inserting a fixed subvector into a scalable vector we rely on
the minimum vector length of the latter. This patch extends the widening logic
to also take into account the minimum vscale from the vscale_range attribute,
so we can handle more scenarios where we know the scalable vector is large
enough to contain the subvector.
Fixes https://github.com/llvm/llvm-project/issues/63437
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D153519
This adds legalization for G_VECREDUCE_FMIN and G_VECREDUCE_FMAX, where the
selection can go via tablegen patterns. I haven't tried to get non-power2 types
working yet, just the more legal types.
Differential Revision: https://reviews.llvm.org/D156614
D141386 changed the semantics of !range metadata to return poison
on violation. If !range is combined with !noundef, violation is
immediate UB instead, matching the old semantics.
In theory, these IR semantics should also carry over into SDAG.
In practice, DAGCombine has at least one key transform that is
invalid in the presence of poison, namely the conversion of logical
and/or to bitwise and/or (c7b537bf09/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp (L11252)).
Ideally, we would fix this transform, but this will require
substantial work to avoid codegen regressions.
In the meantime, avoid transferring !range metadata without
!noundef, effectively restoring the old !range metadata semantics
on the SDAG layer.
Fixes https://github.com/llvm/llvm-project/issues/64589.
Differential Revision: https://reviews.llvm.org/D157685
This fixes https://github.com/llvm/llvm-project/issues/60766
With MSVC style exception-handling (funclets), no registers are
alive when entering the funclet so they must be reloaded from the
stack. MachineLICM can sometimes hoist such reloads out of the
funclet which is not correct, the register will have been clobbered
when entering the funclet. This can happen in any loop that
contains a try-catch.
This has been tested on x86_64-pc-window-msvc. I'm not sure if
funclets work the same on the other windows archs.
Reviewed By: rnk, arsenm
Differential Revision: https://reviews.llvm.org/D153337
`MachineInstr.h` is a commonly included file and this includes
`llvm/ADT/SmallSet.h` for one function `getUsedDebugRegs()`, which is
used only in one place.
According to `ClangBuildAnalyzer` (run solely on building LLVM, no other
projects) the second most expensive template to instantiate is the
`SmallSet::insert` method used in the `inline` implementation in
`getUsedDebugRegs()`:
```
**** Templates that took longest to instantiate:
554239 ms: std::unordered_map<int, int> (2826 times, avg 196 ms)
521187 ms: llvm::SmallSet<llvm::Register, 4>::insert (930 times, avg 560
ms)
...
```
By removing this method and putting its implementation in the one call
site we greatly reduce the template instantiation time and reduce the
number of includes.
When copying the implementation, I removed a check on `MO.getReg()` as
this is checked within `MO.isVirtual()`.
Differential Revision: https://reviews.llvm.org/D157720
We must also track the super sources of a copy, otherwise we introduce a sort of subtle bug.
Consider:
1. DEF r0:r1
2. USE r1
3. r6:r9 = COPY r10:r13
4. r14:15 = COPY r0:r1
5. USE r6
6.. r1:4 = COPY r6:9
BackwardCopyPropagateBlock processes the instructions from bottom up. After processing 6., we will have propagatable copy for r1-r4 and r6-r9. After 5., we invalidate and erase the propagatble copy for r1-r4 and r6 but not for r7-r9.
The issue is that when processing 3., data structures still say we have valid copies for dest regs r7-r9 (from 6.). The corresponding defs for these registers in 6. are r1:r4, which we mark as registers to invalidate. When invalidating, we find the copy that corresponds to r1 is 4. (this was added when processing 4.), and we say that r1 now maps to unpropagatable copies. Thus, when we process 2., we do not have a valid copy, but when we process 1. we do -- because the mapped copy for subregister r0 was never invalidated.
The net result is to propagate the copy from 4. to 1., and replace DEF r0:r1 with DEF r14:r15. Then, we have a use before def in 2.
The main issue is that we have an inconsitent state between which def regs and which src regs are valid. When processing 5., we mark all the defs in 6. as invalid, but only the subreg use as invalid. Either we must only invalidate the individual subreg for both uses and defs, or the super register for both.
Differential Revision: https://reviews.llvm.org//D157564
Change-Id: I99d5e0b1a0d735e8ea3bd7d137b6464690aa9486
This support promotion for vp.bitreverse/bswap/ctlz/ctlz_zero_undef/cttz/cttz_zero_undef/ctpop/fshr/fshl.
Reviewed By: craig.topper, luke
Differential Revision: https://reviews.llvm.org/D157607
Original commit didn't handle the case where one of the stores was a
truncating store of the build_vector. The existing codepath produced
wrong code (which thankfully also failed asserts) instead of guarding
against unexpected types. Original commit message follows..
Ran across this when making a change to RISCV memset lowering. Seems
very odd that manually merging a store into a vector prevents it from
being further merged.
Differential Revision: https://reviews.llvm.org/D156349
