Propeller and pseudo-probes map profiles back to Machine IR via basic block addresses that are stored in metadata sections.
Empty basic blocks (basic blocks without real code) obfuscate the profile mapping because their addresses collide with their next basic blocks.
For instance, the fallthrough block of an empty block should always be adjacent to it. Otherwise, a completely unnecessary jump would be added.
This patch adds a MachineFunction pass named `GCEmptyBasicBlocks` which attempts to garbage-collect the empty blocks before the `BasicBlockSections` and pass.
This pass removes each empty basic block after redirecting its incoming edges to its fall-through block.
The garbage-collection is not complete. We keep the empty block in 4 cases:
1. The empty block is an exception handling pad.
2. The empty block has its address taken.
3. The empty block is the last block of the function and it has
predecessors.
4. The empty block is the only block of the function.
The first three cases are extremely rare in normal code (no cases for the clang binary). Removing the blocks under the first two cases requires modifying exception handling structures and operands of non-terminator instructions -- which is doable but not worth the additional complexity in the pass.
Reviewed By: tmsriram
Differential Revision: https://reviews.llvm.org/D107534
Rewrites some simple rules that cause little to no codegen regressions as MIR patterns.
I may have missed some easy cases, but some other rules have intentionally been left as-is because bigger
changes are needed to make them work.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D157690
This reverts commit 317a0fe5bd7113c0ac9d30b2de58ca409e5ff754.
This reverts commit 30c4b97aec60895a6905816670f493cdd1d7c546.
See post-commit discussions on https://reviews.llvm.org/D157750 that
we should use a different mechanism to handle the error with --cuda-gpu-arch=
The IR/DiagnosticInfo.cpp, warn_drv_for_elf_only, codegne tests in
clang/test/Driver, and the following driver behavior (downgrading error
to warning) changes are undesired.
```
% clang --target=riscv64 -fsplit-machine-functions -c a.c
warning: -fsplit-machine-functions is not valid for riscv64 [-Wbackend-plugin]
```
The class 'DbgVariable' can be in one of three states, and the "is any of them
initialization" logic for them is repeated in a couple of places. We may want to
expand this class in the future; as such, we factor out this common logic so
that it is easier to modify.
Differential Revision: https://reviews.llvm.org/D158438
Use getFixedValue instead of getKnownMinValue to convert TypeSize
to uint64_t. I believe this would have caught the bug fixed by
D157872.
To prevent false failures, I had to treat a scalable 0 as if it
is fixed value.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D158115
This is a reland of 46d2d7599d9ed5e68fb53e910feb10d47ee2667b, which was
reverted because of breaking build
https://lab.llvm.org/buildbot/#/builders/21/builds/78779. However, this
buildbot is spuriously broken due to Flang::underscoring.f90 being
nondeterministic.
Current behavior for relaxing out-of-range conditional branches
is to invert the conditional and insert a fallthrough unconditional
branch to the original destination. This approach biases the branch
predictor in the wrong direction, which can degrading performance.
Machine function splitting introduces many rarely-taken cross-section
conditional branches, which are improperly relaxed. Avoid inverting
these branches; instead, retarget them to trampolines at the end of the
function. Doing so increases the runtime cost of jumping to cold code
but eliminates the misprediction cost of jumping to hot code.
Differential Revision: https://reviews.llvm.org/D156837
D152276 wasn't handling the case where the inserted element is implicitly truncated into the vector - resulting in a i1 element (implicitly truncated from i8) overwriting 8 bits instead of 1 bit.
This patch is intended to be merged into 17.x so I've just disallowed any vector element vs inserted element type mismatch - technically we could be more elegant and permit truncated stores (as long as the store is still byte sized), but the use cases for that are so limited I'd prefer to play it safe for now.
Candidate patch for #64655 17.x merge
Differential Revision: https://reviews.llvm.org/D158366
Expand (s/z/any)ext instructions to be compatible with more
types for GlobalISel.
This patch mainly focuses on 64-bit and 128-bit vectors with
element size of powers of 2.
It also notably handles larger than legal vectors.
Differential Revision: https://reviews.llvm.org/D157113
This patch removes the `getBBIDOrNumber` which was introduced to allow emitting version 1.
Reviewed By: shenhan
Differential Revision: https://reviews.llvm.org/D158299
Some opcodes in MIR are defined to be convergent by the target by setting
IsConvergent in the corresponding TD file. For example, in AMDGPU, the opcodes
G_SI_CALL and G_INTRINSIC* are marked as convergent. But this is too
conservative, since calls to functions that do not execute convergent operations
should not be marked convergent. This information is available in LLVM IR.
The new flag MIFlag::NoConvergent now allows the IR translator to mark an
instruction as not performing any convergent operations. It is relevant only on
occurrences of opcodes that are marked isConvergent in the target.
Differential Revision: https://reviews.llvm.org/D157475
SwiftErrorValueTracking creates vregs at swifterror use sites and then
connects it with appropriate definitions after instruction selection.
To propagate swifterror values SwiftErrorValueTracking::propagateVRegs
iterates over basic blocks in RPO, but some vregs previously created
at use sites may be located in blocks that became unreachable after
instruction selection. Because of that there will no definition for
such vregs and that may cause issues down the pipeline.
To ensure that all vregs created by the SwiftErrorValueTracking will
be defined propagateVRegs was updated to insert IMPLICIT_DEF at the
beginning of unreachable blocks containing swifterror uses.
Related issue: https://github.com/llvm/llvm-project/issues/59751
Reviewed By: compnerd
Differential Revision: https://reviews.llvm.org/D141053
Targets may lose some optimization opportunities for certain vector operation
if we reduce BUILD_VECTOR to BITCAST early.
And if VT is not legal, reduce BUILD_VECTOR to BITCAST before LegailizeTypes
can get benefit. Because type-legalizer often scalarizes illegal type of vectors.
Reviewed By: sebastian-ne
Differential Revision: https://reviews.llvm.org/D156645
On the extract_subvector side, we already have the restriction. With D158201, we'd start getting unprofitable splat combines unless we add the same one on the extract_subvector side.
Differential Revision: https://reviews.llvm.org/D158202
Refactor BasicBlockSections to use the target-specific noop insertion
hook from TargetInstrInfo instead of building it ourselves. Using the
TII hook is both cleaner and makes it easier to extend BBSections to
non-X86 targets.
Differential Revision: https://reviews.llvm.org/D158303
We have an existing DAG combine for when an insert/extract subvector pair is entirely a nop, but we hadn't handled the case where the net result was either an insert or an extract (but not both). The transform is restricted to index = 0 to avoid having to adjust indices after the transform.
Differential Revision: https://reviews.llvm.org/D158201
DemandedBits is forced to all ones if there are multiple users.
The changes X86 test cases looks like they were miscompiles before.
The value of eax/rax from the cmov is returned from the function in
addition to being used by the sar. That usage needs all bits even
though the sar doesn't.
We have an existing DAG combine for when an insert/extract subvector pair is entirely a nop, but we hadn't handled the case where the net result was either an insert or an extract (but not both). The transform is restricted to index = 0 to avoid having to adjust indices after the transform.
Reviews, a couple comments on the test changes:
* Mostly RISCV, mostly schedule reordering.
* One real regression in splats-with-mixed-vl.ll due to a different overly aggressive combine, fix in a follow up patch.
* The test/CodeGen/X86/vector-replicaton-i1-mask.ll diff looked concerning at first, but not the mask size at most 4 i1s. I think the type changes on the mask loads are correct, but would welcome a second opinion with someone more familiar with AVX512 codegen.
Differential Revision: https://reviews.llvm.org/D158201
This reverts commit 54d663d5896008c09c938f80357e2a056454bc65, which breaks the test CodeGen/SystemZ/ctpop-01.ll for stage2-ubsan check (see https://lab.llvm.org/buildbot/#/builders/85/builds/18410)
I manually confirmed that the test had been passing immediately prior to that commit
(BUILDBOT_REVISION=4772c66cfb00d60f8f687930e9dd3aa1b6872228 llvm-zorg/zorg/buildbot/builders/sanitizers/buildbot_bootstrap_ubsan.sh)
This modifies the G_UADDE legalizaton to a version that looks shorter
on Mips and RISC-V when feeding the equivalent IR to SelectionDAG.
This also removes the boolean select from G_USUBE.
Comments taken from LegalizeDAG and tweaked.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D158232
RISC-V found a case where the CombineTo caused N to be CSEd with
an existing node and then deleted. The top level DAGCombiner loop
was surprised to find a node was deleted, but SDValue() was returned
from the visit function.
We need to return SDValue(N, 0) to tell the top level loop that
a change was made, but the worklist updates were already handled.
Fixes#64772.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D158208
If carryin was 1, and RHS is 0xffffffff we were not giving a carry
out.
In that case Res would be equal to LHS, so Res <u LHS would be false.
But there should be a carry out since carryin+RHS wraps around to 0.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D157943
Simple function which scalarizes Ops then ExtOrTruncs them according to function parameters
Differential Revision: https://reviews.llvm.org/D157733
Change-Id: Ie5215069228f7bf530cd2dbb4bd17cbf409e046a
This extends the lowering of ceil, floor, nearbyint, rint, round, roundeven and
trunc. They are all very similar, so can reuse the same legalization info.
selectIntrinsicTrunc and selectIntrinsicRound can be removed as they can be
selected via tablegen patterns, and G_INTRINSIC_ROUNDEVEN is marked as a gisel
equivalent of froundeven. Otherwise this reuses the existing code, filling it
out to handle more types.
Differential Revision: https://reviews.llvm.org/D157679
When building a fatbinary, the driver invokes the compiler multiple
times with different "--target". (For example, with "-x cuda
--cuda-gpu-arch=sm_70" flags, clang will be invoded twice, once with
--target=x86_64_...., once with --target=sm_70) If we use
-fsplit-machine-functions or -fno-split-machine-functions for such
invocation, the driver reports an error.
This CL changes the behavior so:
- "-fsplit-machine-functions" is now passed to all targets, for non-X86
targets, the flag is a NOOP and causes a warning.
- "-fno-split-machine-functions" now negates -fsplit-machine-functions (if
-fno-split-machine-functions appears after any -fsplit-machine-functions)
for any target triple, previously, it causes an error.
- "-fsplit-machine-functions -Xarch_device -fno-split-machine-functions"
enables MFS on host but disables MFS for GPUS without warnings/errors.
- "-Xarch_host -fsplit-machine-functions" enables MFS on host but disables
MFS for GPUS without warnings/errors.
Reviewed by: xur, dhoekwater
Differential Revision: https://reviews.llvm.org/D157750
1. In X86LowerAMXType.cpp dyn_cast could lead to UserI be nullptr which coud be dref in IRBuilder constructor.
2. In AsmPrinter.cpp, doInitialization could make MMI be nullptr if MMIWP->getMMI() is false, then the deref after could be unexpected.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D157948
Fix verification that a PhysReg is live in to an MBB.
isLiveIn does not handle reg units, so cannot identify when a
register would be defined because its super register is partially
defined.
Additionally a PhysReg may be partial defined at block entry and
then fully defined before any use.
Reviewed By: foad, arsenm
Differential Revision: https://reviews.llvm.org/D157086
We should query the subtarget of the calling function, not of the
intrinsic.
This probably makes no functional difference (as libcalls are
unlikely to vary across subtargets), but fixes minor compile-time
regressions from unnecessary subtarget instantiations.
Followup to D157567.
Differential Revision: https://reviews.llvm.org/D157848
As far as I understand - When lowering a G_CONSTANT_FOLD_BARRIER we replace the
DstReg with SrcReg, and need to check that the register class is equivalent
when doing so for the replacement to be legal. During lowering we could end up
visiting nodes in an odd order, leaving a G_CONSTANT_FOLD_BARRIER with a known
regclass for the src, but only a regbank for the dst. Providing the Regbank
contains the regclass, the replacement should still be safe.
This fixes an assert seen in the llvm-test-suite when lowering hoisted
constants, relaxing canReplaceReg to account for the case when the regbank
covers the regclass, so it is better able to handle differences in visiting
order.
Differential Revision: https://reviews.llvm.org/D157202
On X86 for vec types `(X & Y) == Y` is generally preferable to
`(X & Y) != 0`. Creating zero requires an extra instruction and on
pre-avx512 targets there is no vector `pcmpne` so it requires two
additional instructions to invert the `pcmpeq`.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D157014
1) Handle casts a bit more cleanly just with a loop rather than with
recursion.
2) Add additional cases for smin/smax
3 ) For shifts we can also deduce non-zero if the maximum shift amount
on the known 1s is non-zero.
Differential Revision: https://reviews.llvm.org/D156777
Because the code layout is not known during compilation, the distance of
cross-section jumps is not knowable at compile-time. Because of this, we
should assume that any cross-sectional jumps are out of range. This
assumption is necessary for machine function splitting on AArch64, which
introduces cross-section branches in the middle of functions. The linker
relaxes out-of-range unconditional branches, but it clobbers X16 to do
so; it doesn't relax conditional branches, which must be manually
relaxed by the compiler.
Differential Revision: https://reviews.llvm.org/D145211
Move the sra(shl(x,c1),c1) -> sign_extend_inreg(x) fold inside SimplifyDemandedBits so we can recognize hidden splats with DemandedElts masks.
Because the c1 shift amount has multiple uses, hidden splats won't get simplified to a splat constant buildvector - meaning the existing fold in DAGCombiner::visitSRA can't fire as it won't see a uniform shift amount.
I also needed to add TLI preferSextInRegOfTruncate hook to help keep truncate(sign_extend_inreg(x)) vector patterns on X86 so we can use PACKSS more efficiently.
Differential Revision: https://reviews.llvm.org/D157972
