If the pre-truncated value was the same width as the extension, and the assertzext guarantees that the extended bits are already zero, then skip the zext/trunc 'zero_extend_inreg' pattern.
Addresses several regressions noticed in D155472
This modifies the switch-statement generation in SelectionDAGBuilder,
specifically the part that generates case clusters of type CC_JumpTable.
A table-based branch of any kind is at risk of being a JOP gadget, if
it doesn't range-check the offset into the table. For some types of
table branch, such as Arm TBB/TBH, the impact of this is limited
because the value loaded from the table is a relative offset of
limited size; for others, such as a MOV PC,Rn computed branch into a
table of further branch instructions, the gadget is fully general.
When compiling for branch-target enforcement via Arm's BTI system,
many of these table branch idioms use branch instructions of types
that do not require a BTI instruction at the branch destination. This
avoids the need to put a BTI at the start of each case handler,
reducing the number of available gadgets //with// BTIs (i.e. ones
which could be used by a JOP attack in spite of the BTI system). But
without a range check, the use of a non-BTI-requiring branch also
opens up a larger range of followup gadgets for an attacker's use.
A defence against this is to avoid optimising away the range check on
the table offset, even if the compiler believes that no out-of-range
value should be able to reach the table branch. (Rationale: that may
be true for values generated legitimately by the program, but not
those generated maliciously by attackers who have already corrupted
the control flow.)
The effect of keeping the range check and branching to an unreachable
block is that no actual code is generated at that block, so it will
typically point at the end of the function. That may still cause some
kind of unpredictable code execution (such as executing data as code,
or falling through to the next function in the code section), but even
if so, there will only be //one// possible invalid branch target,
rather than giving an attacker the choice of many possibilities.
This defence is enabled only when branch target enforcement is in use.
Without branch target enforcement, the range check is easily bypassed
anyway, by branching in to a location just after it. But with
enforcement, the attacker will have to enter the jump table dispatcher
at the initial BTI and then go through the range check. (Or, if they
don't, it's because they //already// have a general BTI-bypassing
gadget.)
Reviewed By: MaskRay, chill
Differential Revision: https://reviews.llvm.org/D155485
The option is used to force the use of resource intervals
in the machine scheduler, effectively ignoring the value of
`EnableIntervals` in the instance of the `SchedMachineModel`.
Reviewed By: anemet
Differential Revision: https://reviews.llvm.org/D156540
Reapply after D156401, which stops PatternMatch from recognizing
binop constant expressions, which should avoid the infinite loops
and assertion failures this patch previously exposed.
-----
In preparation for removing support for and/or expressions, mark
them as undesirable. As such, we will no longer implicitly create
such expressions, but they still exist.
Introduced the convergent equivalent of the existing G_INTRINSIC opcodes:
- G_INTRINSIC_CONVERGENT
- G_INTRINSIC_CONVERGENT_W_SIDE_EFFECTS
Out of the targets that currently have some support for GlobalISel, the patch
assumes that the convergent intrinsics only relevant to SPIRV and AMDGPU.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D154766
This attempts to expand the handling for G_FMAXNUM/G_FMINNUM for vector types,
which is hopefully fairly straightforward now that fptrunc and fpext are
working.
Differential Revision: https://reviews.llvm.org/D156171
We could use x0 form in vsetvli when we already know the vlmax and avl is equal to it.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D156404
Change the scheduler's physical register dependency tracking from
registers-and-their-aliases to regunits. This has a couple of advantages
when subregisters are used:
- The dependency tracking is more accurate and creates fewer useless
edges in the dependency graph. An AMDGPU example, edited for clarity:
SU(0): $vgpr1 = V_MOV_B32 $sgpr0
SU(1): $vgpr1 = V_ADDC_U32 0, $vgpr1
SU(2): $vgpr0_vgpr1 = FLAT_LOAD_DWORDX2 $vgpr0_vgpr1, 0, 0
There is a data dependency on $vgpr1 from SU(0) to SU(1) and from
SU(1) to SU(2). But the old dependency tracking code also added a
useless edge from SU(0) to SU(2) because it thought that SU(0)'s def
of $vgpr1 aliased with SU(2)'s use of $vgpr0_vgpr1.
- On targets like AMDGPU that make heavy use of subregisters, each
register can have a huge number of aliases - it can be quadratic in
the size of the largest defined register tuple. There is a much lower
bound on the number of regunits per register, so iterating over
regunits is faster than iterating over aliases.
The LLVM compile-time tracker shows a tiny overall improvement of 0.03%
on X86. I expect a larger compile-time improvement on targets like
AMDGPU.
Differential Revision: https://reviews.llvm.org/D156552
Using implicit CHECK prefix in one FileCheck invocation and explicit
CHECK-V83A in the other one seems to misguide to use CHECK: lines as
a common matching prefix at various places. Also note that
; CHECK, CHECK-V83A: ...
line only matches the "CHECK-V83A" prefix.
This commit explicitly splits the checks into common ones (CHECK) and
invocation-specific ones (COMPAT and V83A) and updates the assertions
with the update_llc_test_checks.py script.
Reviewed By: efriedma, MaskRay
Differential Revision: https://reviews.llvm.org/D156327
The current implementation generates a csect with a
".rodata.str.x.y" prefix for a MergeableCString variable definition.
However, a reference to such variable does not get the prefix in its
name because there's not enough information in the containing IR.
In particular, without seeing the initializer and absent of some other
indicators, we cannot tell that the referenced variable is a null-
terminated string.
When the AIX codegen in llvm was being developed, the prefixing was copied
from ELF without having the linker take advantage of the info.
Currently, the AIX linker does not have the capability to merge
MergeableCString variables. If such feature would ever get implemented,
the contract between the linker and compiler would have to be reconsidered.
Here's the before and after of this change:
```
@a = global i64 320255973571806, align 8
@strA = unnamed_addr constant [7 x i8] c"hello\0A\00", align 1 ;; Mergeable1ByteCString
@strB = unnamed_addr constant [8 x i8] c"Blahah\0A\00", align 1 ;; Mergeable1ByteCString
@strC = unnamed_addr constant [2 x i16] [i16 1, i16 0], align 2 ;; Mergeable2ByteCString
@strD = unnamed_addr constant [2 x i16] [i16 1, i16 1], align 2 ;; !isMergeableCString
@strE = external unnamed_addr constant [2 x i16], align 2
-fdata-sections:
.text extern .rodata.str1.1strA .text extern strA
0 SD RO 0 SD RO
.text extern .rodata.str1.1strB .text extern strB
0 SD RO 0 SD RO
.text extern .rodata.str2.2strC ===> .text extern strC
0 SD RO 0 SD RO
.text extern strD .text extern strD
0 SD RO 0 SD RO
.data extern a .data extern a
0 SD RW 0 SD RW
undef extern strE undef extern strE
0 ER UA 0 ER UA
-fno-data-sections:
.text unamex .rodata.str1.1 .text unamex .rodata
0 SD RO 0 SD RO
.text extern strA .text extern strA
0 LD RO 0 LD RO
.text extern strB .text extern strB
0 LD RO 0 LD RO
.text unamex .rodata.str2.2 ===> .text extern strC
0 SD RO 0 LD RO
.text extern strC .text extern strD
0 LD RO 0 LD RO
.text unamex .rodata .data unamex .data
0 SD RO 0 SD RW
.text extern strD .data extern a
0 LD RO 0 LD RW
.data unamex .data undef extern strE
0 SD RW 0 ER UA
.data extern a
0 LD RW
undef extern strE
0 ER UA
```
Reviewed by: David Tenty, Fangrui Song
Differential Revision: https://reviews.llvm.org/D156202
In SelectionDAG InstrEmitter automatically puts dead flags on unused
physreg defs everywhere. The generated selectors should also set dead
on physreg defs that were not used in the pattern.
This is a workaround for a coalescer bug where coalescing
SUBREG_TO_REG ends up losing the liveness of the high bits of the
source register. The result is an incorrect undef subregister def
instead of preserving the high values. Work around the observed
failure after the resulting mov is eliminated during allocation until
a proper fix is ready. I believe the proper fix is to make
SUBREG_TO_REG use a tied operand.
The test should catch a regression originally observed after
b7836d856206ec39509d42529f958c920368166b and should not show a
difference after a496c8be6e638ae58bb45f13113dbe3a4b7b23fd is reverted.
https://reviews.llvm.org/D156164
This is required for many trees produced in practice for i8 CodeGen.
Differential Revision: https://reviews.llvm.org/D155864
Change-Id: Iac01d183d9998b15138bdc7a5051e3bed338e7d9
This adds better support for call frame pseudos that adjust SP in
PEI::replaceFrameIndicesBackward.
Running frame index elimination backwards is preferred because it can
do backwards register scavenging (on targets that require scavenging)
which does not rely on accurate kill flags.
Differential Revision: https://reviews.llvm.org/D156434
Consider the following pattern:
```
%binop_ = binop %x, %y
%select_ = select %c, %binop_, %x
```
If there is an identity `%identity` operand for `binop`, it is possible to transform
the above code to:
```
%opearand = select %c, %y, %identity
%result = binop %x, %operand
```
This transformation is profitable when `%identity` is all zeroes or
ones.
This patch commits a test for such patterns.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D155481
Implement XCVbi intrinsics for CV32E40P according to the specification.
This commit is part of a patch-set to upstream the 7 vendor specific extensions of CV32E40P.
Contributors: @CharKeaney, @jeremybennett, @lewis-revill, @liaolucy, Nandni Jamnadas, @paolos, @simoncook, @xmj.
bf2ad26b4ff856aab9a62ad168e6bdefeedc374f originally commited.
e4777dc4b9cb371971523cc603e1b8a5c7255e7e reverted due to test failures caused by a merge conflict marker in llvm/test/CodeGen/RISCV/attributes that was accidentally checked in.
This commit removed the conflict marker and recommitted.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D154412
Using segmented stacks with execute-only mostly works, but we need to
use the correct movi32 opcode in 6-M, and there's one place where for
thumb1 (i.e. 6-M and 8-M.base) a constant pool was unconditionally
used which needed to be fixed.
Differential Revision: https://reviews.llvm.org/D156339
Implement XCVbi intrinsics for CV32E40P according to the specification.
This commit is part of a patch-set to upstream the 7 vendor specific extensions of CV32E40P.
Contributors: @CharKeaney, @jeremybennett, @lewis-revill, @liaolucy, Nandni Jamnadas, @PaoloS, @simoncook, @xmj.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D154412
Implement XCVsimd intrinsics for CV32E40P according to the specification.
This commit is part of a patch-set to upstream the 7 vendor specific extensions of CV32E40P.
Contributors: @CharKeaney, @jeremybennett, @lewis-revill, @liaolucy, Nandni Jamnadas, @PaoloS, @simoncook, @xmj.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D153721
According to the latest spec, Zvfbfwma requires Zvfbfmin and Zvfbfmin requires Zfbfmin, with FLH/FSH/FMV.H.X/HMV.X.H removed from Zvfbfwma.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D155916
Implement XCValu intrinsics for CV32E40P according to the specification.
This is a commit of the patch-set to upstream the 7 vendor specific extensions of CV32E40P.
Contributors: @CharKeaney, Nandni Jamnadas, Serkan Muhcu, @jeremybennett, @lewis-revill, @liaolucy, @simoncook, @xmj
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D153748
llvm-objdump -d will be changed to not display mapping symbols by
default (D156190).
Add --show-all-symbols to make the intent clearer and prevent test
adjustment with the new behavior.
This patch adds codegen support for vector with bfloat16 type in llvm backend.
With this patch, Zvbfmin/Zvbfwma instructions as well as vle16/vse16 can generated from newly added bf16 IR intrinsics.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D156287
The code was written with the implicit assumption that each IMPLICIT_DEF either a) the tied operand, or b) an untied source, but not both. This is true right now, but an upcoming change may allow CSE of IMPLICIT_DEFs in some cases, so let's rewrite the code to handle that possibility.
I added an MIR case which demonstrates the multiple use IMPLICIT_DEF. To my knowledge, this is not a reachable configuration from IR right now.
As an aside, this makes the structure a much closer match with the sub-reg liveness case, and we can probably just merge these routines. (Future work.)
Differential Revision: https://reviews.llvm.org/D156477
This fixes a code gen issue where savings the swift async context
register (x22) accidentally overwrites the saved value of another
callee-saved register, corrupts its value and causes a crash.
Differential Revision: https://reviews.llvm.org/D156391
PATCHABLE_TYPED_EVENT_CALL and PATCHABLE_EVENT_CALL are pseudo
instructions that expand to XRay sleds, so getInstSizeInBytes
should reflect the size of the sleds, not the pseudo-instructions.
Differential Revision: https://reviews.llvm.org/D156272
The patterns were ripped out in
a4a3ac10cb1a40ccebed4e81cd7e94f1eb71602d so this always needs to be
custom lowered. I absolutely hate how difficult it is to write tests
for these, I have no doubt there are more of these hidden.
Fixes#64142
Record the call frame size on entry to each basic block. This is usually
zero except when a basic block has been split in the middle of a call
sequence.
This simplifies PEI::replaceFrameIndices which previously had to visit
basic blocks in a specific order and had special handling for
unreachable blocks. More importantly it paves the way for an equally
simple implementation of a backwards version of replaceFrameIndices,
which is required to fully convert PrologEpilogInserter to backwards
register scavenging, which is preferred because it does not rely on
accurate kill flags.
Differential Revision: https://reviews.llvm.org/D156113
The indexed fmlal should use a low numbered register for the index operand,
which this fixes by making it V128_lo.
Fixes 64104
Differential Revision: https://reviews.llvm.org/D156296
And dependent commits.
Details in D150388.
This reverts commit 825b7f0ca5f2211ec3c93139f98d1e24048c225c.
This reverts commit 7a98f084c4d121244ef7286bc6503b6a181d446e.
This reverts commit b4a62b1fa546312d882fa12dfdcd015177d66826.
This reverts commit b7836d856206ec39509d42529f958c920368166b.
No conflicts in the code, few tests had conflicts in autogenerated CHECKs:
llvm/test/CodeGen/Thumb2/mve-float32regloops.ll
llvm/test/CodeGen/AMDGPU/fix-frame-reg-in-custom-csr-spills.ll
Reviewed By: alexfh
Differential Revision: https://reviews.llvm.org/D156381
