181244 Commits

Author SHA1 Message Date
Matt Arsenault
f548c4d83c AMDGPU: Add mode register use to s_getreg_b32
This should fix reading the wrong mode after setting the mode.
Ideally we would have separate pseudos for the case that we know
does not read mode.
2024-05-07 16:06:51 +02:00
Emma Pilkington
dcc7ef3ce8
[AMDGPU][MC] Disable sendmsg SYSMSG_OP_HOST_TRAP_ACK on gfx9+ (#90203)
This is no longer supported as of gfx9. Fixes #52903

This commit also includes some refactoring of sendmsg operand parsing:
  - Use CustomOperand for sendmsg operations, this allows them to be
    conditionally available based on a STI check (and automatically in
    sync with SIDefines.h).
  - Move CustomOperand table lookups from AMDGPUBaseInfo to
    AMDGPUAsmUtils. This cleans up an awkward interface where
    AMDGPUAsmUtils defined a table/size as globals that AMDGPUBaseInfo
    had to loop over.
  - Clean up a few of the operand lookup functions while moving them.
2024-05-07 07:38:58 -04:00
Chris Copeland
651bdb96b1
[ARM] Armv8-R does not require fp64 or neon. (#88287)
This was [addressed for AArch64
here](https://github.com/llvm/llvm-project/pull/79004), but the same
applies to ARM.

Move the enablement of neon+fp64 to `-mcpu=cortex-r52`, which optionally
supports these features.
2024-05-07 11:48:30 +01:00
Paul Walker
235cea720c
[NFC][LLVM] Refactor rounding mode detection of constrained fp intrinsic IDs (#90854)
I've refactored the code to genericise the implementation to better
allow for target specific constrained fp intrinsics.
2024-05-07 11:23:55 +01:00
Simon Pilgrim
abd314938d
[X86] Use GFNI for vXi8 shifts/rotates (#89115)
As detailed here: https://github.com/InstLatx64/InstLatX64_Demo/blob/master/GFNI_Demo.h

We can use the gf2p8affine instruction to lower byte shifts/rotates as well as the existing bitreverse case.

Based off the original patch here: https://reviews.llvm.org/D137026
2024-05-07 10:28:55 +01:00
Quentin Colombet
6ce04747cf
[SDISel] Teach the type legalizer about ADDRSPACECAST (#90969)
Vectorized ADDRSPACECASTs were not supported by the type legalizer.

This patch adds the support for:
- splitting the vector result: <2 x ptr> => 2 x <1 x ptr>
- scalarization: <1 x ptr> => ptr
- widening: <3 x ptr> => <4 x ptr>

This is all exercised by the added NVPTX tests.
2024-05-07 11:08:33 +02:00
Simon Pilgrim
d838e5b3e8
[X86] Add FastImm16 tuning flag to Intel Atom + AMD Bobcat/Ryzen Families (#90635)
This patch limits the icmp_i16(x,c) -> icmp_i32(ext(x),ext(c)) fold to CPUs that aren't known to have fast handling for length-changing prefixes for imm16 operands.

We are always assuming that 66/67h length-changing prefixes cause severe stalls and we should always extend imm16 operands and use a i32 icmp instead, the only exception being Intel Bonnell CPUs.

Agner makes this clear (see microarchitecture.pdf) that there are no stalls for any of the Intel Atom family (at least as far as Tremont - not sure about Gracemont or later). This is also true for AMD Bobcat/Jaguar and Ryzen families.

Recent performance Intel CPUs are trickier - Core2/Nehalem and earlier could have a 6-11cy stall, while SandyBridge onwards this is reduced to 3cy or less. I'm not sure if we should accept this as fast or not, we only use this flag for the icmp_i16 case, so that might be acceptable? If so, we should add this to x86-64-v3/v4 tuning as well.

Part of #90355 + #62952
2024-05-07 10:00:00 +01:00
Ben Shi
50da7680d8
[AVR][NFC] Improve format of target description files (#91296) 2024-05-07 16:42:46 +08:00
hev
ad59967336
[LoongArch] Optimize codegen for ISD::{ROTL,ROTR} (#91174) 2024-05-07 15:46:11 +08:00
Thorsten Schütt
b42f553af5
[GlobalIsel] Combine extract vector element (#90339)
look through shuffle vectors
2024-05-07 07:12:58 +02:00
Jianjian Guan
37fcb323f6
[RISCV] Add codegen support for Zvfbfmin (#87911)
This patch adds basic codegen support for Zvfbfmin extension.
2024-05-07 10:25:06 +08:00
S. Bharadwaj Yadavalli
178ff39500
Revert "[DirectX][DXIL] Set DXIL Version in DXIL target triple based on shader model version" (#91290)
Reverts llvm/llvm-project#90809

Need to investigate ASAN failures.
2024-05-06 22:21:37 -04:00
Monad
0ab4458df0
[InstCombine] Fold cttz(lshr(-1, x) + 1) to width - x (#91244)
Fold
``` llvm
define i64 @src(i64 %50) {
  %52 = lshr i64 -1, %50
  %53 = add i64 %52, 1
  %54 = call i64 @llvm.cttz.i64(i64 %53, i1 false)
  ret i64 %54
}
```
to
``` llvm
define i64 @tgt(i64 %50) {
  %52 = sub i64 64, %50
  ret i64 %52
}
```

as
https://github.com/llvm/llvm-project/pull/91171#pullrequestreview-2040663002
pointed out.

Alive2 proof: https://alive2.llvm.org/ce/z/2aHfYa

Note: the `ctlz` version of this pattern seems not exist in dtcxzyw's
benchmark, so put it aside for now.
2024-05-07 11:06:52 +09:00
Monad
fd0ffb7438
[ValueTracking] Recognize LShr(UINT_MAX, Y) + 1 as a power-of-two (#91171)
There is a missed optimization in
``` llvm
define i8 @known_power_of_two_rust_next_power_of_two(i8 %x, i8 %y) {
  %2 = add i8 %x, -1
  %3 = tail call i8 @llvm.ctlz.i8(i8 %2, i1 true)
  %4 = lshr i8 -1, %3
  %5 = add i8 %4, 1
  %6 = icmp ugt i8 %x, 1
  %p = select i1 %6, i8 %5, i8 1

  %r = urem i8 %y, %p
  ret i8 %r
}
```
which is extracted from the Rust code
``` rust
fn func(x: usize, y: usize) -> usize {
    let z = x.next_power_of_two();
    y % z
}
```
Here `%p` (a.k.a `z`) is semantically a power-of-two, so `y urem p` can
be optimized to `y & (p - 1)`. (Alive2 proof:
https://alive2.llvm.org/ce/z/H3zooY)

---

It could be generalized to recognizing `LShr(UINT_MAX, Y) + 1` as a
power-of-two, which is what this PR does.
Alive2 proof: https://alive2.llvm.org/ce/z/zUPTbc
2024-05-07 10:28:36 +09:00
Nikita Popov
f34d30cdae
[FunctionAttrs] Fix incorrect nonnull inference for non-inbounds GEP (#91180)
For inbounds GEPs, if the source pointer is non-null, the result must
also be non-null. However, this does not hold for non-inbounds GEPs.
    
Fixes https://github.com/llvm/llvm-project/issues/91177.
2024-05-07 09:47:28 +09:00
Nikita Popov
924384161f Revert "[coro][CoroSplit] Use llvm.lifetime.end to compute putting objects on the frame vs the stack (#90265)"
This reverts commit fcf341d3ddfe2289ac88aa3c122b25df8732cc8e.

Causes major compile-time regressions when not using coroutines.
2024-05-07 09:41:25 +09:00
S. Bharadwaj Yadavalli
080978dd20
[DirectX][DXIL] Set DXIL Version in DXIL target triple based on shader model version (#90809)
An earlier commit provided a way to decouple DXIL version from Shader
Model version by representing the DXIL version as `SubArch` in the DXIL 
Target Triple and adding corresponding valid DXIL Arch types.

This change constructs DXIL target triple with DXIL version that is
deduced from Shader Model version specified in the following scenarios:

1. When compilation target profile is specified: 
For e.g., DXIL target triple `dxilv1.8-unknown-shader6.8-library` is
constructed when `-T lib_6_8` is specified. 
2. When DXIL target triple without DXIL version is specified:
For e.g., DXIL target triple `dxilv1.8-pc-shadermodel6.8-library` is
constructed when `-mtriple=dxil-pc-shadermodel6.8-library` is specified.

Updated relevant HLSL tests that check for target triple. 

Validated that Clang (`check-clang`) and LLVM (`check-llvm`) regression
tests pass.
2024-05-06 18:33:57 -04:00
Alan Zhao
fcf341d3dd
[coro][CoroSplit] Use llvm.lifetime.end to compute putting objects on the frame vs the stack (#90265)
The current logic for using lifetime intrinsics to determine whether a
coroutine alloca should live on the coroutine frame or stack doesn't
consider `llvm.lifetime.end`. As a result, some allocas are incorrectly
placed on the stack even though their lifetimes may outlive the stack.
For example, SimplifyCFG may generate code that drops the corresponding
`llvm.lifetime.end` of an `llvm.lifetime.start`, and that code is
incorrectly handled by the existing logic.

To fix this, new logic is introduced where if an alloca's address is
escaped, and there is a path from an `llvm.lifetime.start` to a
coroutine suspend point (e.g. `llvm.coro.suspend`) without an
`llvm.lifetime.end`, then we know the object lives beyond the suspension
point and therefore must go on the coroutine frame.

Fixes https://github.com/llvm/llvm-project/issues/86580
2024-05-06 13:43:52 -07:00
Brandt Bucher
d3e77f5408
Try to use non-volatile registers for preserve_none parameters (#88333)
This uses non-volatile registers for the first four (six on Windows)
registers used for `preserve_none` argument passing. This allows these
registers to stay "pinned", even if the body of the `preserve_none`
function contains calls to other "normal" functions.

Example:

```c
void boring(void);

__attribute__((preserve_none)) void (continuation)(void *, void *, void *, void *);

__attribute__((preserve_none)) void entry(void *a, void *b, void *c, void *d)
{
    boring();
    __attribute__((musttail)) return continuation(a, b, c, d);
}
``` 

Before:

```asm
pushq   %rax
movq    %rcx, %rbx
movq    %rdx, %r14
movq    %rsi, %r15
movq    %rdi, %r12
callq   boring@PLT
movq    %r12, %rdi
movq    %r15, %rsi
movq    %r14, %rdx
movq    %rbx, %rcx
popq    %rax
jmp     continuation@PLT
```

After:

```asm
pushq   %rax
callq   boring@PLT
popq    %rax
jmp     continuation@PLT
```
2024-05-06 10:55:36 -07:00
David Green
363ec6f691 [AArch64][GlobalISel] Common some shuffle mask functions.
This removes the GISel versions of isREVMask, isTRNMask, isUZPMask and
isZipMask. They are combined with the existing versions from SDAG into
AArch64PerfectShuffle.h.
2024-05-06 18:37:04 +01:00
Simon Pilgrim
d0be944aa5
[X86] Add slow div64 tuning flag to Nehalem target (#91129)
This appears to have been missed because later cpus don't inherit from Nehalem tuning much.

Noticed while cleaning up for #90985
2024-05-06 18:31:36 +01:00
Alexey Bataev
c144157f3d [SLP]Use last pointer instead of first for reversed strided stores.
Need to use the last address of the vectorized stores for the strided
stores, not the first one, to correctly store the data.
2024-05-06 10:16:28 -07:00
Luke Lau
52187b9f2e
[RISCV] Move RISCVDeadRegisterDefinitions to post vector regalloc (#90636)
Currently RISCVDeadRegisterDefinitions runs after vsetvli insertion, but
in #70549 vsetvli insertion runs after vector regalloc and as a result
we no longer convert some vsetvli a0, a0s to vsetvli x0, a0. This patch
moves it to after vector regalloc, but before scalar regalloc so we
still get the benefits of reducing register pressure.
2024-05-07 00:36:47 +08:00
Luke Lau
9d9bd76458
[RISCV] Check dead flag on VL def op in RISCVCoalesceVSETVLI. NFC (#91168)
Because LiveVariables has been run, we no longer need to lookup the
users in MachineRegisterInfo anymore and can instead just check for the
dead flag.
2024-05-07 00:02:42 +08:00
Alexey Bataev
a476032101 [SLP]Fix PR91025: correctly handle smin/smax of signed operands.
Need to check that the signed operand has an extra sign bit to be sure
that we do not skip signedness, when trying to minimize bitwidth for
smin/smax intrinsics.
2024-05-06 08:10:20 -07:00
Franklin Zhang
1241e7692a
[AggressiveInstCombine] Fix strncmp inlining (#91204)
Fix the issue that `char` constants are converted to `uint64_t` in the
wrong way when doing the inlining.
2024-05-06 22:04:35 +08:00
Jan Voung
d71771dd27
Revert "Reapply "Use an abbrev to reduce size of VALUE_GUID records in ThinLTO summaries" (#90610)" (#91194)
Reverts llvm/llvm-project#90692

Breaking PPC buildbots. The bots are not meant to test LLD, but are
running a test that is using an old version of LLD without the change
(so is incompatible). Revert until a fix is found.
2024-05-06 21:37:07 +08:00
Jie Fu
b2c2fef5fa [X86] Fix -Wunused-function in X86ISelLowering.cpp (NFC)
llvm-project/llvm/lib/Target/X86/X86ISelLowering.cpp:3582:13:
error: unused function 'isBlendOrUndef' [-Werror,-Wunused-function]
static bool isBlendOrUndef(ArrayRef<int> Mask) {
            ^
1 error generated.
2024-05-06 18:56:34 +08:00
Jie Fu
a9241998fc [X86] Fix -Wsign-compare in X86ISelLowering.cpp (NFC)
llvm-project/llvm/lib/Target/X86/X86ISelLowering.cpp:40081:21:
error: comparison of integers of different signs: 'int' and 'unsigned int' [-Werror,-Wsign-compare]
  for (int I = 0; I != NumElts; ++I) {
                  ~ ^  ~~~~~~~
1 error generated.
2024-05-06 18:45:28 +08:00
Simon Pilgrim
8e7618aa21
[X86] Fold BLEND(PERMUTE(X),PERMUTE(Y)) -> PERMUTE(BLEND(X,Y)) (#90219)
If we don't demand the same element from both single source shuffles (permutes), then attempt to blend the sources together first and then perform a merged permute.

For vXi16 blends we have to be careful as these are much more likely to involve byte/word vector shuffles that will result in the creation of additional shuffle instructions.

This fold might be worth it for VSELECT with constant masks on AVX512 targets, but I haven't investigated this yet, but I've tried to write combineBlendOfPermutes so to be prepared for this.

The PR34592 -O0 regression is an unfortunate failure to cleanup with a later pass that calls SimplifyDemandedElts like the -O3 does - I'm not sure how worried we should be tbh.
2024-05-06 11:25:23 +01:00
Ulrich Weigand
0a0cac6dbd
[SystemZ] Simplify f128 atomic load/store (#90977)
Change definition of expandBitCastI128ToF128 and expandBitCastF128ToI128
to allow for simplified use in atomic load/store.

Update logic to split 128-bit loads and stores in DAGCombine to also
handle the f128 case where appropriate. This fixes the regressions
introduced by recent atomic load/store patches.
2024-05-06 12:17:19 +02:00
Simon Pilgrim
522b4bfe5b
[DAG] Fold bitreverse(shl/srl(bitreverse(x),y)) -> srl/shl(x,y) (#89897)
Noticed while investigating GFNI per-element vector shifts (we can form SHL but not SRL/SRA)

Alive2: https://alive2.llvm.org/ce/z/fSH-rf
2024-05-06 11:13:05 +01:00
WANG Rui
0933a7a125 [LoongArch] Rename some OptWInstrs functions. NFC 2024-05-06 18:07:30 +08:00
hev
e9bcd2bf67
[LoongArch] Optimize *W Instructions at MI level (#90463)
Referring to RISC-V, adding an MI level pass to optimize *W instructions
for LoongArch.

First it removes unneeded sext(addi.w rd, rs, 0) instructions. Either
because the sign extended bits aren't consumed or because the input was
already sign extended by an earlier instruction.

Then:
1. Unless explicit disabled or the target prefers instructions with W
suffix, it removes the -w suffix from opw instructions whenever all
users are dependent only on the lower word of the result of the
instruction. The cases handled are:
* addi.w because it helps reduce test differences between LA32 and LA64
w/o being a pessimization.

2. Or if explicit enabled or the target prefers instructions with W
suffix, it adds the W suffix to the instruction whenever all users are
dependent only on the lower word of the result of the instruction. The
cases handled are:
   * add.d/addi.d/sub.d/mul.d.
   * slli.d with imm < 32.
   * ld.d/ld.wu.
2024-05-06 16:41:26 +08:00
Yingwei Zheng
d3dad7a74b
[InstCombine] Fix miscompilation caused by #90436 (#91133)
Proof: https://alive2.llvm.org/ce/z/iRnJ4i

Fixes https://github.com/llvm/llvm-project/issues/91127.
2024-05-06 16:30:07 +08:00
Matt Arsenault
eb75af223f Reapply "SystemZ: Fold copy of vector immediate to gr128" (#91099)
This reverts commit a415b4dfcc02e3e82b8c8a7836f7c04b9d65dc9b.

Modify the instruction in place to transform it into a REG_SEQUENCE,
which is what other implementations of foldImmediate do. Also start
erasing the def instruction if there are no other uses.

Fixes #91110.
2024-05-06 10:00:20 +02:00
Jay Foad
e2c89254e1 [AMDGPU] Fix typo in function name 2024-05-06 08:55:09 +01:00
Matt Arsenault
d654278bde
Reapply "AMDGPU: Implement llvm.set.rounding (#88587)" series (#91113)
Revert "Revert 4 last AMDGPU commits to unbreak Windows bots"

This reverts commit 0d493ed2c6e664849a979b357a606dcd8273b03f.

MSVC does not like constexpr on the definition after an extern
declaration of a global.
2024-05-06 09:09:19 +02:00
Yeting Kuo
d70267fbae
[RISCV] Teach .option arch to support experimental extensions. (#89727)
Previously `.option arch` denied extenions are not belongs to RISC-V
features. But experimental features have experimental- prefix, so
`.option arch` can not serve for experimental extension.
This patch uses the features of extensions to identify extension
existance.
2024-05-06 13:55:37 +08:00
Phoebe Wang
f7bfb078cf
[X86][FP16] Do not create VBROADCAST_LOAD for f16 without AVX2 (#91125)
AVX doesn't provide 16-bit BROADCAST instruction.

Fixes #91005
2024-05-06 10:59:44 +08:00
Doug Wyatt
ddecadabeb
[clang backend] In AArch64's DataLayout, specify a minimum function alignment of 4. (#90702)
This addresses an issue where the explicit alignment of 2 (for C++ ABI
reasons) was being propagated to the back end and causing under-aligned
functions (in special sections).

This is an alternate approach suggested by @efriedma-quic in PR #90415.

Fixes #90358
2024-05-05 19:05:15 -07:00
Allen
e12364301e
[AArch64][SelectionDAG] Lower multiplication by a constant to shl+sub+shl+sub (#90199)
Change the costmodel to lower a = b * C where C = 1 - (1 - 2^m) * 2^n to
              sub  w8, w0, w0, lsl #m
              sub  w0, w0, w8, lsl #n
Fix https://github.com/llvm/llvm-project/issues/89430
2024-05-06 08:56:34 +08:00
Fangrui Song
2aaec48d88 X86FixupBWInsts: Remove redundant code. NFC 2024-05-05 15:44:55 -07:00
Kazu Hirata
c18bcd0a57
[Target] Use StringRef::operator== instead of StringRef::equals (NFC) (#91072) (#91138)
I'm planning to remove StringRef::equals in favor of
StringRef::operator==.

- StringRef::operator==/!= outnumber StringRef::equals by a factor of
  38 under llvm/ in terms of their usage.

- The elimination of StringRef::equals brings StringRef closer to
  std::string_view, which has operator== but not equals.

- S == "foo" is more readable than S.equals("foo"), especially for
  !Long.Expression.equals("str") vs Long.Expression != "str".
2024-05-05 13:43:10 -07:00
Florian Hahn
3219c0edb2
[LAA] Directly pass DepChecker to getSource/getDestination (NFC).
Instead of passing LoopAccessInfo only to fetch the MemoryDepChecker,
directly pass MemoryDepChecker. This simplifies the code and also allows
new uses in places where no LAI is available.
2024-05-05 21:16:20 +01:00
DianQK
c6e4f63091
Revert "[InlineCost] Correct the default branch cost for the switch statement (#85160)"
This reverts commit 882814edd33cab853859f07b1dd4c4fa1393e0ea.
2024-05-05 21:54:30 +08:00
Quentin Dian
882814edd3
[InlineCost] Correct the default branch cost for the switch statement (#85160)
Fixes #81723.

The earliest commit of the related code is:
919f9e8d65.
I tried to understand the following code with
https://github.com/llvm/llvm-project/pull/77856#issuecomment-1993499085.

5932fcc478/llvm/lib/Analysis/InlineCost.cpp (L709-L720)

I think only scenarios where there is a default branch were considered.
2024-05-05 21:28:31 +08:00
Weihang Fan
72eaa0ed99
[AArch64][SelectionDAG] Mask for SUBS with multiple users cannot be elided (#90911)
In DAGCombiner, the `performCONDCombine` function attempts to remove AND
instructions in front of SUBS (cmp) instructions for which the AND is
transparent. The rules for that are correct, but it fails to take into
account the case where the SUBS instruction has multiple users with
different condition codes for comparison and simply removes the AND for
all of them. This causes a miscompilation in the attached test case.
2024-05-05 12:01:13 +01:00
Phoebe Wang
7963d9a2b3
[X86][EVEX512] Add HasEVEX512 when NoVLX used for 512-bit patterns (#91106)
With KNL/KNC being deprecated, we don't need to care about such no VLX
cases anymore. We may remove such patterns in the future.

Fixes #90844
2024-05-05 18:40:27 +08:00
David Green
d145f40963 [VectorCombine] shuffleToIdentity - guard against call instructions.
The shuffleToIdentity fold needs to be a bit more careful about the difference
between call instructions and intrinsics. The second can be handled, but the
first should result in bailing out. This patch also adds some extra intrinsic
tests from #91000.

Fixes #91078
2024-05-05 10:47:11 +01:00