7450 Commits

Author SHA1 Message Date
Matt Arsenault
94f73fd6f8 AMDGPU: Code simplification for ctor/dtor lowering
Move the shared global variable lookup into the function.
2022-11-28 15:39:50 -05:00
David Stuttard
7940888c59 [AMDGPU] Intrinsic to expose s_wait_event for export ready
Differential Revision: https://reviews.llvm.org/D138216
2022-11-28 11:26:15 +00:00
Ivan Kosarev
536b8c5377 [AMDGPU][AsmParser] Remove the now-unused OptionalOperand structure.
Became unused in https://reviews.llvm.org/D138492 .
2022-11-28 11:00:05 +00:00
Guillaume Chatelet
702126aec5 [NFC] Add helper method to ensure min alignment on MCSection
Follow up on D138653.

Differential Revision: https://reviews.llvm.org/D138686
2022-11-28 10:00:34 +00:00
Bjorn Pettersson
99c47d9e31 Remove TargetMachine::adjustPassManager
Since opt no longer supports to run default (O0/O1/O2/O3/Os/Oz)
pipelines using the legacy PM, there are no in-tree uses of
TargetMachine::adjustPassManager remaining. This patch removes the
no longer used adjustPassManager functions.

Reviewed By: aeubanks

Differential Revision: https://reviews.llvm.org/D137796
2022-11-28 10:24:16 +01:00
Kazu Hirata
589725f6e8 [llvm] Use std::size (NFC)
std::size, introduced in C++17, allows us to directly obtain the
number of elements of an array.
2022-11-26 13:47:32 -08:00
Kazu Hirata
aad2d272bf [Utils] Use std::optional in AMDGPUBaseInfo.cpp (NFC)
This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-11-25 22:43:00 -08:00
Kazu Hirata
09e0aeaaaa [AMDGPU] Use std::optional in SIPeepholeSDWA.cpp (NFC)
This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-11-25 22:40:00 -08:00
Kazu Hirata
96cb34ab5a [AsmParser] Use std::optional in AMDGPUAsmParser.cpp (NFC)
This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-11-25 22:36:37 -08:00
Kazu Hirata
ebe3281793 [AMDGPU] Use std::optional in AMDGPUReleaseVGPRs.cpp (NFC)
This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-11-25 22:30:26 -08:00
Kazu Hirata
959c9cc7ac [AMDGPU] Use std::optional in AMDGPUInstructionSelector.cpp (NFC)
This is part of an effort to migrate from llvm::Optional to
std::optional:

https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
2022-11-25 22:23:09 -08:00
Dmitry Preobrazhensky
a0a0dc6f8b [AMDGPU][MC] Refactor MC Code Emitter to avoid using magic values
Differential Revision: https://reviews.llvm.org/D138594
2022-11-25 18:02:00 +03:00
Guillaume Chatelet
6c09ea3fdd [Alignment][NFC] Use Align in MCStreamer::emitValueToAlignment
Differential Revision: https://reviews.llvm.org/D138674
2022-11-24 16:09:44 +00:00
Guillaume Chatelet
e647b4f519 [reland][Alignment][NFC] Use the Align type in MCSection
Differential Revision: https://reviews.llvm.org/D138653
2022-11-24 13:19:18 +00:00
Guillaume Chatelet
3467f9c7d6 Revert D138653 [Alignment][NFC] Use the Align type in MCSection"
This breaks the bolt project.
This reverts commit 409f0dc4a420db1c6b259d5ae965a070c169d930.
2022-11-24 12:42:30 +00:00
Guillaume Chatelet
409f0dc4a4 [Alignment][NFC] Use the Align type in MCSection
Differential Revision: https://reviews.llvm.org/D138653
2022-11-24 12:32:58 +00:00
Ivan Kosarev
fce7a7aa9f [AMDGPU][AsmParser] Refine parsing instruction operands.
Eliminates the need for working around optional and token operands being
mistakenly parsed as expressions.

Reviewed By: dp

Differential Revision: https://reviews.llvm.org/D138492
2022-11-24 10:50:26 +00:00
Ivan Kosarev
ec8ede8177 [AMDGPU][CodeGen] Support raw format TFE buffer loads other than byte, short and d16 ones.
Differential Revision: https://reviews.llvm.org/D138215
2022-11-24 10:50:26 +00:00
Alex Richardson
88218d5c52 [SelectionDAG] Remove deprecated MemSDNode->getAlignment()
I noticed a an assertion error when building MIPS code that loaded from
NULL. Loading from NULL ends up being a load with maximum alignment, and
due to integer truncation the value maximum was interpreted as 0 and the
assertion in MipsDAGToDAGISel::Select() failed. This previously happened
to work, but the maximum alignment was increased in
df84c1fe78130a86445d57563dea742e1b85156a, so it no longer fits into a 32
bit integer.
Instead of just fixing the one MIPS case, this patch removes all uses of
the deprecated getAlignment() call and replaces them with getAlign().

Differential Revision: https://reviews.llvm.org/D138420
2022-11-23 09:04:42 +00:00
Ruiling Song
0eaf6759ae [AMDGPU][InsertWaits] No wait for WAW for global/scratch_load
global/scratch_load will return in order they are issued. No
need to insert a s_waitcnt for WAW hazard.

Reviewed By: foad

Differential Revision: https://reviews.llvm.org/D138476
2022-11-23 09:57:50 +08:00
Jay Foad
bc0c89d3d8 [AMDGPU] More cleanup after D117544. NFC. 2022-11-22 21:15:22 +00:00
Alexander Timofeev
48ab3e7527 [AMDGPU] Avoid SCC clobbering before S_CSELECT_B32
Frame lowering inserts scalar addition to compute the offset to the
stack objects.  This instructions inserted in arbitrary place and may clobber
SCC between its definition and S_CSELECT_B32 instruction.  This change
workarounds this particular code pattern. It queries the scavenger for SGPR and
if available saves SCC to it and restore its value after frame lowering code
insertion.

Reviewed By: foad

Differential Revision: https://reviews.llvm.org/D136169
2022-11-22 19:08:04 +01:00
Jay Foad
6ed6e8e3b8 [AMDGPU] Remove RegStrictDom variable. NFC.
D117544 removed the only substantive use of RegStrictDom. Now we can
simplify by using StrictDom for everything.
2022-11-22 17:08:00 +00:00
Jay Foad
dce7c09bd4 [AMDGPU] Define and use new allZeroWaitcnt helper. NFC. 2022-11-22 17:01:36 +00:00
Pierre van Houtryve
4d39552abe [AMDGPU][NFC] Remove isLegalVOP3PShuffleMask
Unused function since D134967

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D138493
2022-11-22 14:31:59 +00:00
Pierre van Houtryve
9e7febb4f7 [AMDGPU][GISel] Select llvm.amdgcn.fcmp intrinsics
Adds FP CCs opcodes/selection logic, including src mods selection

Depends on D136591, D136448
Resolves #58326 (https://github.com/llvm/llvm-project/issues/58326)

Reviewed By: arsenm, foad

Differential Revision: https://reviews.llvm.org/D136592
2022-11-22 14:18:58 +00:00
Pierre van Houtryve
220147d536 [AMDGPU] Make aperture registers 64 bit
Makes the SRC_(SHARED|PRIVATE)_(BASE|LIMIT) registers 64 bit instead of 32.
They're still usable as 32 bit operands by using the _LO suffix.

Preparation for D137542

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D137767
2022-11-22 09:17:59 +00:00
Pierre van Houtryve
a751676f98 [AMDGPU][GISel] Add llvm.amdgcn.icmp selection
Add missing logic to select i16 variants and enable GISel testing.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D136448
2022-11-22 08:26:50 +00:00
Kazu Hirata
7524db4d44 [llvm] Remove unused forward declarations (NFC) 2022-11-20 09:59:36 -08:00
Matt Arsenault
b0847b0095 AMDGPU/GlobalISel: Insert freeze when splitting vector G_SEXT_INREG
This transform is broken for undef or poison inputs without a freeze.
This is also broken in lots of other places where shifts are split
into 32-bit pieces.

Amt < 32 case:
; Broken: https://alive2.llvm.org/ce/z/7bb4vc
; Freezing the low half of the bits makes it correct
; Fixed: https://alive2.llvm.org/ce/z/zJAZFr
define i64 @src(i64 %val) {
  %shl = shl i64 %val, 55
  %shr = ashr i64 %shl, 55
  ret i64 %shr
}

define i64 @tgt(i64 %val) {
  %lo32 = trunc i64 %val to i32
  %shr.half = lshr i64 %val, 32
  %hi32 = trunc i64 %shr.half to i32
  %inreg.0 = shl i32 %lo32, 23
  %new.lo = ashr i32 %inreg.0, 23
  %new.hi = ashr i32 %new.lo, 31
  %zext.lo = zext i32 %new.lo to i64
  %zext.hi = zext i32 %new.hi to i64
  %hi.ins = shl i64 %zext.hi, 32
  %or = or i64 %hi.ins, %zext.lo
  ret i64 %or
}

Amt == 32 case:
Broken: https://alive2.llvm.org/ce/z/5f4qwQ
Fixed: https://alive2.llvm.org/ce/z/A2hWWF
This one times out alive; works if argument is made noundef or
scaled down to a smaller bitwidth.

define i64 @src(i64 %val) {
  %shl = shl i64 %val, 32
  %shr = ashr i64 %shl, 32
  ret i64 %shr
}

define i64 @tgt(i64 %val) {
  %lo32 = trunc i64 %val to i32
  %shr.half = lshr i64 %val, 32
  %hi32 = trunc i64 %shr.half to i32
  %new.hi = ashr i32 %lo32, 31
  %zext.lo = zext i32 %lo32 to i64
  %zext.hi = zext i32 %new.hi to i64
  %hi.ins = shl i64 %zext.hi, 32
  %or = or i64 %hi.ins, %zext.lo
  ret i64 %or
}

Amt > 32 case:
; Correct: https://alive2.llvm.org/ce/z/tvrhPf
define i64 @src(i64 %val) {
  %shl = shl i64 %val, 9
  %shr = ashr i64 %shl, 9
  ret i64 %shr
}

define i64 @tgt(i64 %val) {
  %lo32 = trunc i64 %val to i32
  %lshr = lshr i64 %val, 32
  %hi32 = trunc i64 %lshr to i32
  %inreg.0 = shl i32 %hi32, 9
  %new.hi = ashr i32 %inreg.0, 9
  %zext.lo = zext i32 %lo32 to i64
  %zext.hi = zext i32 %new.hi to i64
  %hi.ins = shl i64 %zext.hi, 32
  %or = or i64 %hi.ins, %zext.lo
  ret i64 %or
}
2022-11-18 16:05:19 -08:00
Matt Arsenault
1fe1299a93 GlobalISel: Legalize strict_fsub
In the future should probably have a more convenient
way to switch between building strict and non-strict ops.
2022-11-18 15:21:41 -08:00
Matt Arsenault
fe56afc4d7 AMDGPU: Fix fcanonicalize constant folding not correctly handling -0.0 2022-11-18 10:03:29 -08:00
Mirko Brkusanin
e58b116843 [AMDGPU] Add subtarget feature for MAD_U64/I64 bug on GFX11
Differential Revision: https://reviews.llvm.org/D133012
2022-11-18 18:19:27 +01:00
Petar Avramovic
0f3e72e86c AMDGPU/GlobalISel: Fix crash after mad/fma_mix fails selection
When selectVOP3PMadMixModsImpl fails, it can still create new copy instr
via selectVOP3ModsImpl. When selectG_FMA_FMAD gives up, new copy instr
will remain dead but will not be automatically removed.
InstructionSelect does not check if instructions created during selection
are dead.
Such dead copy doesn't have register class on dst operand and causes crash.
Fix is to build copy when operands are being added to selected instruction.

Differential Revision: https://reviews.llvm.org/D138044
2022-11-18 18:02:26 +01:00
Jay Foad
38302c60ef [AMDGPU] Stop looking for implicit M0 uses on MOV instructions
Before D114230, indirect moves used regular MOV opcodes and were
identified by having an implicit use of M0. Since D114230 they use
dedicated opcodes instead, so remove some old code that checks for
implicit uses of M0. NFCI.

Differential Revision: https://reviews.llvm.org/D138308
2022-11-18 16:57:55 +00:00
Matt Arsenault
08ec15e44b AMDGPU/GlobalISel: Fix strictfp fmul 2022-11-18 08:53:49 -08:00
Alexander Timofeev
32bd75716c PEI should be able to use backward walk in replaceFrameIndicesBackward.
The backward register scavenger has correct register
liveness information. PEI should leverage the backward register scavenger.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D137574
2022-11-18 15:57:34 +01:00
Dmitry Preobrazhensky
96155bf44b [AMDGPU][GFX11][NFC] Refactor VOPD operands handling (part 2)
Rename interface functions and operands to make code clearer.

Differential Revision: https://reviews.llvm.org/D138133
2022-11-18 14:15:05 +03:00
Valery Pykhtin
a35ba2a256 [AMDGPU] Fix PreRARematStage::sinkTriviallyRematInsts region boundary update after sinking.
First boundary of a region wasn't updated when a sinked instruction was added first into the region.

Reviewed By: vangthao

Differential Revision: https://reviews.llvm.org/D138256
2022-11-18 12:13:14 +01:00
Matt Arsenault
fe5b9a6a11 AMDGPU/GlobalISel: Make strict fadd, fmul and fma legal 2022-11-17 20:50:04 -08:00
Matt Arsenault
ae43420f39 AMDGPU/GlobalISel: Fix not selecting modifiers for f16 fma on gfx9
VOP3OpSel wasn't trying to match any modifiers. Just try to match the
basic case, like the DAG does.
2022-11-17 18:51:45 -08:00
Stanislav Mekhanoshin
bcaf31ec3f [AMDGPU] Allow finer grain control of an unaligned access speed
A target can return if a misaligned access is 'fast' as defined
by the target or not. In reality there can be different levels
of 'fast' and 'slow'. This patch changes the boolean 'Fast'
argument of the allowsMisalignedMemoryAccesses family of functions
to an unsigned representing its speed.

A target can still define it as it wants and the direct translation
of the current code uses 0 and 1 for current false and true. This
makes the change an NFC.

Subsequent patch will start using an actual value of speed in
the load/store vectorizer to compare if a vectorized access going
to be not just fast, but not slower than before.

Differential Revision: https://reviews.llvm.org/D124217
2022-11-17 09:23:53 -08:00
Jay Foad
49762162ea [AMDGPU] Remove isLiteralConstant and isLiteralConstantLike
isLiteralConstant and isLiteralConstantLike were similar to
!isInlineConstant with slight differences like handling isReg operands.

To avoid a profusion of similar functions with undocumented differences,
this patch removes all the isLiteralConstant* variants. Callers are responsible
for handling the isReg case.

Differential Revision: https://reviews.llvm.org/D125759
2022-11-17 16:45:48 +00:00
Yashwant Singh
2652db4d68 Handling ADD|SUB U64 decomposed Pseudos not getting lowered to SDWA form
This patch fixes some of the V_ADD/SUB_U64_PSEUDO not getting converted to their sdwa form.
We still get below patterns in generated code:
v_and_b32_e32 v0, 0xff, v0
v_add_co_u32_e32 v0, vcc, v1, v0
v_addc_co_u32_e64 v1, s[0:1], 0, 0, vcc

and,
v_and_b32_e32 v2, 0xff, v2
v_add_co_u32_e32 v0, vcc, v0, v2
v_addc_co_u32_e32 v1, vcc, 0, v1, vcc

1st and 2nd instructions of both above examples should have been folded into sdwa add with BYTE_0 src operand.

The reason being the pseudo instruction is broken down into VOP3 instruction pair of V_ADD_CO_U32_e64 and V_ADDC_U32_e64.
The sdwa pass attempts lowering them to their VOP2 form before converting them into sdwa instructions. However V_ADDC_U32_e64
cannot be shrunk to it's VOP2 form if it has non-reg src1 operand.
This change attempts to fix that problem by only shrinking V_ADD_CO_U32_e64 instruction.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D136663
2022-11-17 10:01:40 +05:30
Matt Arsenault
3830e4e58c AMDGPU: Create poison values instead of undef
These placeholders don't care about the finer points on
the difference between the two.
2022-11-16 14:47:24 -08:00
Dmitry Preobrazhensky
e468b1b740 [AMDGPU][GFX11] Refactor VOPD operands handling
Differential Revision: https://reviews.llvm.org/D137952
2022-11-16 16:29:12 +03:00
Jay Foad
342642dc75 [AMDGPU][GISel] Smaller code for scalar 32 to 64-bit extensions
Differential Revision: https://reviews.llvm.org/D107639
2022-11-16 06:57:21 +00:00
Christudasan Devadasan
349b6afef4 [AMDGPU] Remove the assertion for MUBUF instruction with voffset
Currently, there is an assertion that limits the MUBUF instruction
with voffset used for a VGPR spill inside kernel functions when
the frame pointer is really used. It seemed more like an unwanted
limitation and hence removing the assertion.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D137892
2022-11-16 12:06:37 +05:30
Fangrui Song
6c7666a408 Revert D137574 "PEI should be able to use backward walk in replaceFrameIndicesBackward."
This reverts commit e05ce03cfa0b36e9b99149e21afcb1fc039df813.

Caused asan use-after-poison to 4 DebugInfo/AMDGPU/ tests.
Triggered in PEI::replaceFrameIndicesBackward called llvm::MachineInstr::getNumOperands
2022-11-15 19:19:46 +00:00
Alexander Timofeev
e05ce03cfa PEI should be able to use backward walk in replaceFrameIndicesBackward.
The backward register scavenger has correct register
liveness information. PEI should leverage the backward register scavenger.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D137574
2022-11-15 15:20:25 +01:00