The new format matches the official ISA spec and ensures the
disassembler prints 'export mrt0, v0, off, off, off' instead of 'export
mrt0 v0, off, off, off'.
No functional encoding changes; printing/AsmString only.
This change landed without approval.
This reverts commit 45e666a8531c1148bdb170b9a120f99e1500c427.
This reverts commit a636dd4c37f12594275de2fe180ca35bc04d76ea.
Clang isn't able to support multiple constraints on inputs and outputs,
like "rm". Instead, it picks the "safest" one to use, i.e. the memory
constraint for "rm". This leads to obviously horrible code:
asm __volatile__ ("pushf\n\t"
"popq %0"
: "=rm" (x));
is compiled to:
pushf
popq -8(%rsp)
movq -8(%rsp), %rax
It gets worse when inlined into other functions, because it may
introduce
a stack where none is needed.
With this change, Clang now generates IR for the more optimistic choice
("r"). All but the fast register allocator are able to fold registers if
it turns out that register pressure is too high.
This leaves the fast register allocator. The fast register allocator, as
the name suggests, is built for execution speed, not code quality. Thus,
we add special processing to convert the "optimistic" IR into the
"conservative" choice (again at the IR level), which we know it can
handle.
We focus on "rm" for the initial commit, but that can be expanded in the
future for other constraints where Clang generates ++ungood code (like
"g").
Fixes: https://github.com/llvm/llvm-project/issues/20571
Currently, the compiler doesn't create a COPY for undef operands while
lowering REG_SEQUENCE, and only if LIS information is available, it
propagates the undef flag to the subreg uses. So, if LIS isn't
available, we can end up with some uses without def of those lanes.
Now, we check which lanes are used in a single scan of
use_nodbg_operands() per REG_SEQ, and perform the skip of the COPY only
if LIS is avaible (as undef will be propagated later) or if there are no
uses for that lane.
There is still a scan of the use list, but now it's only one per REG_SEQ
and I think it's necessary, as there is no guarantee to have LIS or
other analysis pass information at this stage.
This is a proposal fix for issue:
https://github.com/llvm/llvm-project/issues/175596
---------
Co-authored-by: Carl Ritson <critson@perlfu.co.uk>
This fixes an issue in the scheduler's rematerialization stage wherein
slot indices could end up being inconsistent with respect to MI order
following a scheduling revert coupled with rematerialization rollbacks.
When a region's schedule is reverted, instructions are re-ordered into
the region's original instruction order. Furthermore, non-debug
instructions have their slot moved if necessary to maintain consistency
between MI and slot order. Importantly, the rematerialization stage
marks rematerialized MIs as debug during re-scheduling. When reverting
the schedule of regions that have had some of their MIs rematerialized,
the latter are still marked debug, and therefore maintain their
pre-re-scheduling slot even if it no longer is consistent with MI order.
This fixes the issue by restoring the original state of rematerialized
MIs before reverting region schedules, ensuring they are moved if
necessary. An EXPENSIVE_CHECK is added to check that slot order
makes sense with respect to MI order in re-scheduled regions.
Fixes SWDEV-579187.
This optimisation is motivated by [this minimal
example](https://godbolt.org/z/9o83GvGsM).
Particularly, if `mask` is a 64-bit integer, a select
```cpp
auto out = ((mask >> 32) != 0) ? a : b;
```
converts the null-check on the higher 32-bits of `mask` into a
`v_cmp_lt_u64` with the integer `1 << 32` stored in a pair of VGPRs
(effectively wasting two VGPRs).
More generally, if a 64-bit integer (whose lower 32 bits are known to be
irrelevant) is compared with a 64-bit constant, two VGPRs are wasted to
construct this constant.
This patch modifies ISel to take advantage of how 64-bit values are
stored in pairs of VGPRs (or SGPRs), and truncates the 64-bit constant
to its upper 32-bit constant where possible.
Alive2 proof for analogous middle-end transformation:
https://alive2.llvm.org/ce/z/zizKms
Add support for expanding fptosi.sat and fptoui.sat via IR expansions.
Similar to fptosi/fptoui we would get legalization errors otherwise.
The previous expansion for fptosi/fptoui was already saturating -- but
those instructions do not actually require saturation, and the
implementation of the saturation was incorrect in lots of ways. What
this PR does is:
* For fptosi, remove the unnecessary saturation handling.
* For fptoui, remove the unnecessary saturation handling and sign
multiplication.
* For fptosi, use the previous saturation handling with fixes: We need
to map NaNs to 0 and the saturation condition on the exponent was
incorrect. (I'm performing the NaN check via fcmp -- there's no
requirement to do everything bitwise here.)
* For fptoui use a variation of the signed saturation handling: Negative
values need to go to zero and we saturate to unsigned max.
Proofs: https://alive2.llvm.org/ce/z/Xv9FNd
Asynchronous operations are memory transfers (usually between the global
memory and LDS) that are completed independently at an unspecified
scope. A thread that requests one or more asynchronous transfers can use
async marks to track their completion. The thread waits for each mark to
be completed, which indicates that requests initiated in program order
before this mark have also completed.
For now, we implement asyncmark/wait operations on pre-GFX12
architectures that support "LDS DMA" operations. Future work will extend
support to GFX12Plus architectures that support "true" async operations.
This is part of a stack split out from #173259
- #180467
- #180466
Co-authored-by: Ryan Mitchell ryan.mitchell@amd.com
Fixes: SWDEV-521121
The existing "LDS DMA" builtins/intrinsics copy data from global/buffer
pointer to LDS. These are now augmented with their ".async" version,
where the compiler does not automatically track completion. The
completion is now tracked using explicit mark/wait intrinsics, which
must be inserted by the user. This makes it possible to write programs
with efficient waits in software pipeline loops. The program can now
wait for only the oldest outstanding operations to finish, while
launching more operations for later use.
This change only contains the new names of the builtins/intrinsics,
which continue to behave exactly like their non-async counterparts. A
later change will implement the actual mark/wait semantics in
SIInsertWaitcnts.
This is part of a stack split out from #173259:
- #180467
- #180466
Fixes: SWDEV-521121
`updateAsyncLDSAddress`, introduces by
https://github.com/llvm/llvm-project/pull/176816, previously only
handled async loads , where the LDS address is in the `vdst` operand.
Therefore Async stores produced a nullptr dereference since the LDS
address is in `vdata` for those instructions.
---------
Co-authored-by: Jay Foad <jay.foad@gmail.com>
Two code paths in `reassociationCanBreakAddressingModePattern` were
missing a `hasUniqueMemOperand()` guard before calling
`getAddressSpace()`. Note that on `L1214` we already have the same guard
in place.
`getAddressSpace()` chains through `getPointerInfo()` to
`getMemOperand()`, which asserts that the node has exactly one memory
operand.
Adds rules for G_ATOMICRMW_{MAX, MIN, UMAX, UMIN, UINC_WRAP, UDEC_WRAP}.
Each of these generic opcode are supported for S32 and S64 types
on flat, global and local address spaces.
Start trying to use SimplifyDemandedFPClass on instructions, starting
with fmul. This subsumes the old transform on multiply of 0. The
main change is the introduction of nnan/ninf. I do not think anywhere
was systematically trying to introduce fast math flags before, though
a few odd transforms would set them.
Previously we only called SimplifyDemandedFPClass on function returns
with nofpclass annotations. Start following the pattern of
SimplifyDemandedBits, where this will be called from relevant root
instructions.
I was wondering if this should go into InstCombineAggressive, but that
apparently does not make use of InstCombineInternal's worklist.
Make it possible to use `s_alloc_vgpr` at the IR level. This is a huge
footgun and use for anything other than compiler internal purposes is
heavily discouraged. The calling code must make sure that it does not
allocate fewer VGPRs than necessary - the intrinsic is NOT a request to
the backend to limit the number of VGPRs it uses (in essence it's not so
different from what we do with the dynamic VGPR flags of the
`amdgcn.cs.chain` intrinsic, it just makes it possible to use this
functionality in other scenarios).
Sinking ShuffleVectors / ExtractElement / InsertElement into user blocks
can help enable SDAG combines by providing visibility to the values
instead of emitting CopyTo/FromRegs. The sink IR pass disables sinking
into loops, so this PR extends the CodeGenPrepare target hook
shouldSinkOperands.
Co-authored-by: Jeffrey Byrnes <Jeffrey.Byrnes@amd.com>
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
`getLit64Encoding` uses a different approach to determine whether 64-bit
literal encoding is used, which caused a size mismatch between the
`MachineInstr` and the `MCInst`.
For `!isValid32BitLiteral`, it is effectively `!(isInt<32>(Val) ||
isUInt<32>(Val))`, which is `!isInt<32>(Val) && !isUInt<32>(Val)`, but
in `getLit64Encoding`, it is `!isInt<32>(Val) || !isUInt<32>(Val)`.
The definition for V_INDIRECT_REG_READ_GPR_IDX_B32_V*'s SSrc_b32 operand
allows immediates, but the expansion logic handles only register cases
now. This can result in expansion failures when e.g.
llvm.amdgcn.wave.reduce.umin.i32 is folded into a constant and then used
as an insertelement idx.
Load monitor operations make more sense as atomic operations, as
non-atomic operations cannot be used for inter-thread communication w/o
additional synchronization.
The previous built-in made it work because one could just override the
CPol bits, but that bypasses the memory model and forces the user to learn
about ISA bits encoding.
Making load monitor an atomic operation has a couple of advantages.
First, the memory model foundation for it is stronger. We just lean on the
existing rules for atomic operations. Second, the CPol bits are abstracted away
from the user, which avoids leaking ISA details into the API.
This patch also adds supporting memory model and intrinsics
documentation to AMDGPUUsage.
Solves SWDEV-516398.
Exactly match the s_wait_event instruction. For some reason we already
had this instruction used through llvm.amdgcn.s.wait.event.export.ready,
but that hardcodes a specific value. This should really be a bitmask
that
can combine multiple wait types.
gfx11 -> gfx12 broke compatabilty in a weird way, by inverting the
interpretation of the bit but also shifting the used bit by 1. Simplify
the selection of the old intrinsic by just using the magic number 2,
which should satisfy both cases.
This PR fixes#177753, converting disjoint S_OR_B32 to S_ADDK_I32
whenever possible, it avoids this transformation in case S_OR_B32 can be
converted to bitset.
Note on Test Failures (Draft Status) This change causes significant
register reshuffling across the test suite due to the new allocation
hints and the swaps performed in case src0 is not a register and src1,
along with the change from or to addk. To avoid a massive, noisy diff
during the initial logic review:
This Draft PR only includes a representative sample of updated tests.
CodeGen/AMDGPU/combine-reg-or-const.ll -> Showcases change from S_OR to
S_ADDK
CodeGen/AMDGPU/s-barrier.ll -> Showcases swap between Src0 and Src1 if
src0 is not a register
The rest of the tests show the result of the register allocation hint we
give, I have checked every test I updated and they seem ok to me.
Once the core logic is approved, I will run the update script across the
remaining ~70 failing tests and mark the PR as "Ready for Review."
Select (fsub x, y) -> (fma y, -1.0, x). Using -1.0 as the constant
avoids the need for ComplexPatterns to negate x or y.
This also fixes the bad pattern (fsub x, y) -> (fma -x, 1.0, y).
