Use an `OptimizationRemark` for them even though it's not really an
optimization. It just integrates better with the other diagnostics
(enabling is easy with `-pass-remark`).
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D147703
Summary:
This is to avoid using the callee saved registers for the return address
of the tail call return instruction.
Reviewers:
arsenm, cdevadas
Differential Revision:
https://reviews.llvm.org/D147096
The test was added in D69953, but since D67794 landed a bit later it no
longer catches the bug. Update the test to be representative again.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D146934
GlobalISel happens to insert some constant materializes before SI_END_CF
in one test. These need to be excluded from AliveBlocks since they
are defined in the original block and used in the split block,
so they aren't fully alive through either block.
The case where the value defined in the first block which was originally used
in a later block is still broken.
Avoids a verifier error in a future patch.
set_inactive is actually a kind of operation that is passing certain
value from active threads to inactive threads. In later WWM operation,
the activated threads which were disabled before would read such
values passed to them by set_inactive operation. So I think the
set_inactive is a convergent operation.
Differential Revision: https://reviews.llvm.org/D147683
The set_inactive was originally inside a if branch, but was incorrectly
hoisted out of the loop. So the lanes that were disabled later after the
if() branch inside the loop were not set correctly. It should stay in
the if branch.
Differential Revision: https://reviews.llvm.org/D147682
The inverse ballot intrinsic takes in a boolean mask for all lanes and
returns the boolean for the current lane. See SPIR-V's
`subgroupInverseBallot()` in the [[ https://github.com/KhronosGroup/GLSL/blob/master/extensions/khr/GL_KHR_shader_subgroup.txt | GL_KHR_shader_subgroup extension ]].
This allows decision making via branch and select instructions with a manually
manipulated mask.
Implemented in GlobalISel and SelectionDAG, since currently both are supported.
The SelectionDAG required pseudo instructions to use the custom inserter.
The boolean mask needs to be uniform for all lanes.
Therefore we expect SGPR input. In case the source is in a
VGPR, we insert one or more `v_readfirstlane` instructions.
Reviewed By: nhaehnle
Differential Revision: https://reviews.llvm.org/D146287
This was assuming all register operands were assigned to physical registers.
This should ignore the operands which weren't assigned in this run.
Fixes#61134
Pass disabled since approximately D104962 for miscompiling openmp
The functions under ReplaceConstant miscompile phis as noted in D112717 and
have no users in tree other than the disabled pass. It seems likely it has no
users out of tree.
Deletes the test cases associated with the disabled pass as well.
Reviewed By: rampitec
Differential Revision: https://reviews.llvm.org/D147586
The premise here is to allow non-kernel functions to locate external LDS variables without using LDS or extra magic SGPRs to do so.
1/ First it crawls the callgraph to work out which external LDS variables are reachable from a given kernel
2/ Then it creates a new `extern char[0]` variable for each kernel, which will alias all the other extern LDS variables because that's the documented behaviour of these variables
3/ The address of that variable is written to a lookup table. The global variable is tagged with metadata to track what address it was allocated at by codegen
4/ The assembler builds the lookup table using the metadata
5/ Any non-kernel functions use the same magic intrinsic used by table lookups of non-dynamic LDS variables to find the address to use
Heavy overlap with the code paths taken for other lowering, in particular the same intrinsic is used to pass the dynamic scope information through the same sgpr as for table lookups of static LDS.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D144233
Summary:
This is to avoid using the callee saved registers for the return address of the tail call return instruction.
Reviewers:
arsenm, cdevadas
Differential Revision:
https://reviews.llvm.org/D147096
This reverts commit 64b45db34a0cd979dae9ca3016e9da517e57b987.
Reason: the patterns are wrong which can result in a miscompilation.
However, fixing the pattern is not trivial due to how i8 values
are handled, and due to the additional type-checking performed by
D147127: trunc/smax/smin are all defined as int ops in the DAG
despite them working on vectors too.
As this is not a much-needed pattern, I prefer reverting for now
until I can find time to properly rewrite the pattern.
Since `RK::Recs` is sorted by character order, anonymous defs will be
enumerated like this;
- anonymous_0
- anonymous_1
- anonymous_10
- anonymous_100
- anonymous_1000
- ...
- anonymous_99
- anonymous_990
- ...
- anonymous_999
Some records around each gap might be wrapped around along increase or
decrease of records in middle. Then output order of anonymous defs
might be changed.
Numeric sort is expected to prevent such wrap-arounds.
This can be implemented with `StringRef::compare_numeric()`.
- ...
- anonymous_99
- anonymous_100
- ...
- anonymous_999
- anonymous_1000
- ...
See also discussions in D145874.
Differential Revision: https://reviews.llvm.org/D145874
Define and use a MachineOperand overload of isPlainlyKilled. This
improves codegen in a couple of tests because it catches the case where
MO does not kill Reg but another operand of the same instruction does.
Differential Revision: https://reviews.llvm.org/D147167
Summary:
This is to avoid using the callee saved registers for the return address of the tail call return instruction.
Reviewers:
arsenm, cdevadas
Differential Revision:
https://reviews.llvm.org/D147096
Allows allocas with memset users to be promoted.
This is intended to prevent patterns such as `memset(&alloca, 0, sizeof(alloca))` (which I think can be emitted by frontends) from preventing a vectorization of allocas.
Fixes SWDEV-388784
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D146225
DAGISel uses CopyToReg/CopyFromReg to lower PHI nodes. With large PHIs, this can result in poor codegen.
This is because it introduces a need to have a build_vector before copying the PHI value, and that build_vector may have many undef elements. This can cause very high register pressure and abnormal stack usage in some cases.
This scalarization/phi "break-up" can be easily tuned/disabled through CL options in case it's not beneficial for some users.
It's also only enabled for DAGIsel and GlobalISel handles PHIs much better (as it works on the whole function).
This can both scalarize (break a vector into its elements) and simplify (break a vector into smaller, more manageable subvectors) PHIs.
Fixes SWDEV-321581
Reviewed By: kzhuravl
Differential Revision: https://reviews.llvm.org/D143731
Generalize `tryFoldLCSSAPhi` into `tryFoldPhiAGPR` which works
on any kind of PHI node (not just LCSSA ones) and attempts to
create AGPR Phis more aggressively.
Also adds a GFX908-only "cleanup" function `tryOptimizeAGPRPhis`
which tries to minimize AGPR to AGPR copies on GFX908, which doesn't
have a ACCVGPR MOV instruction (so AGPR-AGPR copies become 2 or 3 instructions
as they need a VGPR temp). The reason why this is needed is because D143731
+ the new `tryFoldPhiAGPR` may create a lot more PHIs (one 32xfloat PHI becomes
32 float phis), and if each PHI hits the same AGPR (like in `test_mfma_loop_agpr_init`)
they will be lowered to 32 copies from the same AGPR, which will each become 2-3 instructions.
Creating a VGPR cache in this case prevents all those copies from being generated
(we have AGPR-VGPR copies instead which are trivial).
This is a prepation patch intended to prevent regressions in D143731 when
AGPRs are involved.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D144099
AMDGPUPrintfRuntimeBindingPass is not run in the IR optimization
pipeline with -O0.
This means that with OpenCL the printf definition coming from
device_libs gets linked with the user's code, which blocks
AMDGPUPrintfRuntimeBindingPass from working after the linkage is done.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D146720
The legacy pass is only used in AMDGPU codegen, which doesn't care about running it in call graph order (it actually has to work around that fact).
Make the legacy pass a module pass and share code with the new pass.
This allows us to remove the legacy inliner infrastructure.
Reviewed By: mtrofin
Differential Revision: https://reviews.llvm.org/D146446
ASMPrinter was relying on feature bits to setup extra SGRPs in the knerel
descriptor for the xnack_mask. This was broken for the dynamic XNACK "any" TID
setting which could cause user SGPRs to be clobbered if the number of SGPRs
reserved was near a granulated block boundary.
When XNACK was enabled this worked correctly in the ASMParser which meant some
kernels were only failing without "-save-temps".
Fixes: SWDEV-382764
Reviewed By: kzhuravl
Differential Revision: https://reviews.llvm.org/D145401
For D141247 - if that pattern was used by GISel it could cause constant bus limitation failures.
Just use inline immediates instead of S_MOV to avoid the issue.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D146131
Adds & uses a new `isDivergentUse` API in UA.
UniformityAnalysis now requires CycleInfo as well as the new temporal divergence API can query it.
-----
Original patch that adds `isDivergentUse` by @sameerds
The user of a temporally divergent value is marked as divergent in the
uniformity analysis. But the same user may also have been marked divergent for
other reasons, thus losing this information about temporal divergence. But some
clients need to specificly check for temporal divergence. This change restores
such an API, that already existed in DivergenceAnalysis.
Reviewed By: sameerds, foad
Differential Revision: https://reviews.llvm.org/D146018
The backend knew about `v_sat_pk_u8_i16` but never made use of it.
This patch adds selection patterns (DAG/GISel) for that instruction.
I think it'll be very rarely used, but at least it's possible to use it.
Solves #58266 (https://github.com/llvm/llvm-project/issues/58266)
Reviewed By: foad
Differential Revision: https://reviews.llvm.org/D144729
Cover more cases in preparation for making greater use
of fcmp based lowerings. Also add more tests for the inverted
cases. Test iszero | isnan test masks. We should probably just
generate every combination of test masks.
