This moves the responsibility for cleaning up dead intrinsics from
DXILFinalizeLinkage to DXILOpLowering, and moves DXILFinalizeLinkage
back to it's pre-#136244 place in the pipeline. Doing this avoids issues
with DXIL passes running on obviously dead code, and makes it more clear
what DXILFinalizeLinkage is really doing.
This also helps with the story for #134260, as cleaning up dead
intrinsics doesn't make sense if this becomes a more generic pass.
Note that test/CodeGen/DirectX/remove-dead-intriniscs.ll already covers
most of the testing here. It'd be nice to have something that catches
the regression from changing the pass ordering but I couldn't come up
with anything that wouldn't be incredibly fragile.
Fixes#138180.
Pull invariant load identification, which was previously part of
DAGToDAG ISel, into a new IR pass NVPTXTagInvariantLoads. This makes it
possible to disable this optimization at O0 and reduces the complexity
of the SelectionDAG pass. Moving this logic to an IR pass also allows
for implementing a more powerful traversal in the future.
Fixes https://github.com/llvm/llvm-project/issues/138138
Implement proper splitting functions for PARTIAL_REDUCE_MLA ISD nodes.
This makes the udot_8to64 and sdot_8to64 tests generate dot product
instructions for when the new ISD nodes are used.
---------
Co-authored-by: James Chesterman <james.chesterman@arm.com>
The default maximum waves/EU returned by the family of
`AMDGPUSubtarget::getWavesPerEU` is currently the maximum number of
waves/EU supported by the subtarget (only a valid occupancy range in
"amdgpu-waves-per-eu" may lower that maximum). This ignores maximum
achievable occupancy imposed by flat workgroup size and LDS usage,
resulting in situations where `AMDGPUSubtarget::getWavesPerEU` produces
a maximum higher than the one from
`AMDGPUSubtarget::getOccupancyWithWorkGroupSizes`.
This limits the waves/EU range's maximum to the maximum achievable
occupancy derived from flat workgroup sizes and LDS usage. This only has
an impact on functions which restrict flat workgroup size with
"amdgpu-flat-work-group-size", since the default range of flat workgroup
sizes achieves the maximum number of waves/EU supported by the
subtarget.
Improvements to the handling of "amdgpu-waves-per-eu" are left for a
follow up PR (e.g., I think the attribute should be able to lower the
full range of waves/EU produced by these methods).
If the SDNode is used it can pick up the wrong results number, for
example looking at the known bits of the first result where it should be
looking at the second. The SDValue is already present as the
SelectCodeCommon checks move from parent to child, pass the SDValue
through to CheckNodePredicate as Op so that it can use it if necessary.
SDNode *N is still generated, keeping most PatFrags the same.
Fixes#137274
Ensure that when custom lowering a vector load/store to a multi-output
load/store node we confirm that the memory value type matches the type
used by the node. Also add some asserts for basic sanity checking of
load size.
Fixes https://github.com/llvm/llvm-project/issues/138034
Fixes https://github.com/llvm/llvm-project/issues/130020
This fixes an issue where the si-fold-operands pass would incorrectly
fold immediate values into COPY instructions targeting av_32 registers.
The pass now checks register class constraints before attempting to fold
the immediate.
The aarch64-ldst-opt pass tries to merge two load instructions
(LDR*) to a load pair instruction (LDP*).
When merging the instructions, there is a case where one of the
loads would have to also be sign extended. In either case,
(sign extend or not), the pass needs to preserve the debug-instr-number
from the original loads to the load pair instruction to make sure debug
info
isn't lost in the case where instruction referencing is being used.
For example:
We can have something like this:
```
debugValueSubstitutions:[]
$x1 = LDRXui $x0, 1, debug-instr-number 1
DBG_INSTR_REF !13, dbg-instr-ref(1, 0), debug-location !11
$x0 = LDRXui killed $x0, 0, debug-instr-number 2
DBG_INSTR_REF !14, dbg-instr-ref(2, 0), debug-location !11
```
This would be changed to:
```
debugValueSubstitutions: []
$x0, $x1 = LDPXi $x0, 0
DBG_INSTR_REF !12, dbg-instr-ref(1, 0), debug-location !14
DBG_INSTR_REF !13, dbg-instr-ref(2, 0), debug-location !14
```
In this case, we need to create a new debug instruction number
for the `LDP` instruction, we then need to add entries into the
debugSubstitutions table to map the old instr-refs to the new ones.
After this patch, the result will be:
```
debugValueSubstitutions:
- { srcinst: 1, srcop: 0, dstinst: 3, dstop: 1, subreg: 0 }
- { srcinst: 2, srcop: 0, dstinst: 3, dstop: 0, subreg: 0 }
$x0, $x1 = LDPXi $x0, 0, debug-instr-number 3
DBG_INSTR_REF !12, dbg-instr-ref(1, 0), debug-location !14
DBG_INSTR_REF !12, dbg-instr-ref(2, 0), debug-location !14
```
However, this is not all, we also can have a case where there is a
sign-extend involved, let's look at the case:
```
debugValueSubstitutions:[]
$w1 = LDRWui $x0, 1, debug-instr-number 1
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
$x0 = LDRSWui $x0, 0, debug-instr-number 2
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
This will become:
```
debugValueSubstitutions:[]
$w0, $w1 = LDPWi $x0, 0
$w0 = KILL $w0, implicit-def $x0
$x0 = SBFMXri $x0, 0, 31
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
$x0 is where the final value is stored, so the sign extend (SBFMXri)
instruction contains the final value we care about we give it a new
debug-instr-number 3. Whereas, $w1 contains the final value that we care
about, therefore the LDP instruction is also given a new
debug-instr-number 4. We have to add these subsitutions to the
debugValueSubstitutions table. However, we also have to ensure that the
OpIndex that pointed to debug-instr-number 1 gets updated to 1, because
$w1 is the second operand of the LDP instruction.
The result after the patch looks like:
```
debugValueSubstitutions:
- { srcinst: 1, srcop: 0, dstinst: 4, dstop: 1, subreg: 0 }
- { srcinst: 2, srcop: 0, dstinst: 3, dstop: 0, subreg: 0 }
$w0, $w1 = LDPWi $x0, 0, debug-instr-number 4
$w0 = KILL $w0, implicit-def $x0
$x0 = SBFMXri $x0, 0, 31, debug-instr-number 3
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
This patch addresses that problem.
fixes#136243
This change converts memset into a series of geps and stores It is
intentionally limited to memsets of fixed size It also converts the byte
stores to type stores.
DXIL does not support i8 plus this reduces the total number of gep and
store instructions.
This change also moves DXILFinalizeLinkage to run after Legalization to
clean up any dead intrinsic definitions.
This patch adds support for LLVM IR atomicrmw `fmaximum` and `fminimum`
instructions.
These mirror the `llvm.maximum.*` and `llvm.minimum.*` instructions, but
are atomic and use IEEE754 2019 handling for NaNs, which is different to
`fmax` and `fmin`. See:
https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic
for more details.
Future changes will allow this LLVM IR to be lowered to specialised
assembler instructions on suitable targets, such as AArch64.
Since e39f6c1844fab59c638d8059a6cf139adb42279a opt will infer the
correct datalayout when given a triple. Avoid explicitly specifying it
in tests that depend on the AMDGPU target being present to avoid the
string becoming out of sync with the TargetInfo value.
Only tests with REQUIRES: amdgpu-registered-target or a local lit.cfg
were updated to ensure that tests for non-target-specific passes that
happen to use the AMDGPU layout still pass when building with a limited
set of targets.
Reviewed By: shiltian, arsenm
Pull Request: https://github.com/llvm/llvm-project/pull/137921
In PowerPC, if a borrow occurs during a subtraction, the carry bit is
zero (unset). The carry bit is set if no borrow occurs.
For ISD::USUBO_CARRY, the nodes produce two results: the normal result
of the addition or subtraction, and a boolean value that is 1 if and
only if there is an outgoing carry or borrow.
Therefore, we need to convert a 1 (which indicates a borrow in
ISD::USUBO_CARRY) to 0 to match PowerPC's definition of borrow.
Similarly, we need to convert a 0 (no borrow in ISD::USUBO_CARRY) to 1
for PowerPC.
To perform this conversion, we use XOR 1 instead of XOR
DAG.getAllOnesConstant(DL, CarryOp.getValueType()).
`
Fix issue 131298 where an undefined $scc register causes verifier errors
when using SI_KILL_F32_COND_IMM_TERMINATOR instructions. The problem
occurs because the $scc register defined in a comparison before the kill
terminator is used in successor blocks, but was not properly marked as live-in.
This patch:
- Adds code to check if SCC is used in the successor block
- Adds SCC as a live-in to successor blocks
- Handles both explicit and implicit uses of SCC
With this patch the machine verifier no longer reports undefined $scc
errors in following kill terminator instruction.
Fixes#131298
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
An update to the spirv validator is now correctly rejecting vulkan
shaders with the linkage capability. We have a couple tests that need
updating to remove the capability.
Fixes https://github.com/llvm/llvm-project/issues/136344
We currently accept label arguments to inline asm calls. This support
predates both blockaddresses and callbr and is only covered by one X86
test. Remove it in favor of callbr (or at least blockaddress, though
that cannot guarantee correct codegen, just like using block labels
directly can't).
I didn't bother implementing bitcode upgrade support for this, but I can
add it if desired.
fixes#137202
investingating i8 allocas I came to find some missing instructions from
out i8 legalization around load, store, and select.
Added those three.
To do i8 allocas right though we needed to walk the uses and find the
casts.
After finding the casts I chose to pick the smallest cast as the cast to
transform to. That would then let me preserve the larger casts that come
later
Linkers do not currently support PT_GNU_RELRO for SHF_X86_64_LARGE
sections; that would require the linker to emit more than one
PT_GNU_RELRO because large sections are discontiguous by design,
and most ELF dynamic loaders do not support that (bionic appears to
support it but glibc/musl/FreeBSD/NetBSD/OpenBSD appear not to). With
current linkers these sections will end up in .ldata which results
in silently disabling RELRO. Therefore, disable SHF_X86_64_LARGE for
RELRO sections. If this ever gets supported by downstream components
in the future we could add an opt-in flag for moving these sections
to .ldata.rel.ro which would trigger the creation of a second
PT_GNU_RELRO.
Reviewers: MaskRay, aeubanks
Reviewed By: aeubanks
Pull Request: https://github.com/llvm/llvm-project/pull/137742
The code below the removed check looks generic enough to support
arbitrary integer widths. This change helps 32-bit targets avoid
expensive expansion/libcalls in the case of zero input.
Pull Request: https://github.com/llvm/llvm-project/pull/137197
Fixes#112272
In addition to the implementation of the UAVsAtEveryStage shader flag
analysis, several unrelated tests have had the `dx.valver` module
metadata defined to avoid setting the UAVsAtEveryStage shader flag in
them.
Example:
```
!dx.valver = !{!0}
!0 = !{i32 1, i32 8}
```
---------
Co-authored-by: Justin Bogner <mail@justinbogner.com>
This PR revises the descriptions of DXIL module flags.
Descriptions such as `D3D11_1_SB_GLOBAL_FLAG_SKIP_OPTIMIZATION` are
referring to Global Flags in DXBC.
DXBC is not a supported backend target, so references to DXBC should not
be present.
There is also confusion with regards to the description of the
`LowPrecisionPresent` DXIL module flag, which currently reads
`D3D11_1_SB_GLOBAL_FLAG_ENABLE_MINIMUM_PRECISION` and implies the use of
minimum-precision to handle 16-bit types.
However this is not true, because both the flags `LowPrecisionPresent`
and `UseNativeLowPrecision` can simultaneously be set in the same DXIL
module, and minimum precision mode is mutually exclusive with native low
precision.
This PR revises the description of the `LowPrecisionPresent` flag to
accurately describe what it represents.
We are adding new test cases to see the transformation impact due to #134979.
These are similar to previous commit #688c3ffb057a87b86c6c1e77040418adf511efbb.
Handling struct with two member and different vector factor.
---------
Co-authored-by: Rohit Aggarwal <Rohit.Aggarwal@amd.com>
LegalizerHelper::reduceLoadStoreWidth does not work for non-byte-sized
types, because this would require (un)packing of bits across byte
boundaries.
Precommit tests: #134904
Suppress EGPR/NDD instructions for relocations to avoid APX relocation
types emitted. This is to keep backward compatibility with old version
of linkers without APX support. The use case is to try APX features with
LLVM + old built-in linker on RHEL9 OS which is expected to be EOL in
2032.
If there are APX relocation types, the old version of linkers would
raise "unsupported relocation type" error. Example:
```
$ llvm-mc -filetype=obj -o got.o -triple=x86_64-unknown-linux got.s
$ ld got.o -o got.exe
ld: got.o: unsupported relocation type 0x2b
...
$ cat got.s
...
movq foo@GOTPCREL(%rip), %r16
$ llvm-objdump -dr got.o
...
1: d5 48 8b 05 00 00 00 00 movq (%rip), %r16
0000000000000005: R_X86_64_CODE_4_GOTPCRELX foo-0x4
```
Currently shouldReduceLoadWidth is very relaxed about when loads can be
split to avoid extractions from the original full width load - resulting
in many cases where the number of memory operations notably increases,
replacing the cost of a extract_subvector for additional loads.
This patch adjusts the 256/512-bit vector load splitting metric to
detect cases where ANY use of the full width load is be used directly -
in which case we will now reuse that load for smaller types, unless we'd
need to extract an upper subvector / integer element - i.e. we now
correctly treat (extract_subvector cst, 0) as free.
We retain the existing logic of never splitting loads if all uses are
extract+stores but we improve this by peeking through bitcasts while
looking for extract_subvector/store chains.
This required a number of fixes - shouldReduceLoadWidth now needs to
peek through bitcasts UP the use-chain to find final users (limited to
hasOneUse cases to reduce complexity). It also exposed an issue in
isTargetCanonicalConstantNode which assumed that a load of vector
constant data would always extract, which is no longer the case.
The expensive checks bots found issues with #132481, due to not setting
defs/uses correctly. In 31bd7a507152 I added verify flags, so that the
failure is reproduced without requiring expensive checks, and xfailed
the test.
This change:
- Ensures that registers are correctly marked as defs/uses.
- Removes the xfail.
- Leaves the tests with `-verify-machineinstrs` which should have been
present originally.
This patch adds the following Dense Math Facility 16-bit half-precision
floating-point calculation instructions: dmxvf16gerx2, dmxvf16gerx2pp,
dmxvf16gerx2pn, dmxvf16gerx2np, dmxvf16gerx2nn, pmdmxvf16gerx2,
pmdmxvf16gerx2pp, pmdmxvf16gerx2pn, pmdmxvf16gerx2np, pmdmxvf16gerx2nn,
along with their corresponding intrinsics and tests.
This PR introduces a Metadata Node Kind allowlist. The purpose is to
prevent newer Metadata Node Kinds to be used and inserted into the
outputted DXIL module. Only the metadata kinds that are accepted in the
DXIL Validator are on the allowlist. The Github DXC validator doesn't
support these newer Metadata Node Kinds, so we need to filter them out.
We introduce this restrictive allowlist into LLVM and strip all metadata
that isn't found in the list.
The accompanying test would add the `llvm.loop.mustprogress` metadata
node kind, but thanks to the allowlist, filters it out, and so the
whitelist is proven to work.
The test also has two separate metadata kinds that are on the allowlist,
and remain after the DXIL Prepare pass.
Currently, if there is already noalias metadata present on loads and
stores, lower module lds pass is generating a more conservative aliasing
set. This results in inhibiting scheduling intrinsics that would have
otherwise generated a better pipelined instruction.
The fix is not to always intersect already existing noalias metadata
with noalias created for lowering of LDS. But to intersect only if
noalias scopes are from the same domain, otherwise concatenate exising
noalias sets with LDS noalias.
There a few patches that have come for scopedAA in the past. Following
three should be enough background information.
https://reviews.llvm.org/D91576https://reviews.llvm.org/D108315https://reviews.llvm.org/D110049
Essentially, after a pass that might change aliasing info, one should
check if that pass results in change number of MayAlias or ModRef using
the following:
`opt -S -aa-pipeline=basic-aa,scoped-noalias-aa -passes=aa-eval
-evaluate-aa-metadata -print-all-alias-modref-info -disable-output`
Current implementation tries to fold the operand before
rematerialization because it can reduce one register usage. But if there
is a physical register available we can still rematerialize it without
causing high register pressure.
This patch do this check to find the better choice. Then we can produce
xorps %xmm1, %xmm1
ucomiss %xmm1, %xmm0
instead of
ucomiss LCPI0_1(%rip), %xmm0
The __spirv_GenericCastToPtrExplicit_To* builtins and its equivalent
OpenCL builtins (to_global, to_local and to_private) were mapped to
OpGenericCastToPtr instead of OpGenericCastToPtrExplicit.
The patch now uses OpGenericCastToPtrExplicit for these builtins.
When the SPV_INTEL_arbitrary_precision_integers extension is allowed to
be used, the backend will unconditionnally add it to the module used
extensions.
The patch prevent SPV_INTEL_arbitrary_precision_integers from being
declared if unneeded.
