The aarch64-ldst-opt pass tries to merge two load instructions
(LDR*) to a load pair instruction (LDP*).
When merging the instructions, there is a case where one of the
loads would have to also be sign extended. In either case,
(sign extend or not), the pass needs to preserve the debug-instr-number
from the original loads to the load pair instruction to make sure debug
info
isn't lost in the case where instruction referencing is being used.
For example:
We can have something like this:
```
debugValueSubstitutions:[]
$x1 = LDRXui $x0, 1, debug-instr-number 1
DBG_INSTR_REF !13, dbg-instr-ref(1, 0), debug-location !11
$x0 = LDRXui killed $x0, 0, debug-instr-number 2
DBG_INSTR_REF !14, dbg-instr-ref(2, 0), debug-location !11
```
This would be changed to:
```
debugValueSubstitutions: []
$x0, $x1 = LDPXi $x0, 0
DBG_INSTR_REF !12, dbg-instr-ref(1, 0), debug-location !14
DBG_INSTR_REF !13, dbg-instr-ref(2, 0), debug-location !14
```
In this case, we need to create a new debug instruction number
for the `LDP` instruction, we then need to add entries into the
debugSubstitutions table to map the old instr-refs to the new ones.
After this patch, the result will be:
```
debugValueSubstitutions:
- { srcinst: 1, srcop: 0, dstinst: 3, dstop: 1, subreg: 0 }
- { srcinst: 2, srcop: 0, dstinst: 3, dstop: 0, subreg: 0 }
$x0, $x1 = LDPXi $x0, 0, debug-instr-number 3
DBG_INSTR_REF !12, dbg-instr-ref(1, 0), debug-location !14
DBG_INSTR_REF !12, dbg-instr-ref(2, 0), debug-location !14
```
However, this is not all, we also can have a case where there is a
sign-extend involved, let's look at the case:
```
debugValueSubstitutions:[]
$w1 = LDRWui $x0, 1, debug-instr-number 1
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
$x0 = LDRSWui $x0, 0, debug-instr-number 2
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
This will become:
```
debugValueSubstitutions:[]
$w0, $w1 = LDPWi $x0, 0
$w0 = KILL $w0, implicit-def $x0
$x0 = SBFMXri $x0, 0, 31
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
$x0 is where the final value is stored, so the sign extend (SBFMXri)
instruction contains the final value we care about we give it a new
debug-instr-number 3. Whereas, $w1 contains the final value that we care
about, therefore the LDP instruction is also given a new
debug-instr-number 4. We have to add these subsitutions to the
debugValueSubstitutions table. However, we also have to ensure that the
OpIndex that pointed to debug-instr-number 1 gets updated to 1, because
$w1 is the second operand of the LDP instruction.
The result after the patch looks like:
```
debugValueSubstitutions:
- { srcinst: 1, srcop: 0, dstinst: 4, dstop: 1, subreg: 0 }
- { srcinst: 2, srcop: 0, dstinst: 3, dstop: 0, subreg: 0 }
$w0, $w1 = LDPWi $x0, 0, debug-instr-number 4
$w0 = KILL $w0, implicit-def $x0
$x0 = SBFMXri $x0, 0, 31, debug-instr-number 3
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
This patch addresses that problem.
fixes#136243
This change converts memset into a series of geps and stores It is
intentionally limited to memsets of fixed size It also converts the byte
stores to type stores.
DXIL does not support i8 plus this reduces the total number of gep and
store instructions.
This change also moves DXILFinalizeLinkage to run after Legalization to
clean up any dead intrinsic definitions.
This patch adds support for LLVM IR atomicrmw `fmaximum` and `fminimum`
instructions.
These mirror the `llvm.maximum.*` and `llvm.minimum.*` instructions, but
are atomic and use IEEE754 2019 handling for NaNs, which is different to
`fmax` and `fmin`. See:
https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic
for more details.
Future changes will allow this LLVM IR to be lowered to specialised
assembler instructions on suitable targets, such as AArch64.
Since e39f6c1844fab59c638d8059a6cf139adb42279a opt will infer the
correct datalayout when given a triple. Avoid explicitly specifying it
in tests that depend on the AMDGPU target being present to avoid the
string becoming out of sync with the TargetInfo value.
Only tests with REQUIRES: amdgpu-registered-target or a local lit.cfg
were updated to ensure that tests for non-target-specific passes that
happen to use the AMDGPU layout still pass when building with a limited
set of targets.
Reviewed By: shiltian, arsenm
Pull Request: https://github.com/llvm/llvm-project/pull/137921
In PowerPC, if a borrow occurs during a subtraction, the carry bit is
zero (unset). The carry bit is set if no borrow occurs.
For ISD::USUBO_CARRY, the nodes produce two results: the normal result
of the addition or subtraction, and a boolean value that is 1 if and
only if there is an outgoing carry or borrow.
Therefore, we need to convert a 1 (which indicates a borrow in
ISD::USUBO_CARRY) to 0 to match PowerPC's definition of borrow.
Similarly, we need to convert a 0 (no borrow in ISD::USUBO_CARRY) to 1
for PowerPC.
To perform this conversion, we use XOR 1 instead of XOR
DAG.getAllOnesConstant(DL, CarryOp.getValueType()).
`
Fix issue 131298 where an undefined $scc register causes verifier errors
when using SI_KILL_F32_COND_IMM_TERMINATOR instructions. The problem
occurs because the $scc register defined in a comparison before the kill
terminator is used in successor blocks, but was not properly marked as live-in.
This patch:
- Adds code to check if SCC is used in the successor block
- Adds SCC as a live-in to successor blocks
- Handles both explicit and implicit uses of SCC
With this patch the machine verifier no longer reports undefined $scc
errors in following kill terminator instruction.
Fixes#131298
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
An update to the spirv validator is now correctly rejecting vulkan
shaders with the linkage capability. We have a couple tests that need
updating to remove the capability.
Fixes https://github.com/llvm/llvm-project/issues/136344
We currently accept label arguments to inline asm calls. This support
predates both blockaddresses and callbr and is only covered by one X86
test. Remove it in favor of callbr (or at least blockaddress, though
that cannot guarantee correct codegen, just like using block labels
directly can't).
I didn't bother implementing bitcode upgrade support for this, but I can
add it if desired.
fixes#137202
investingating i8 allocas I came to find some missing instructions from
out i8 legalization around load, store, and select.
Added those three.
To do i8 allocas right though we needed to walk the uses and find the
casts.
After finding the casts I chose to pick the smallest cast as the cast to
transform to. That would then let me preserve the larger casts that come
later
Linkers do not currently support PT_GNU_RELRO for SHF_X86_64_LARGE
sections; that would require the linker to emit more than one
PT_GNU_RELRO because large sections are discontiguous by design,
and most ELF dynamic loaders do not support that (bionic appears to
support it but glibc/musl/FreeBSD/NetBSD/OpenBSD appear not to). With
current linkers these sections will end up in .ldata which results
in silently disabling RELRO. Therefore, disable SHF_X86_64_LARGE for
RELRO sections. If this ever gets supported by downstream components
in the future we could add an opt-in flag for moving these sections
to .ldata.rel.ro which would trigger the creation of a second
PT_GNU_RELRO.
Reviewers: MaskRay, aeubanks
Reviewed By: aeubanks
Pull Request: https://github.com/llvm/llvm-project/pull/137742
The code below the removed check looks generic enough to support
arbitrary integer widths. This change helps 32-bit targets avoid
expensive expansion/libcalls in the case of zero input.
Pull Request: https://github.com/llvm/llvm-project/pull/137197
Fixes#112272
In addition to the implementation of the UAVsAtEveryStage shader flag
analysis, several unrelated tests have had the `dx.valver` module
metadata defined to avoid setting the UAVsAtEveryStage shader flag in
them.
Example:
```
!dx.valver = !{!0}
!0 = !{i32 1, i32 8}
```
---------
Co-authored-by: Justin Bogner <mail@justinbogner.com>
This PR revises the descriptions of DXIL module flags.
Descriptions such as `D3D11_1_SB_GLOBAL_FLAG_SKIP_OPTIMIZATION` are
referring to Global Flags in DXBC.
DXBC is not a supported backend target, so references to DXBC should not
be present.
There is also confusion with regards to the description of the
`LowPrecisionPresent` DXIL module flag, which currently reads
`D3D11_1_SB_GLOBAL_FLAG_ENABLE_MINIMUM_PRECISION` and implies the use of
minimum-precision to handle 16-bit types.
However this is not true, because both the flags `LowPrecisionPresent`
and `UseNativeLowPrecision` can simultaneously be set in the same DXIL
module, and minimum precision mode is mutually exclusive with native low
precision.
This PR revises the description of the `LowPrecisionPresent` flag to
accurately describe what it represents.
We are adding new test cases to see the transformation impact due to #134979.
These are similar to previous commit #688c3ffb057a87b86c6c1e77040418adf511efbb.
Handling struct with two member and different vector factor.
---------
Co-authored-by: Rohit Aggarwal <Rohit.Aggarwal@amd.com>
LegalizerHelper::reduceLoadStoreWidth does not work for non-byte-sized
types, because this would require (un)packing of bits across byte
boundaries.
Precommit tests: #134904
Suppress EGPR/NDD instructions for relocations to avoid APX relocation
types emitted. This is to keep backward compatibility with old version
of linkers without APX support. The use case is to try APX features with
LLVM + old built-in linker on RHEL9 OS which is expected to be EOL in
2032.
If there are APX relocation types, the old version of linkers would
raise "unsupported relocation type" error. Example:
```
$ llvm-mc -filetype=obj -o got.o -triple=x86_64-unknown-linux got.s
$ ld got.o -o got.exe
ld: got.o: unsupported relocation type 0x2b
...
$ cat got.s
...
movq foo@GOTPCREL(%rip), %r16
$ llvm-objdump -dr got.o
...
1: d5 48 8b 05 00 00 00 00 movq (%rip), %r16
0000000000000005: R_X86_64_CODE_4_GOTPCRELX foo-0x4
```
Currently shouldReduceLoadWidth is very relaxed about when loads can be
split to avoid extractions from the original full width load - resulting
in many cases where the number of memory operations notably increases,
replacing the cost of a extract_subvector for additional loads.
This patch adjusts the 256/512-bit vector load splitting metric to
detect cases where ANY use of the full width load is be used directly -
in which case we will now reuse that load for smaller types, unless we'd
need to extract an upper subvector / integer element - i.e. we now
correctly treat (extract_subvector cst, 0) as free.
We retain the existing logic of never splitting loads if all uses are
extract+stores but we improve this by peeking through bitcasts while
looking for extract_subvector/store chains.
This required a number of fixes - shouldReduceLoadWidth now needs to
peek through bitcasts UP the use-chain to find final users (limited to
hasOneUse cases to reduce complexity). It also exposed an issue in
isTargetCanonicalConstantNode which assumed that a load of vector
constant data would always extract, which is no longer the case.
The expensive checks bots found issues with #132481, due to not setting
defs/uses correctly. In 31bd7a507152 I added verify flags, so that the
failure is reproduced without requiring expensive checks, and xfailed
the test.
This change:
- Ensures that registers are correctly marked as defs/uses.
- Removes the xfail.
- Leaves the tests with `-verify-machineinstrs` which should have been
present originally.
This patch adds the following Dense Math Facility 16-bit half-precision
floating-point calculation instructions: dmxvf16gerx2, dmxvf16gerx2pp,
dmxvf16gerx2pn, dmxvf16gerx2np, dmxvf16gerx2nn, pmdmxvf16gerx2,
pmdmxvf16gerx2pp, pmdmxvf16gerx2pn, pmdmxvf16gerx2np, pmdmxvf16gerx2nn,
along with their corresponding intrinsics and tests.
This PR introduces a Metadata Node Kind allowlist. The purpose is to
prevent newer Metadata Node Kinds to be used and inserted into the
outputted DXIL module. Only the metadata kinds that are accepted in the
DXIL Validator are on the allowlist. The Github DXC validator doesn't
support these newer Metadata Node Kinds, so we need to filter them out.
We introduce this restrictive allowlist into LLVM and strip all metadata
that isn't found in the list.
The accompanying test would add the `llvm.loop.mustprogress` metadata
node kind, but thanks to the allowlist, filters it out, and so the
whitelist is proven to work.
The test also has two separate metadata kinds that are on the allowlist,
and remain after the DXIL Prepare pass.
Currently, if there is already noalias metadata present on loads and
stores, lower module lds pass is generating a more conservative aliasing
set. This results in inhibiting scheduling intrinsics that would have
otherwise generated a better pipelined instruction.
The fix is not to always intersect already existing noalias metadata
with noalias created for lowering of LDS. But to intersect only if
noalias scopes are from the same domain, otherwise concatenate exising
noalias sets with LDS noalias.
There a few patches that have come for scopedAA in the past. Following
three should be enough background information.
https://reviews.llvm.org/D91576https://reviews.llvm.org/D108315https://reviews.llvm.org/D110049
Essentially, after a pass that might change aliasing info, one should
check if that pass results in change number of MayAlias or ModRef using
the following:
`opt -S -aa-pipeline=basic-aa,scoped-noalias-aa -passes=aa-eval
-evaluate-aa-metadata -print-all-alias-modref-info -disable-output`
Current implementation tries to fold the operand before
rematerialization because it can reduce one register usage. But if there
is a physical register available we can still rematerialize it without
causing high register pressure.
This patch do this check to find the better choice. Then we can produce
xorps %xmm1, %xmm1
ucomiss %xmm1, %xmm0
instead of
ucomiss LCPI0_1(%rip), %xmm0
The __spirv_GenericCastToPtrExplicit_To* builtins and its equivalent
OpenCL builtins (to_global, to_local and to_private) were mapped to
OpGenericCastToPtr instead of OpGenericCastToPtrExplicit.
The patch now uses OpGenericCastToPtrExplicit for these builtins.
When the SPV_INTEL_arbitrary_precision_integers extension is allowed to
be used, the backend will unconditionnally add it to the module used
extensions.
The patch prevent SPV_INTEL_arbitrary_precision_integers from being
declared if unneeded.
Extends changes from
[ff687af](ff687af04f).
Fixes https://github.com/llvm/llvm-project/issues/131476.
This patch adds a DAG combine to replace an `AND` of an `ATOMIC_LOAD`
with a full-bit mask (e.g. `0xFF`, `0xFFFF`, etc.) which is generated as
a result of `(zext (atomic_load))`, by a zero-extended load, provided
the atomic operation is monotonic or weaker.
The `qc.c.mienter` and `qc.c.mienter.nest` instructions, broadly only
save the argument and temporary registers. The exceptions are that they
also save `fp` (`s0`) to construct a frame chain from the signal handler
to the frame below, and they also save `ra`. They are designed this way
so that (if needed) push and pop instructions can be used to save the
callee-saved registers.
This patch implements this optimisation, constructing the following
rather than a long sequence of `sw` and `lw` instructions for saving the
callee-saved registers:
```asm
qc.c.mienter
qc.cm.push {ra, s0-sN}, -M
...
qc.cm.pop {ra, s0-sN}, M
qc.c.mileaveret
```
There are some carefully-worked-out details here, especially around CFI
information. For any register saved by both `qc.c.mienter(.nest)` and
the push (which is `ra` and `s0` at most), we point the CFI information
at the version saved by `qc.c.mienter(.nest)`. This ensures the CFI
points at the same `fp` copy as a frame pointer unwinder would find.
This patch adds support for LLVM IR atomicrmw `fmaximum` and `fminimum`
instructions.
These mirror the `llvm.maximum.*` and `llvm.minimum.*` instructions, but
are atomic and use IEEE754 2019 handling for NaNs, which is different to
`fmax` and `fmin`. See:
https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic
for more details.
Future changes will allow this LLVM IR to be lowered to specialised
assembler instructions on suitable targets, such as AArch64.
When floating-point operations are legalized to operations of a higher
precision (e.g. f16 fadd being legalized to f32 fadd) then we get
narrowing then widening operations between each operation. With the
appropriate fast math flags (nnan ninf contract) we can eliminate these
casts.
An i8 and i16 vector extract/insert has to go via a i32 to make sure the
types are legal. This patch adds patterns for extract from a i8/i16
vector, inserted into a i16/i32 vector. This avoids the round trip via a
GPR which can limit performance.
