This moves the responsibility for cleaning up dead intrinsics from
DXILFinalizeLinkage to DXILOpLowering, and moves DXILFinalizeLinkage
back to it's pre-#136244 place in the pipeline. Doing this avoids issues
with DXIL passes running on obviously dead code, and makes it more clear
what DXILFinalizeLinkage is really doing.
This also helps with the story for #134260, as cleaning up dead
intrinsics doesn't make sense if this becomes a more generic pass.
Note that test/CodeGen/DirectX/remove-dead-intriniscs.ll already covers
most of the testing here. It'd be nice to have something that catches
the regression from changing the pass ordering but I couldn't come up
with anything that wouldn't be incredibly fragile.
Fixes#138180.
This patch adds CalleeGuids to the serialized format and increments the version number to 4. The unit tests are updated to include a new test for v4 and the YAML format is also updated to be able to roundtrip the v4 format.
Following from the discussion in
https://github.com/llvm/llvm-project/pull/138095#discussion_r2070484664,
these intrinsics are poison if any of their operands are poison, and are
marked as such in propagatesPoison in ValueTracking.cpp.
This will help fold away leftover vectors produced by VectorCombine when
scalarizing intrinsics.
Pull invariant load identification, which was previously part of
DAGToDAG ISel, into a new IR pass NVPTXTagInvariantLoads. This makes it
possible to disable this optimization at O0 and reduces the complexity
of the SelectionDAG pass. Moving this logic to an IR pass also allows
for implementing a more powerful traversal in the future.
Fixes https://github.com/llvm/llvm-project/issues/138138
- Move `MIPrinter` class to anonymous namespace, and remove it as a
friend of `MachineBasicBlock`.
- Move `canPredictBranchProbabilities` to `MachineBasicBlock` and change
it to use the new `BranchProbability::normalizeProbabilities` function
that accepts a range, and also to use `llvm::equal()` to check equality
of the two vectors.
- Use `ListSeparator` to print comma separate lists instead of manual
code to do that.
This experimental option was introduced in 2015 via commit 1192294, and
the target hook was added in 2020 via commit 99e865b6. There does not
appear to have ever been a use of this target hook in tree.
This code is complicating one of the most complicated and hard to
understand parts of our code base, and was an experiment introduced
nearly 10 years ago. Let's get rid of it.
Note that the idea described in the original patch is not neccessarily a
bad one, and we might return to it someday.
- Change MachineInstr operand accessors to use `ArrayRef` internally to
slice the operand array into sub-arrays.
- Minor: remove unnecessary {} on `MachineInstrBuilder::add`.
I tried to use these with a const reference in a separate patch, but the
pointers weren't marked as const. It turns out that these don't mutate
the instruction.
Implement proper splitting functions for PARTIAL_REDUCE_MLA ISD nodes.
This makes the udot_8to64 and sdot_8to64 tests generate dot product
instructions for when the new ISD nodes are used.
---------
Co-authored-by: James Chesterman <james.chesterman@arm.com>
The default maximum waves/EU returned by the family of
`AMDGPUSubtarget::getWavesPerEU` is currently the maximum number of
waves/EU supported by the subtarget (only a valid occupancy range in
"amdgpu-waves-per-eu" may lower that maximum). This ignores maximum
achievable occupancy imposed by flat workgroup size and LDS usage,
resulting in situations where `AMDGPUSubtarget::getWavesPerEU` produces
a maximum higher than the one from
`AMDGPUSubtarget::getOccupancyWithWorkGroupSizes`.
This limits the waves/EU range's maximum to the maximum achievable
occupancy derived from flat workgroup sizes and LDS usage. This only has
an impact on functions which restrict flat workgroup size with
"amdgpu-flat-work-group-size", since the default range of flat workgroup
sizes achieves the maximum number of waves/EU supported by the
subtarget.
Improvements to the handling of "amdgpu-waves-per-eu" are left for a
follow up PR (e.g., I think the attribute should be able to lower the
full range of waves/EU produced by these methods).
If the SDNode is used it can pick up the wrong results number, for
example looking at the known bits of the first result where it should be
looking at the second. The SDValue is already present as the
SelectCodeCommon checks move from parent to child, pass the SDValue
through to CheckNodePredicate as Op so that it can use it if necessary.
SDNode *N is still generated, keeping most PatFrags the same.
Fixes#137274
Ensure that when custom lowering a vector load/store to a multi-output
load/store node we confirm that the memory value type matches the type
used by the node. Also add some asserts for basic sanity checking of
load size.
Fixes https://github.com/llvm/llvm-project/issues/138034
Reland https://github.com/llvm/llvm-project/pull/133173
Clang spawns a new thread to avoid running out of stack space. This can
make debugging and performance analysis more difficult as how the
threads are connected is difficult to recover.
This patch introduces `runOnNewStack` and applies it in Clang. On
platforms that have good support for it this allocates a new stack and
moves to it using assembly. Doing split stacks like this actually runs
on most platforms, but many debuggers and unwinders reject the large or
backwards stack offsets that occur. Apple platforms and tools are known
to support this, so this only enables it there for now.
Fixes https://github.com/llvm/llvm-project/issues/130020
This fixes an issue where the si-fold-operands pass would incorrectly
fold immediate values into COPY instructions targeting av_32 registers.
The pass now checks register class constraints before attempting to fold
the immediate.
The aarch64-ldst-opt pass tries to merge two load instructions
(LDR*) to a load pair instruction (LDP*).
When merging the instructions, there is a case where one of the
loads would have to also be sign extended. In either case,
(sign extend or not), the pass needs to preserve the debug-instr-number
from the original loads to the load pair instruction to make sure debug
info
isn't lost in the case where instruction referencing is being used.
For example:
We can have something like this:
```
debugValueSubstitutions:[]
$x1 = LDRXui $x0, 1, debug-instr-number 1
DBG_INSTR_REF !13, dbg-instr-ref(1, 0), debug-location !11
$x0 = LDRXui killed $x0, 0, debug-instr-number 2
DBG_INSTR_REF !14, dbg-instr-ref(2, 0), debug-location !11
```
This would be changed to:
```
debugValueSubstitutions: []
$x0, $x1 = LDPXi $x0, 0
DBG_INSTR_REF !12, dbg-instr-ref(1, 0), debug-location !14
DBG_INSTR_REF !13, dbg-instr-ref(2, 0), debug-location !14
```
In this case, we need to create a new debug instruction number
for the `LDP` instruction, we then need to add entries into the
debugSubstitutions table to map the old instr-refs to the new ones.
After this patch, the result will be:
```
debugValueSubstitutions:
- { srcinst: 1, srcop: 0, dstinst: 3, dstop: 1, subreg: 0 }
- { srcinst: 2, srcop: 0, dstinst: 3, dstop: 0, subreg: 0 }
$x0, $x1 = LDPXi $x0, 0, debug-instr-number 3
DBG_INSTR_REF !12, dbg-instr-ref(1, 0), debug-location !14
DBG_INSTR_REF !12, dbg-instr-ref(2, 0), debug-location !14
```
However, this is not all, we also can have a case where there is a
sign-extend involved, let's look at the case:
```
debugValueSubstitutions:[]
$w1 = LDRWui $x0, 1, debug-instr-number 1
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
$x0 = LDRSWui $x0, 0, debug-instr-number 2
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
This will become:
```
debugValueSubstitutions:[]
$w0, $w1 = LDPWi $x0, 0
$w0 = KILL $w0, implicit-def $x0
$x0 = SBFMXri $x0, 0, 31
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
$x0 is where the final value is stored, so the sign extend (SBFMXri)
instruction contains the final value we care about we give it a new
debug-instr-number 3. Whereas, $w1 contains the final value that we care
about, therefore the LDP instruction is also given a new
debug-instr-number 4. We have to add these subsitutions to the
debugValueSubstitutions table. However, we also have to ensure that the
OpIndex that pointed to debug-instr-number 1 gets updated to 1, because
$w1 is the second operand of the LDP instruction.
The result after the patch looks like:
```
debugValueSubstitutions:
- { srcinst: 1, srcop: 0, dstinst: 4, dstop: 1, subreg: 0 }
- { srcinst: 2, srcop: 0, dstinst: 3, dstop: 0, subreg: 0 }
$w0, $w1 = LDPWi $x0, 0, debug-instr-number 4
$w0 = KILL $w0, implicit-def $x0
$x0 = SBFMXri $x0, 0, 31, debug-instr-number 3
DBG_INSTR_REF !7, dbg-instr-ref(1, 0), debug-location !9
DBG_INSTR_REF !8, dbg-instr-ref(2, 0), debug-location !9
```
This patch addresses that problem.
fixes#136243
This change converts memset into a series of geps and stores It is
intentionally limited to memsets of fixed size It also converts the byte
stores to type stores.
DXIL does not support i8 plus this reduces the total number of gep and
store instructions.
This change also moves DXILFinalizeLinkage to run after Legalization to
clean up any dead intrinsic definitions.
This patch adds support for LLVM IR atomicrmw `fmaximum` and `fminimum`
instructions.
These mirror the `llvm.maximum.*` and `llvm.minimum.*` instructions, but
are atomic and use IEEE754 2019 handling for NaNs, which is different to
`fmax` and `fmin`. See:
https://llvm.org/docs/LangRef.html#llvm-minimum-intrinsic
for more details.
Future changes will allow this LLVM IR to be lowered to specialised
assembler instructions on suitable targets, such as AArch64.
This patch adds a new flag, -ftime-report-json, which outputs the same
information as -ftime-report but as JSON instead of -ftime-report's
pretty printed format.
We can reuse isVectorIntrinsicWithScalarOpAtArg in VectorUtils to
determine if only the first lane will be used for a
VPWidenIntrinsicRecipe, provided that we also move the VP EVL operand
check into it.
This was needed by a local patch I was working on that created a
VPWidenIntrinsicRecipe with a VP intrinsic, and prevents the need to
update the scalar arguments in two places.
`ad9909d "[SLP]Fix perfect diamond match with extractelements in scalars" `
changed SLPVectorizer getScalarizationOverhead() to call
TTI.getVectorInstrCost() instead of TTI.getScalarizationOverhead() in some
cases. This was due to X86 specific handlings in these (overridden) methods,
and unfortunately the general preference of TTI.getScalarizationOverhead()
was dropped. If VL is available it should always be preferred to use
getScalarizationOverhead(), and this is indeed the case for SystemZ which
has a special insertion instruction that can insert two GPR64s.
Then ` 33af951 "[SLP]Synchronize cost of gather/buildvector nodes with
codegen"` reworked SLPVectorizer getGatherCost() which together with
ad9909d caused the SystemZ test vec-elt-insertion.ll to fail.
This patch restores the SystemZ test and reverts the change in SLPVectorizer
getScalarizationOverhead() so that TTI.getScalarizationOverhead() is always
called again. The ForPoisonSrc argument is now passed on to the TTI method
so that X86 can handle this as required.
Fixes: #135346
In PowerPC, if a borrow occurs during a subtraction, the carry bit is
zero (unset). The carry bit is set if no borrow occurs.
For ISD::USUBO_CARRY, the nodes produce two results: the normal result
of the addition or subtraction, and a boolean value that is 1 if and
only if there is an outgoing carry or borrow.
Therefore, we need to convert a 1 (which indicates a borrow in
ISD::USUBO_CARRY) to 0 to match PowerPC's definition of borrow.
Similarly, we need to convert a 0 (no borrow in ISD::USUBO_CARRY) to 1
for PowerPC.
To perform this conversion, we use XOR 1 instead of XOR
DAG.getAllOnesConstant(DL, CarryOp.getValueType()).
`
Fix issue 131298 where an undefined $scc register causes verifier errors
when using SI_KILL_F32_COND_IMM_TERMINATOR instructions. The problem
occurs because the $scc register defined in a comparison before the kill
terminator is used in successor blocks, but was not properly marked as live-in.
This patch:
- Adds code to check if SCC is used in the successor block
- Adds SCC as a live-in to successor blocks
- Handles both explicit and implicit uses of SCC
With this patch the machine verifier no longer reports undefined $scc
errors in following kill terminator instruction.
Fixes#131298
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
The `.machine push` and `.machine pop` directives were missing from the
SystemZ Backend Asm Parser. This PR adds them, and expands the
corresponding test to test proper operation.
Internally, this is modeled as a simple stack implemented on a
`SmallVector<StringRef>`.
The SVE intrinsics support shift amounts greater-than-or-equal to the
element type's bit-length, essentially saturating the shift amount to
the bit-length. However, the IR instructions consider this undefined
behaviour that results in poison. To account for this we now ignore the
result of the simplifications that result in poison. This allows
existing code to be used to simplify the shifts but does mean:
1) We don't simplify cases like "svlsl_s32(x, splat(32)) => 0".
2) We no longer constant fold cases like "svadd(poison, X) => poison"
For (1) we'd need dedicated target specific combines anyway and the
result of (2) is not specified by the ACLE and replicating LLVM IR
behaviour might be confusing to ACLE writers.
