This fixes an allocation failure in the new test.
In cases where getLargestLegalSuperClass can inflate the register class,
rematerialization could effectively undo a split which was done to
inflate
the register class, if the defining instruction can only write a
subclass
and the use can read the superclass.
Some of the x86 tests changes look like improvements, but some are
likely regressions.
I'm not entirely sure this is the correct place to fix this. It also
seems more complicated than necessary, but the decision to change
the register class is far removed from the point where the decision
to split the virtual register is made. I'm also also not sure if this
should be considering the register classes of all the use indexes
in getUseSlots, rather than just checking if this use index instruction
reads the register.
The current implementation to combine xor patterns for conjugation with
complex multiplies will not perform the transformation when either the
conjugate xor result or other multiplicand have other uses. This change
eliminates both single-use checks.
The transformation will eliminate the xor dependence and hence should be
profitable even if the conjugate is used elsewhere - and more profitable
if the xor is used in multiple fmulc/fcmulc instructions, eventually
going dead.
The check of the other multiplicand isn't required for correctness and
has no apparent performance implications.
Failure to sink a candidate should not block us from attempting to sink
other candidates. There are mechanisms in place to handle the case where
the failed to be sunk instruction uses an instruction that gets sunk (we
do not delete the original instruction corresponding with the sunk
instruction if it still has uses).
[NVPTX] Add Intrinsics for discard.*
This PR adds intrinsics for all variations of discard.*
* These intrinsics supports generic or global for all variations.
* The lowering is handled from nvvm to nvptx tablegen directly.
* Lit tests are added as part of discard.ll
* The generated PTX is verified with a 12.3 ptxas executable.
* Added docs for these intrinsics in NVPTXUsage.rst.
For more information, refer to the PTX ISA
<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-discard>_.
---------
Co-authored-by: abmajumder <abmajumder@nvidia.com>
Co-authored-by: gonzalobg <65027571+gonzalobg@users.noreply.github.com>
Fixes https://github.com/llvm/llvm-project/issues/127119
Remove `hasSideEffects` from `xxspltib` since there is no special
restriction specified in the ISA that prevent it from being reordered,
move, CSE, or LICM. Removing this restriction will allow `xxspltib` to
be hoisted out of loop bodies.
The si-peephole-sdwa pass adjusts the selections on sdwa instructions to
the selections on their operands during its conversions. For instance,
if an instruction selects `BYTE_0` and its operand selects `WORD_1`, the
combined selection should be `BYTE_2`, i.e. "`BYTE_0` of `WORD_1`". The
existing implementation does not always handle this correctly in some
complex situations with instructions across different basic blocks as
demonstrated by the test cases included in this PR.
This PR adds an additional selection combination step to the conversion
to fix this issue. It reverts the changes made by PR #123942 which had
disabled the conversion of preexisting SDWA instructions completely as a
quick fix.
---------
Co-authored-by: Jeffrey Byrnes <Jeffrey.Byrnes@amd.com>
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
When we have an integer used as a bit mask the llvm ir looks something
like this
```
%1 = and <16 x i32> %.splat, <i32 1, i32 2, i32 4, i32 8, i32 16, i32 32, i32 64, i32 128, i32 256, i32 512, i32 1024, i32 2048, i32 4096, i32 8192, i32 16384, i32 32768>
%cmp1 = icmp ne <16 x i32> %1, zeroinitializer
```
where `.splat` is vector containing the mask in all lanes. The assembly
generated for this looks like
```
vpbroadcastd %ecx, %zmm0
vptestmd .LCPI0_0(%rip), %zmm0, %k1
```
where we have a constant table of powers of 2.
Instead of doing this we could just move the relevant bits directly to
`k` registers using a `kmov` instruction.
```
kmovw %ecx, %k1
```
This is faster and also reduces code size.
This patch combines (iN vector.reduce.add (zext (vXi1 A to vXiN)) into
vcpop.m instruction (similarly to bitcast + ctpop pattern). It can be
useful for counting number of set bits in scalable vector types, which
can't be expressed with bitcast + ctpop (this was previously discussed
here: https://github.com/llvm/llvm-project/pull/74294).
Patch #96594 substitutes setcc + zext pair with setzucc, but it results
in redundant test because X86FlagsCopyLowering doesn't recognize it.
This patch removes redundant test by reverting setzucc to setcc
(optimized) + zext.
If a (two-result) node like `FMODF` or `FFREXP` is expanded to a library
call, where said library has the function prototype like: `float(float,
float*)` -- that is it returns a float from the call and via an output
pointer. The first result of the node maps to the value returned by
value and the second result maps to the value returned via the output
pointer.
If only the second result is used after the expansion, we hit an issue
on x87 targets:
```
// Before expansion:
t0, t1 = fmodf x
return t1 // t0 is unused
```
Expanded result:
```
ptr = alloca
ch0 = call modf ptr
t0, ch1 = copy_from_reg, ch0 // t0 unused
t1, ch2 = ldr ptr, ch1
return t1
```
So far things are alright, but the DAGCombiner optimizes this to:
```
ptr = alloca
ch0 = call modf ptr
// copy_from_reg optimized out
t1, ch1 = ldr ptr, ch0
return t1
```
On most targets this is fine. The optimized out `copy_from_reg` is
unused and is a NOP. However, x87 uses a floating-point stack, and if
the `copy_from_reg` is optimized out it won't emit a pop needed to
remove the unused result.
The prior solution for this was to attach the chain from the
`copy_from_reg` to the root, which did work, however, the root is not
always available (it's set to null during legalize types). So the
alternate solution in this patch is to replace the `copy_from_reg` with
an `X86ISD::POP_FROM_X87_REG` within the X86 call lowering. This node is
the same as `copy_from_reg` except this node makes it explicit that it
may lower to an x87 FPU stack pop. Optimizations should be more cautious
when handling this node than a normal CopyFromReg to avoid removing a
required FPU stack pop.
```
ptr = alloca
ch0 = call modf ptr
t0, ch1 = pop_from_x87_reg, ch0 // t0 unused
t1, ch2 = ldr ptr, ch1
return t1
```
Using this node ensures a required x87 FPU pop is not removed due to the
DAGCombiner.
This is an alternate solution for #127976.
This was trying to hack around the intermediate VGPR requirement
to copy to AGPRs on gfx908. We should still use a copy for all
reg-to-reg cases. This should matter less these days, as we
reserve a VGPR to handle it when required (and no end to end tests
need updating).
This was also an obstacle to handling this fold for input registers
which are larger than 32-bits.
Use `-passes="regallocgreedy<[all|sgpr|wwm|vgpr]>` to insert the greedy
RA with a filter and `-regalloc-npm=<type>` to control which RA to use
in existing pipeline.
`TypeWidenVector` makes an illegal vector a larger one
e.g. in lsx
v2i32 -> v4i32
v4i16 -> v8i16
With this we can make good use of `vilvh`, `vilvl` instructions in
vector `sext`, `zext` in later pr.
Previous action is `TypePromoteInteger`, which replaces integer with a
larger one
e.g. in lsx
v2i32 -> v2i64
v4i16 -> v4i32
Fixes#126200
At the moment, GlobalISel is missing an implementation for emulated TLS
variables.
I fixed the issue by falling back to SDAG in this case, as I currently
don't have the knowledge to implement it myself.
Co-authored-by: Schaller, Sebastian <sebastian.schaller@dentsplysirona.com>
Since N2 will be reused in the fold, we cannot skip N2's undef elements
if the corresponding element in N1 is well-defined.
For example:
```
t2: v4i32 = BUILD_VECTOR Constant:i32<0>, Constant:i32<0>, Constant:i32<0>, Constant:i32<0>
t24: v4i32 = BUILD_VECTOR undef:i32, undef:i32, Constant:i32<1>, undef:i32
t11: v4i32 = vselect t8, t2, t10
```
Before this patch, we fold t11 into:
```
t26: v4i32 = sign_extend t8
t27: v4i32 = add t26, t24
```
The last element of t27 is incorrect.
Closes https://github.com/llvm/llvm-project/issues/129181.
As discussed in #126448, the fold implemented by #126448 / #128353 can
be extended to operations other than `add`. This patch extends the fold
performed by `combinei64TruncSrlAdd` to include `or` and `xor` (proof:
https://alive2.llvm.org/ce/z/AXuaQu). There's no need to extend it to
`sub` and `and`, as similar folds are already being performed for those
operations.
CC: @phoebewang @RKSimon
PTX supports 2 methods of accessing device function parameters:
- "simple" case: If a parameters is only loaded, and all loads can
address the parameter via a constant offset, then the parameter may be
loaded via the ".param" address space. This case is not possible if the
parameters is stored to or has it's address taken. This method is
preferable when possible.
- "move param" case: For more complex cases the address of the param may
be placed in a register via a "mov" instruction. This mov also
implicitly moves the param to the ".local" address space and allows for
it to be written to. This essentially defers the responsibilty of the
byval copy to the PTX calling convention.
The handling of these cases in the NVPTX backend for byval pointers has
some major issues. We currently attempt to determine if a copy is
necessary in NVPTXLowerArgs and either explicitly make an additional
copy in the IR, or insert "addrspacecast" to move the param to the param
address space. Unfortunately the criteria for determining which case is
possible are not correct, leading to miscompilations
(https://godbolt.org/z/Gq1fP7a3G). Further, the criteria for the
"simple" case aren't enforceable in LLVM IR across other transformations
and instruction selection, making deciding between the 2 cases in
NVPTXLowerArgs brittle and buggy.
This patch aims to fix these issues and improve address space related
optimization. In NVPTXLowerArgs, we conservatively assume that all
parameters will use the "move param" case and the local address space.
Responsibility for switching to the "simple" case is given to a new
MachineIR pass, NVPTXForwardParams, which runs once it has become clear
whether or not this is possible. This ensures that the correct address
space is known for the "move param" case allowing for optimization,
while still using the "simple" case where ever possible.
The following reproducer demonstrates the issue with invalid definition
of GEP results during type inference
```
define spir_kernel void @foo(i1 %fl, i64 %idx, ptr addrspace(1) %dest, ptr addrspace(3) %src) {
%p1 = getelementptr inbounds i8, ptr addrspace(1) %dest, i64 %idx
%res = tail call spir_func target("spirv.Event") @_Z22__spirv_GroupAsyncCopyjPU3AS1iPU3AS3Kimm9ocl_event(i32 2, ptr addrspace(1) %p1, ptr addrspace(3) %src, i64 128, i64 1, target("spirv.Event") zeroinitializer)
ret void
}
declare dso_local spir_func target("spirv.Event") @_Z22__spirv_GroupAsyncCopyjPU3AS1iPU3AS3Kimm9ocl_event(i32, ptr addrspace(1), ptr addrspace(3), i64, i64, target("spirv.Event"))
```
Here `OpGroupAsyncCopy` expects i32* arguments and type inference fails
to set a correct type of the GEP result `%p1`, because it is an argument
of `OpGroupAsyncCopy`.
This PR fixes the issue by preventing type change of GEP results in type
inference.
If the ControlFlowHub did not perform any change to the control flow,
there is no need to repair SSA, update the loop structure, and verify a
bunch of things. This is not completely NFC though, repairSSA introduced
PHI nodes with a single entry that are now missing.
My code went from 400+ seconds to 1 second, since no loop required the
exits to be unified, but there were many "complex" loops.
True16 Add OpSel when optimizing exec mask
True16 VOPCX have the opsel argument. Add it when we create these
instructions in SIOptimizeExecMasking.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
It seems that there can be other cases with this that also can lead to
wrong code (discovered with csmith). This time it involved not the kill
flag but the undef flag.
Use the intersection of the flags from both MachineOperand:s instead
of the RegState from just one of them.
This change fold together the _ari, _ari64, and _asi variants of these
instructions into a single instruction capable of holding any address.
This allows for the removal of a lot of unnecessary code and moves us
towards a standard way of representing an address in NVPTX.
Atomicrmw xchg can directly take a pointer operand, so we should
treat it similarly to store or cmpxchg.
In practice, I believe that all targets that support stack protectors
will convert this to an integer atomicrmw xchg in AtomicExpand, so
there is no issue in practice. We still should handle it correctly
if that doesn't happen.
Support true16 format for v_alignbit_b32 in MC.
Since we are replacing `v_alignbit_b32` to
`v_alignbit_b32_t16/v_alignbit_b32_fake16` in Post-GFX11, have to update
the CodeGen pattern for `v_alignbit_b32_fake16` to get CodeGen test
passing. There is no pattern modified/created, but just replacing the
`v_alignbit_b32` with fake16 format.
Some of the true16 CodeGen test are impacted since `v_alignbit_b32`
selection are removed in Post-GFX11 while `v_alignbit_b32_t16` are not
yet supported. The CodeGen patch for `v_alignbit_b32_t16` will be done
in the following patch.
