The intrinsic @llvm.amdgcn.flat.atomic.{fadd,fmax,fmin} can only be
selected for flat address spaces (constant, flat and global).
This patch restricts the cases over which GCNTTIImpl::rewriteIntrinsicWithAddressSpace
rewrites the intrinsic.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D149938
This reverts commit 5cafecf9f952818400fa32645695e79838f1bc2c.
Buildbots are not happy with the patch.
Lots of crashes and assertion failures such as
llvm::LegalizeRuleSet &llvm::LegalizerInfo::getActionDefinitionsBuilder(
std::initializer_list<unsigned int>): Assertion `Opcodes.size() >= 2 &&
"Initializer list must have at least two opcodes"' failed.
These intrinsics are used to implement the pext intrinsics that extract
two predicates (mask) from a predicate-as-counter value, e.g.
__attribute__((arm_streaming))
svboolx2_t svpext_lane_c8_x2(svcount_t pnn, uint64_t imm);
As described in https://github.com/ARM-software/acle/pull/217
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D150442
These intrinsics are used to implement the pext intrinsics that extract
a predicate (mask) from a predicate-as-counter value, e.g.
__attribute__((arm_streaming))
svbool_t svpext_lane_c8(svcount_t pnn, uint64_t imm);
As described in https://github.com/ARM-software/acle/pull/217
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D150441
This alters the lowering of shifts by a constant, so that the type is lowered
to a v1i64 instead of a i64. This helps communicate that the type will live in
a neon register, and can help clean up surrounding code. Note this is only
currently for the scalar shifts of a constant that go through the nodes in
tryCombineShiftImm.
ssra instructions are no longer being recognized in places, but that can be
cleaned up in a followup patch that combines the i64 add into a v1i64 add.
Differential Revision: https://reviews.llvm.org/D148309
Define intersectWith and unionWith as two complementary ways of
combining KnownBits. The names are chosen for consistency with
ConstantRange.
Deprecate commonBits as a synonym for intersectWith.
Differential Revision: https://reviews.llvm.org/D150443
The logic from line 633 to 640 is specific for ARM as the comments said, it will make all the targets will prefer to using instruction with more predicates when compiler do AsmMatching.
And for code from line 642 to 649, X86 want to use the order records written in source file to sort the instructions. So X86 could be affected by this logic. (These code could be arrived only by X86)
After change this, seems AVX instructions have not be affected but it exposed some other errors for instruction push and call.
CALLpcrel16 could not be used in 64 bit mode, we need add Predicate for it. And for push instruction, previously because pushi32 has predicates = [Not64bitmode], so it precede pushi16, which is incorrect here, we should get pushw here and it also align with gcc.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D150436
These aren't currently created from anything that has a passthru
operand. If we need to support this in the future, we should add
a policy operand to the ISD node definition.
The COPY inserted in the epilog block before return instruction as part
of ABI lowering, can get optimized during machine copy propagation if
the same register is used earlier in a wwm operation that demands the
prolog/epilog wwm-spill store/restore to preserve its inactive lanes.
With the spill restore in the epilog, the preceding COPY appears to be
dead during machine-cp. To avoid it, mark the same register as a tied-op
in the spill restore instruction to ensure a usage for the COPY.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D150381
The MachineUA now queries the target to determine if a given register holds a
uniform value. This is determined using the corresponding register bank if
available, or by a combination of the register class and value type. This
assumes that the target is optimizing for performance by choosing registers, and
the target is responsible for any mismatch with the inferred uniformity.
For example, on AMDGPU, an SGPR is now treated as uniform, except if the
register bank is VCC (i.e., the register holds a wave-wide vector of 1-bit
values) or equivalently if it has a value type of s1.
- This does not always work with inline asm, where the register bank or the
value type might not be present. We assume that the SGPR is uniform, because
it is not expected to be s1 in the vast majority of cases.
- The pseudo branch instruction SI_LOOP is now hard-coded to be always
divergent, although its condition is an SGPR.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D150438
For image and buffer stores the default behaviour on GFX11 and
older is to set all unset components to zero. So if we pass
only X component it will be the same as X000, or XY same as XY00.
This patch simplifies the passed vector of components in InstCombine
by removing zero components from the end.
For image stores it also trims DMask if necessary.
Reviewed By: foad, arsenm
Differential Revision: https://reviews.llvm.org/D146737
Currently any cast operation with fixed length vectors uses NEON costs,
If those operations are end up using SVE instruction then we estimate
those operations based upon SVE costs.
Differential Revision: https://reviews.llvm.org/D133955
Prior to this patch, for the DWARF frame base LLVM uses the frame pointer
register if available, otherwise the stack pointer register. If the stack
pointer register is being used and a call or other code modifies the stack
pointer during the body of the function this results in the locations being
wrong and the debugger displaying the wrong values for variables.
By using DW_OP_call_frame_cfa in these situations the emitted location for
the variable will automatically handle changes in the stack pointer.
The CFA needs to be adjusted for the offset between the frame pointer/stack
pointer to allow the variable locations themselves to remain unchanged by
this patch.
Reviewed By: #debug-info, scott.linder, jryans
Differential Revision: https://reviews.llvm.org/D143463
Splitting and improving from the https://reviews.llvm.org/D146357
When running tests for LowerShift, I discovered some poor codegen in rotate and funnel shift tests. This patch attempts to address some of them.
Using unpack for splitting and using double-bitwidth shifts may improve performance according to https://uica.uops.info tests.
No cross-lane shuffles
No dirtying double-width registers
Massive improvement for AVX2 rotates in some cases (var_funnnel_v8i16, var_funnnel_v16i16) — because unpack is currently only used for vXi8 vectors.
Differential Revision: https://reviews.llvm.org/D149071
D147786 made the transform more conservative by adding heuristics,
which was a good idea. However, the transform got a bit
too conservative at times.
This caused a surprise in some rocRAND benchmarks because D143731 greatly helped a few of them.
For instance, a few xorwow-uniform tests saw a +30% boost in performance after that pass, which was lost when D147786 landed.
This patch is an attempt at reaching a middleground that makes
the pass a bit more permissive. It continues in the same spirit as
D147786 but does the following changes:
- PHI users of a PHI node are now recursively checked. When loops are encountered, we consider the PHIs non-breakable. (Considering them breakable had very negative effect in one app I tested)
- `shufflevector` is now considered interesting, given that it satisfies a few trivial checks.
Reviewed By: arsenm, #amdgpu, jmmartinez
Differential Revision: https://reviews.llvm.org/D150266
Use it to replace isel patterns with a DAG combine of FP_EXTEND_VL+VFMADD_VL.
This makes it similar to how other widening operations are handled.
I plan to use this to make it easier to form tail undisturbed vfwmacc.
We should not call tryOptimizeLEAtoMOV() in eliminateFrameIndex() when
the base register is a virtual register, because tryOptimizeLEAtoMOV
would assume the base register be physical register. Although we can
also optimize LEA to MOV with virtual register, I'd like to leave the
optimization in another patch.
Differential Revision: https://reviews.llvm.org/D150521
If we know that zero constant operand is already in the right place we do not need
to re-order anything.
Reviewed By: e-kud
Differential Revision: https://reviews.llvm.org/D150249
Previously, LegalizeVectorOps used the result VT while LegalizeDAG
used the operand VT. This patch makes them both use the operand VT.
This also makes it consistent with how the default cost model works.
I've hacked the AArch64 cost model to maintain old behavior for some
f16 vectors.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D149572
We can detect the case under the following circumstances:
Take `(Pow2_Ceil(C1) - (1 << C0))` as `C2`.
1) `C2` is NOT a power of 2.
2) `C2 + LeastSignificantBit(C2)` is a nonzero power of 2.
3) `C2 u>= C1`
The motivation is the middle end transforms:
`(-x << C0) & C1`
to
`(x * (Pow2_Ceil(C1) - (1 << C2))) & C1`
As it saves IR instructions. On X86 the two instruction, `sub` and
`shl`, and better than the `mul` so we want to undo the transform.
This comes up when shifting a bit-mask by a byte-misalignment i.e:
`y << ((-(uintptr)x * 8) & 63)`
Alive2 Proofs (including all cases with undefs in the vector):
https://alive2.llvm.org/ce/z/f-65b6
Reviewed By: RKSimon, pengfei
Differential Revision: https://reviews.llvm.org/D150294
This was reverted in 4e84149c0aea0e9f16c51cc92f50d90992b13d57 due to a problem report which has been confirmed to be misattributed.
Original commit message:
Use a switch to avoid repeat tests on the opcode, and factor out some common code out of another switch. (Not all branches had both common bits, but the one which didn't left the values unchanged and their starting value is the respective operand - so storing it back is a nop.)
This reverts commit 13984608992246e42e66c019f09764650d60af63. A problem was reported - the attribution seems slightly questionable, but this is low enough value after 297e06c that reverting is easy.
This patch migrates uses of StringRef::{starts,ends}with_insensitive
to StringRef::{starts,ends}_with_insensitive so that we can use names
similar to those used in std::string_view. I'm planning to deprecate
StringRef::{starts,ends}with_insensitive once the migration is
complete across the code base.
Differential Revision: https://reviews.llvm.org/D150426
IndirectCopyToAGPR should be reworked as to avoid optimizing during copy lowering. However, as it stands, the code is buggy. This patch replaces the call to definesRegister with modifiesRegister, and confirms that the dest reg of the found accvgpr_write is in fact the src reg of our copy.
Differential Revision: https://reviews.llvm.org/D149873
Change-Id: Id8a61659ac15565dcb970069d0624f0925a46e6d
We can apply the same logic as for multiply since a left shift is just a multiply by a power of two. Note that since shl is not commutative, we do need to be careful to match sure that the splat is the RHS of the instruction.
Differential Revision: https://reviews.llvm.org/D150471
This has been present from the original patch which added the pass, and doesn't appear to be strongly justified. We do need to be careful of commutativity.
Differential Revision: https://reviews.llvm.org/D150468
Use a switch to avoid repeat tests on the opcode, and factor out some common code out of another switch. (Not all branches had both common bits, but the one which didn't left the values unchanged and their starting value is the respective operand - so storing it back is a nop.)
Correct the legality of i32 mul_lohi on AArch64.
Previously, AArch64 incorrectly reported i32 mul_lohi as Legal.
This allowed BuildUDIV/SDIV to use them. A later DAGCombiner would
replace them with MULHS/MULHU because only the high half was used.
This conversion does not check the legality of MULHS/MULHU under
the assumption that LegalizeDAG can turn it back into MUL_LOHI later.
After they are converted to MULHS/MULHU, DAGCombine ran and saw that
these operations aren't supported but an i64 MUL is. So they get
converted to that plus a shift. Without this, LegalizeDAG would
convert back MUL_LOHI and isel would fail to find a pattern.
This patch teaches BuildUDIV/SDIV to create the wide mul and shift
so that we can report the correct operation legality on AArch64. It
also enables div by constant folding for more cases on VE.
I don't know if VE wants this div by constant optimization or not. If they
don't want it, they can use the isIntDivCheap hook to disable it.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D150333
While pointers in address space 7 (128 bit rsrc + 32 bit offset)
should be rewritten out of the code before IR translation on AMDGPU,
higher-level analyses may still call MVT getPointerTy() and the like
on the target machine. Currently, since there is no MVT::i160, this
operation ends up causing crashes.
The changes to the data layout that caused such crashes were D149776.
This patch causes getPointerTy() to return the type MVT::v5i32
and getPointerMemTy() to be MVT::v8i32. These are accurate types,
but mean that we can't use vectors of address space 7 pointers during
codegen. This is mostly OK, since vectors of buffers aren't supported
in LPC anyway, but it's a noticable limitation.
Potential alternative solutions include adjusting getPointerTy() to return
an EVT or adding MVT::i160 and MVT::i256, both of which are rather
disruptive to the rest of the compiler.
Reviewed By: foad
Differential Revision: https://reviews.llvm.org/D150002
1. Share code `optimizeInstFromVEX3ToVEX2` with MCInstLower
2. Move the code of optimization for shift/rotate to a separate file
3. Since the function is shared, a side effect is that more encoding
optimizations are done on the Asmparser side. Considering we already
use reverse-encoding for optimization in AsmParser before this patch,
I believe the change is positive and expected.
This is a reland of D150068 with the fix D150440.
