If all the demanded bits of the AND mask covering the inserted subvector 'X' are known to be one, then the mask isn't affecting the subvector at all.
In which case, if the base vector 'C' is undef/constant, then move the AND mask up to just (constant) fold it directly.
Addresses some of the regressions from D129150, particularly the cases where we're attempting to zero the upper elements of a widened vector.
Differential Revision: https://reviews.llvm.org/D129290
This is almost the same as the abandoned D48529, but it
allows splat vector constants too.
This replaces the x86-specific code that was added with
the alternate patch D48557 with the original generic
combine.
This transform is a less restricted form of an existing
InstCombine and the proposed SDAG equivalent for that
in D128080:
https://alive2.llvm.org/ce/z/OUm6N_
Differential Revision: https://reviews.llvm.org/D128123
Current implementation will rename both register in store instructions if
we store base address into memory with same base register, it's OK if
the offset is 0, however that is wrong transform if offset isn't 0, give
a smalle example here:
sd a0, 808(a0)
We should not transform into:
addi a2, a0, 768
sd a2, 40(a2)
That should just rename base address like this:
addi a2, a0, 768
sd a0, 40(a2)
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D128876
Use following example to demo what happened now:
li a1, 1
sd a1, 800(a0)
sd a0, 808(a0) # Store base address into base + offset
li a1, 2
sd a1, 816(a0)
Current will optimizate into:
li a1, 1
addi a2, a0, 768
sd a1, 32(a2)
sd a2, 40(a2) # Wrong replacement for the source register.
li a1, 2
sd a1, 48(a2)
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D128875
Add GFX11 test coverage to a bunch of tests where it was easy to do so,
mostly because the checks are autogenerated and/or GFX11 can share the
same checks as GFX10.
Differential Revision: https://reviews.llvm.org/D129295
This change replaces the C++ predicates with the HasNoUse builtin
predicate that would enable the no-ret atomic op selection in
GlobalISel.
Differential Revision: https://reviews.llvm.org/D125213
When add has additional users, we should indentify whether add's
user is phi that forms loop rather than root's.
Reviewed By: LuoYuanke
Differential Revision: https://reviews.llvm.org/D129169
Try to re-use an already extended operand for SetCC with vector operands
feeding an extended select. Doing so avoids requiring another full
extension of the SET_CC result when lowering the select.
This improves lowering for certain extend/cmp/select patterns operating.
For example with v16i8, this replaces 6 instructions for the extra extension
with 4 separate selects.
This improves the generated code for loops like the one below in
combination with D96522.
int foo(uint8_t *p, int N) {
unsigned long long sum = 0;
for (int i = 0; i < N ; i++, p++) {
unsigned int v = *p;
sum += (v < 127) ? v : 256 - v;
}
return sum;
}
https://clang.godbolt.org/z/Wco866MjY
On the AArch64 cores I have access to, the patch improves performance of
the vector loop by ~10%.
This could be generalized per follow-ups, but the initial version
targets one of the more important cases in combination with D96522.
Alive2 modeling:
* sext EQ https://alive2.llvm.org/ce/z/5upBvb
* sext NE https://alive2.llvm.org/ce/z/zbEcJp
* zext EQ https://alive2.llvm.org/ce/z/_xMwof
* zext NE https://alive2.llvm.org/ce/z/5FwKfc
* zext unsigned predicate: https://alive2.llvm.org/ce/z/iEwLU3
* sext signed predicate: https://alive2.llvm.org/ce/z/aMBega
Reviewed By: dmgreen
Differential Revision: https://reviews.llvm.org/D120481
Make sure we include at least one case where the vsadd/vmsltu lowering
requires only LMUL1. We should be able to generate all of the fixed
vector variants from scalar to vector idioms, but this is probably not
very important right now given the fixed length variants we'd actually
use when vectorizing with LMUL=1 are reasonable.
This patchs adds a new metadata kind `exclude` which implies that the
global variable should be given the necessary flags during code
generation to not be included in the final executable. This is done
using the ``SHF_EXCLUDE`` flag on ELF for example. This should make it
easier to specify this flag on a variable without needing to explicitly
check the section name in the target backend.
Depends on D129053 D129052
Reviewed By: jdoerfert
Differential Revision: https://reviews.llvm.org/D129151
Currently we use the `.llvm.offloading` section to store device-side
objects inside the host, creating a fat binary. The contents of these
sections is currently determined by the name of the section while it
should ideally be determined by its type. This patch adds the new
`SHT_LLVM_OFFLOADING` section type to the ELF section types. Which
should make it easier to identify this specific data format.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D129052
SPIR-V module typically contains some global entities that were not
global before made it to SPIR-V, e.g. types and constants are not usually
declared globally in LLVM. By design SPIR-V requires such stuff to be declared
once and in the module's global section. Since MIR is not able to represent
such things properly they were generated per-function, and then at the very end
of the backend's pipeline hoisted into some 'meta' function minding possible
duplicates.
New SPIRVDuplicatesTracker keeps mapping of the original LLVM entities such
as types, constant, global variables, etc to their MIR counterparts -
(MachineFunction, Register). Later SPIRVModuleAnalysis (apart from other
thing it's responsible for) performs topological sorting of the
tracker's entries to ensure proper ordering before the hoisting,
and actually performs the hoisting in a duplicates-free manner
by the tracker's nature.
Differential Revision: https://reviews.llvm.org/D128471
This is done during type legalization since the target representation of
these nodes may not be valid until after type legalization, and after
type legalization the fact that these are dealing with i1 types may be
lost.
Differential Revision: https://reviews.llvm.org/D128996
Instead of using <vscale x 16 x i1> for all the loads/stores, we now use the appropriate
predicate type according to the element size, e.g.
ld1b uses <vscale x 16 x i1>
ld1w uses <vscale x 4 x i1>
ld1q uses <vscale x 1 x i1>
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D129083
Truncates and compares require some changes to generic legalisation functions
to use ElementCount instead of getNumElements.
Reviewed By: paulwalker-arm
Differential Revision: https://reviews.llvm.org/D129082
`dxil` is an architecture supported by the DirectX backend. These
intrinsics will likely be shared with other DirectX architectures like
`dxbc`. Using a common prefix `dx` will make it more intuitive.
Also the `dx` prefix is already set in the Triple, which causes
intrinsics described here to be unmatchable via the ClangBuiltin
mechanism.
When the predicate vector is unpacked, we cannot assume anything about the
values in the other lanes. We have to make sure we use the correct
predicate where we know that the other lanes have been zeroed.
Reviewed By: RosieSumpter
Differential Revision: https://reviews.llvm.org/D129081
Prior to this change, live variable operands passed to
`llvm.experimental.stackmap` would be emitted directly to target nodes,
meaning that they don't get legalised. The upshot of this is that LLVM
may crash when encountering illegally typed target nodes.
e.g. https://github.com/llvm/llvm-project/issues/21657
This change introduces a platform independent stackmap DAG node whose
operands are legalised as per usual, thus avoiding aforementioned
crashes.
Note that some kinds of argument are still not handled properly, namely
vectors, structs, and large integers, like i128s. These will need to be
addressed in follow-up changes.
Note also that this does not change the behaviour of
`llvm.experimental.patchpoint`. A follow up change will do the same for
this intrinsic.
Differential review:
https://reviews.llvm.org/D125680
This patch adds the support for `fmax` and `fmin` operations in `atomicrmw`
instruction. For now (at least in this patch), the instruction will be expanded
to CAS loop. There are already a couple of targets supporting the feature. I'll
create another patch(es) to enable them accordingly.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D127041
Add support for the RDPRU instruction on Zen2 processors.
User-facing features:
- Clang option -m[no-]rdpru to enable/disable the feature
- Support is implicit for znver2/znver3 processors
- Preprocessor symbol __RDPRU__ to indicate support
- Header rdpruintrin.h to define intrinsics
- "rdpru" mnemonic supported for assembler code
Internal features:
- Clang builtin __builtin_ia32_rdpru
- IR intrinsic @llvm.x86.rdpru
Differential Revision: https://reviews.llvm.org/D128934
These operations include sdiv/udiv/srem/urem.
As the ISA [https://loongson.github.io/LoongArch-Documentation/LoongArch-Vol1-EN.html#_div_wudu_mod_wudu]
described, when the divisor is 0, the result can be any value, but no
exception will be triggered. Unlike gcc, which by default emit code
that checks divide-by-zero after the division or modulus instruction,
we only emit this check when the `-loongarch-check-zero-division`
option is passed.
Differential Revision: https://reviews.llvm.org/D128572
D128820 stopped creating div/rem constant expressions by default;
this patch removes support for them entirely.
The getUDiv(), getExactUDiv(), getSDiv(), getExactSDiv(), getURem()
and getSRem() on ConstantExpr are removed, and ConstantExpr::get()
now only accepts binary operators for which
ConstantExpr::isSupportedBinOp() returns true. Uses of these methods
may be replaced either by corresponding IRBuilder methods, or
ConstantFoldBinaryOpOperands (if a constant result is required).
On the C API side, LLVMConstUDiv, LLVMConstExactUDiv, LLVMConstSDiv,
LLVMConstExactSDiv, LLVMConstURem and LLVMConstSRem are removed and
corresponding LLVMBuild methods should be used.
Importantly, this also means that constant expressions can no longer
trap! This patch still keeps the canTrap() method to minimize diff --
I plan to drop it in a separate NFC patch.
Differential Revision: https://reviews.llvm.org/D129148
