This was originally added to preserve FMF on SETCC. Unfortunately,
it also incorrectly preserves nuw/nsw on ADD/SUB in some cases.
There's also no guarantee the new opcode is even the same opcode
as the original node.
This patch removes the code and adds code to explicitly preserve
FMF flags in the SETCC promotion function.
The other test changes are from nuw/nsw not being preserved. I
believe for all these tests it was correct to preserve the flags,
so we need new code to preserve the flags when possible. I'll post
another patch for that since it's a riskier change.
This should unblock D150769.
Differential Revision: https://reviews.llvm.org/D151472
Note that the builders are protected by is64Bit().
More fine-grained availibility checks.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D150790
Even though we only need to write to the bottom NumElts - Rotation
elements for the vslidedown.vi, we can save an extra vsetivli toggle if
we just keep the wide VL.
(I may be missing something here: is there a reason why we want to explicitly keep the vslidedown narrow?)
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D151390
This is an attempt to reland D42600 and enabling this optimisation by default.
This also resolves the issue pointed out in the context of PGO build.
Differential Revision: https://reviews.llvm.org/D42600
This adds the vfslide1down (and vfslide1up for consistency) nodes. These mostly parallel the existing vslide1down/up nodes. (See note below on instruction semantics.) We then use the vfslide1down in build_vector lowering instead of going through the stack.
The specification is more than a bit vague on the meaning of these instructions. All we're given is "The vfslide1down instruction is defined analogously, but sources its scalar argument from an f register."
We have to combine this with a general note at the beginning of section 10. Vector Arithmetic Instruction Formats which reads: "For floating-point operations, the scalar can be taken from a scalar f register. If FLEN > SEW, the value in the f registers is checked for a valid NaN-boxed value, in which case the least-signicant SEW bits of the f register are used, else the canonical NaN value is used. Vector instructions where any floating-point vector operand’s EEW is not a supported floating-point type width (which includes when FLEN < SEW) are reserved.".
Note that floats are NaN-boxed when D is implemented.
Combining that all together, we're fine as long as the element type matches the vector type - which is does by construction. We shouldn't have legal vectors which hit the reserved encoding case. An assert is included, just to be careful.
Differential Revision: https://reviews.llvm.org/D151347
The immediate field on the vsetivli is fairly limited. For larger vectors, we end up having to materialize a constant in a register. We hadn't plumbed the infrastructure to treat such materialized constants as constants for purpose of vsetvli elimination.
I only bothered to handle LI. We could extend this to LUI sequences, but well, 2048 elements is probably enough for all practical fixed length vector codegen. :)
The test delta does point out a related problem. At LMUL8, we see increased register allocation pressure, and we should probably either a) address register allocation remat, or b) be less aggressive about eliminating vsetvlis at high lmul. Note that high LMUL code is not generated much by default.
Differential Revision: https://reviews.llvm.org/D151212
Commit 8064caf83fb166b709bfe0e7641c5181341cb064 added a call
to a function that performs this combine without checking whether
the target supports FPCVT. This caused asserts to trip on BE bots
as the default target does not have this feature.
For stores of small fixed-length vector constants, we can store them
with a sequence of lui/addi/sh/sw to avoid the cost of building the
vector and the vsetivli toggle, provided the constant materialization
cost isn't too high.
This subsumes the optimisation for stores of zeroes in
4dc9a2c5b93682c12d7a80bbe790b14ddb301877
(This is a reapply of 0ca13f9d2701e23af2d000a5d8f48b33fe0878b7)
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D151221
For stores of small fixed-length vector constants, we can store them
with a sequence of lui/addi/sh/sw to avoid the cost of building the
vector and the vsetivli toggle.
Note that this only handles vectors that are 32 bits or smaller, but
could be expanded to 64 bits if we know that the constant
materialization cost isn't too high.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D151221
Live intervals for physical registers are calculated lazily on demand.
In a case like this:
16B %0:gpr32 = IMPLICIT_DEF
32B $wzr = COPY %0
if the live interval for $wzr did not already exist then the update code
in joinReservedPhysReg would create it with a definition at 32B, which
would remain even after the COPY was deleted.
Differential Revision: https://reviews.llvm.org/D151314
When the scavenger is not allowed to spill, the only difference between
forward and backward should be the heuristics used to pick an available
register. Forwards scavenging tries to pick a register that can be used
again later in the BB; backwards scavenging tries to pick one that can
be used earlier.
Backwards scavenging is preferred because it does not rely on accurate
kill flags.
Differential Revision: https://reviews.llvm.org/D151323
We only expect these ctor / dtor functions to be called with a single
thread. Add the appropriate attributes to indicate this to the backend.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D151153
For small fixed-length vector copies like
vsetivli zero, 2, e16, m1, ta, ma
vle16.v v8, (a0)
vse16.v v8, (a1)
We can scalarize them if the total vector size < XLEN:
lw a0, 0(a0)
sw a0, 0(a1)
This patch adds a DAG combine to do so, reusing much of the existing
logic in https://reviews.llvm.org/D150717
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D151103
This information helps to avoid considering cloning for blocks with indirect branches.
Reviewed By: jhenderson
Differential Revision: https://reviews.llvm.org/D150611
Add minimal support to lower return, and introduce an OutgoingValueHandler and an OutgoingValueAssigner for returns.
Supports return values with integer, pointer and aggregate types.
(Update of D69808 - avoiding commandeering that revision)
Co-authored By: lewis-revill
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D117318
There are two motivations.
`-fno-pic -fstack-protector -mstack-protector-guard=global` created
`__stack_chk_guard` is referenced directly on all ELF OSes except FreeBSD.
This patch allows referencing the symbol indirectly with
-fno-direct-access-external-data.
Some Linux kernel folks want
`-fno-pic -fstack-protector -mstack-protector-guard-reg=gs -mstack-protector-guard-symbol=__stack_chk_guard`
created `__stack_chk_guard` to be referenced directly, avoiding
R_X86_64_REX_GOTPCRELX (even if the relocation may be optimized out by the linker).
https://github.com/llvm/llvm-project/issues/60116
Why they need this isn't so clear to me.
---
Add module flag "direct-access-external-data" and set the dso_local property of
the stack protector symbol. The module flag can benefit other LLVMCodeGen
synthesized symbols that are not represented in LLVM IR.
Nowadays, with `-fno-pic` being uncommon, ideally we should set
"direct-access-external-data" when it is true. However, doing so would require
~90 clang/test tests to be updated, which are too much.
As a compromise, we set "direct-access-external-data" only when it's different
from the implied default value.
Reviewed By: nickdesaulniers
Differential Revision: https://reviews.llvm.org/D150841
We can compute a simpler expression for Lo for these cases. This
is an alternative for the test cases in D151180 that works for
more targets.
This is similar to some of the special cases we have for expanding
setcc operands.
Differential Revision: https://reviews.llvm.org/D151182
Currently AMDGPU offers extra ctor / dtor lowering by emitting a kernel
that can be called. It's possible to handle ctors and dtors using the
standard method as shown in D149340's commit message. In which case we
on't need these extra kernels as they won't be called. This patch simply
adds a way to conditionally turn off this handling if we do not want to
get extra kernels in the output.
Unrelated, but we could convert this handling to an ODR function that simply
calls the code in D149340 constructed via LLVM-IR. That would handle priority
correctly and would then be correct if not run in LTO mode.
Reviewed By: yaxunl
Differential Revision: https://reviews.llvm.org/D150565
For always poison shifts, any KnownBits return value is valid.
Currently we return unknown, but returning zero is generally more
profitable. We had some code in ValueTracking that tried to do this,
but was actually dead code.
Differential Revision: https://reviews.llvm.org/D150648
This patch analyzes AVX512 instructions for full vector width folded loads from the constant pool and attempts to determine if it can be replaced with a smaller broadcast folded variant. Typically the broadcast opportunities were missed by type-width mismatches or mulituse limitations which have been removed in later passes.
As well as introducing broadcast fold tables (which can hopefully be extended/automated in the future), this also handles mismatches in the AND/ANDN/OR/XOR/TERNLOG type-widths, catching additional missed opportunities.
This is patch is pulled from the ongoing work based on D150143, but without removing the existing DAG constant broadcast lowering code - this patch is currently a late stage cleanup only.
The intention is to add additional broadcast/extension handling of constants in future patches, but it turned out that AVX512 broadcast handling was the easiest to start with.
Differential Revision: https://reviews.llvm.org/D150526
On PowerPC VSX targets, fp-to-int will be transformed into xscv with
mfvsr. When the result is to be stored, mfvsr can be replaced by a
direct store.
This change simplifies the optimization by using existing fp-to-int
code, which helps CSE and handling strictfp cases.
Reviewed By: shchenz
Differential Revision: https://reviews.llvm.org/D141473
A couple of tests were setting liveins to add fake live registers, but
that only works if you track liveness forwards. Add some implicit uses
too, so that it also works if you track liveness backwards.
Differential Revision: https://reviews.llvm.org/D150559
If we have legal f16 instructions but no f16 med3, we can save
one instruction by expanding out the min/max sequence compared
to casting to f32 and casting back.
This is basically a partial revert of https://reviews.llvm.org/D145586 ( fd1d60873fdc )
D145586 was originally introduced to help with SWDEV-363662, and it did, but
it also caused a 25% drop in performance in
some MIOpen benchmarks where, it seems,
functions are inlined more conservatively.
This patch restores the pre-D145586 behavior
for PromoteAlloca: functions with a non-entry CC
have a 32 VGPRs threshold, but only if the function
is not marked with "alwaysinline".
A good number of AMDGPU code makes uses of
the AMDGPUAlwaysInline pass anyway, so in our
backend "alwaysinline" seems very common.
This change does not affect SWDEV-363662 (the motivating issue for introducing D145586).
Fixes SWDEV-399519
Reviewed By: rampitec, #amdgpu
Differential Revision: https://reviews.llvm.org/D150551
This assert should have the same set of vector types as the binary
and ternary case (although this assert is kind of pointless, the code
should work for any vector type as-is).
Fixes part of issue #32650.
If the second argument of V_SET_INACTIVE is undef/poison,
SIWholeQuadMode lowered it to a COPY from the first argument, but that
caused invalid MIR if the first argument was an immediate rather than a
register.
Fix this by lowering to a V_MOV instruction instead of a COPY.
Fixes https://github.com/llvm/llvm-project/issues/62862
Differential Revision: https://reviews.llvm.org/D151105
The convert intrinsics can be used to implement existing operations on svcount_t
when the actual bits/content of the predicate register doesn't matter (such
as PSEL, which copies the full contents of the first source register to the
destination register).
Reviewed By: CarolineConcatto, david-arm
Differential Revision: https://reviews.llvm.org/D150959
All the bits of the first operand are copied to the destination register,
if the tested bit (in the second source operand) is active. This means we
copy over all vscale x 16 x i1's of the first operand. There is no need to
overload that type.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D150958
These intrinsics are used to implement multi-vector load/store intrinsics that loads
or stores a tuple of 2 or 4 values, based on a predicate-as-counter operand, e.g.
__attribute__((arm_streaming))
svuint8x2_t svld1[_u8]_x2(svcount_t png, const uint8_t *rn);
__attribute__((arm_streaming))
void svst1[_u8_x2](svcount_t png, uint8_t *rn, svuint8x2_t zt);
As described in https://github.com/ARM-software/acle/pull/217
Reviewed By: CarolineConcatto
Differential Revision: https://reviews.llvm.org/D150956
These intrinsics are used to implement the sel intrinsics that selects
a tuple of 2 or 4 values based on a predicate-as-counter operand, e.g.
__attribute__((arm_streaming))
svuint8x2_t svsel[_u8_x2](svcount_t png, svuint8x2_t zn, svuint8x2_t zm);
__attribute__((arm_streaming))
svuint8x4_t svsel[_u8_x4](svcount_t png, svuint8x4_t zn, svuint8x4_t zm);
As described in https://github.com/ARM-software/acle/pull/217
Reviewed By: CarolineConcatto
Differential Revision: https://reviews.llvm.org/D150951
