The iterator over super and sub registers doesn't include both 8-bit
registers in its list. So if both registers are used and only one of
them is live on return, then we need to make sure that the other 8-bit
register is also marked as live and not zeroed out.
Reviewed By: nickdesaulniers
Differential Revision: https://reviews.llvm.org/D139679
At least some of the seemingly-bad codegen was because we really need to
consume the replicated i1 mask by the mask-consuming instruction
if we want faithful codegen for the replication shuffle.
But some of the odd codegen is still there.
Materialize zeros by copying from %g0, which is now marked as constant.
This makes it possible for some common operations (like integer negation) to be
performed in fewer instructions.
This continues @arichardson's patch at D132561.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D138887
This is an alternative patch on a path to D137530.
The basic problem being tackled here is that we need to place a scalar into lane 0 of a vector register before our reduction instructions. Since we only care about lane 0 of the vector, we can use any VL >= 1 provided that the total amount of work performed matches the work performed for a VL=1.
This change does not contain the logic from D137530 to perform the insert at the original VT, and then extract down to LMUL1. That turns out to be a good choice, as discussion in this review has indicated there are issues around LMUL2 and above with our representation of vmv.s.x. We'd also need to be careful with the splat logic for the same reasons.
The only potentially concerning codegen change I spot here is that we stop using a broadcast load (for VL=1) and instead do a scalar load and insert. I think this is probably reasonable; if reviewers disagree, I can investigate using a broadcast load which writes to the undef lanes. If we want to do that, we should do it for VECTOR_INSERT_ELT as well, so that'll end up as it's own patch series.
Differential Revision: https://reviews.llvm.org/D139656
Due to reversed arguments, the loop start was almost always skipping the
whole loop, since FinalStackProbed is probably less than StackPtr for
large alignments. The intent was to skip the loop if the first sub on
StackPtr made it less than FinalStackProbed already, so flip it.
Reviewed By: serge-sans-paille
Differential Revision: https://reviews.llvm.org/D139756
The implementation is inspired by code from PPCInstrInfo.
We look for a copy from X10(A0) preceded by an ADJCALLSTACKUP and
a PseudoCall. We use the PseudoCall to look up the IR function being
called to find it's return attributes.
Reviewed By: asb
Differential Revision: https://reviews.llvm.org/D139808
An expression of the form `gep(base, select(pred, const1, const2))` can result
in a set of offsets instead of just one. PointerInfo can now track these sets
instead of conservatively modeling them as Unknown. In general, AAPointerInfo
now uses AAPotentialConstantValues to examine the operands of the GEP.
Reviewed By: jdoerfert
Differential Revision: https://reviews.llvm.org/D138646
This patch adds lowering for function calls with variadic number of
arguments as well as enables support for the following
instructions/intrinsics:
- va_arg
- va_start
- va_end
- va_copy
Note that this patch doesn't intent to include clang's support for
variadic functions for CUDA.
According to the docs:
PTX version 6.0 supports passing unsized array parameter to a
function which can be used to implement variadic functions. [0]
The last parameter in the parameter list may be a .param array of
type .b8 with no size specified. It is used to pass an arbitrary
number of parameters to the function packed into a single array
object.
When calling a function with such an unsized last argument, the last
argument may be omitted from the call instruction if no parameter is
passed through it. Accesses to this array parameter must be within
the bounds of the array. The result of an access is undefined if no
array was passed, or if the access was outside the bounds of the
actual array being passed. [1]
Note that aggregates passed by value as variadic arguments are not
currently supported.
[0] https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#variadic-functions
[1] https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#kernel-and-function-directives-func
Differential Revision: https://reviews.llvm.org/D138531
If we're initializing lane 0 of an undef vector, we can optionally write to other lanes of the vector. Doing so may require additional work, so we don't want to e.g. always use a splat. However, since we don't have an immediate form of vmv.s.x it's useful to use a vmv.v.i if the work required is expected to be equal in practice. We restrict this to when LMUL <= 1 to a) prevent doing additional work at higher LMULs, and b) avoid overconstraining the register allocator.
At the moment, the new utility is only used by one case in INSERT_VECTOR_ELT lowering. My expectation is that we will reuse this in a couple other places, but each of those deserve individual review.
This change is inspired by D137530, but is not directly related to it. I vaguely remember we discussed the tradeoffs of using vmv.v.i in another recent review, but couldn't find it.
Differential Revision: https://reviews.llvm.org/D139648
Adds Workgroup Processor Mode (WGP) to the HSA Metadata for Code Object v5/GFX10+.
The field is already present as an asm directive and in the compute program resource register but is also needed in the MD.
Reviewed By: kzhuravl
Differential Revision: https://reviews.llvm.org/D139931
- [Clang] Declare AMDGPU target as supporting BF16 for storage-only purposes on amdgcn
- Add Sema & CodeGen tests cases.
- Also add cases that D138651 would have covered as this patch replaces it.
- [AMDGPU] Add BF16 storage-only support
- Support legalization/dealing with bf16 operations in DAGIsel.
- bf16 as a type remains illegal and is represented as i16 for storage purposes.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D139398
Extend the <0,Scale,2*Scale,..> pattern to allow for a fixed offset <Offset,Offset+Scale,Offset+2*Scale,..> pattern, which will lower to a single additional bitshift/pshufd.
At the moment I've limited this to cases where the LHS/RHS operands are concatenated for free, but this is only to avoid a couple of regressions that should be easily addressable in followups.
Now with fix to limit added tagged-globals.ll to risc-v platform
--
[RISC-V][HWASAN] Support tagging global variables for RISC-V HWASAN
Reviewed by: luismarques
Differential Revision: https://reviews.llvm.org/D132995
In order to be compatible with the form of the atomic instruction in
GAS that accepts the fourth operand as 0 (i.e. `am* $rd, $rk, $rj, 0`),
we need to treat `$rj, 0` as one operand, but only print `$rj`.
For this, the number of result operands of inline assembly memory
operand `ZB` constraint is modified to 2 (reg + 0).
Restrictions on register usage in `am*` instructions have also been
adjusted. When `$rd` is equal to `$r0`, the instruction must be
considered legal, because of some special usage like `PseudoUNIMP`.
Reviewed By: SixWeining, xen0n
Differential Revision: https://reviews.llvm.org/D139303
Copied from the existing llvm.amdgcn.class handling; eventually I will
fold that to the generic intrinsic when legal. The tests should
probably move into an instsimplify only test.
This matches the coverage with the Costmodel tests,
adds stride 5/7/8, and improves AVX512 checks.
I *think* i've compressed check prefixes
as much as possible, but it's a bit hard to tell.
But hey, at one no longer has to fight against FileCheck+UTC :).
This matches the coverage with the Costmodel tests,
adds stride 5/7/8, and improves AVX512 checks.
I *think* i've compressed check prefixes
as much as possible, but it's a bit hard to tell.
But hey, at one no longer has to fight against FileCheck+UTC :).
For reductions, we need to put the start value into a source vector. For fractional LMULs, we can perform the operation at the original LMUL. For LMUL > 1, we eventually want to use a scalar insert, but that's outside the scope of this patch.
Differential Revision: https://reviews.llvm.org/D139747
This tries to approach the problem noted by @arsenm:
terrible codegen for `__builtin_fpclassify()`:
https://godbolt.org/z/388zqdE37
Just because the PHI in the common successor happens to have different
incoming values for these two blocks, doesn't mean we have to give up.
It's quite easy to deal with this, we just need to produce a select:
https://alive2.llvm.org/ce/z/000srb
Now, the cost model for this transform is rather overly strict,
so this will basically never fire. We tally all (over all preds)
the selects needed to the NumBonusInsts
Differential Revision: https://reviews.llvm.org/D139275
This is a recommit of f9e0390751cb5eefbbbc191f851c52422acacab1
The previous commit failed to handle cases where the zero extended operand is an extended `BUILD_VECTOR`.
We don't replace zext with a sext operand to select smull if any operand is `BUILD_VECTOR`
Original commit message:
we can safely replace a `zext` instruction with `sext` if the top bit is zero. This is useful because we can select `smull` when both operands are sign extended.
Reviewed By: fhahn, dmgreen
Differential Revision: https://reviews.llvm.org/D134711
The tests in question are not particularly useful for testing complex
deinterleaving, especially due to i64 complex adds not being supported in mve.
The tests are being removed as they are hitting an unrelated pre-existing
condition regarding register spilling.
The prior code worked before SVE DIV was enabled 128 bit vectors.
With 128 bit vectors, when run on a 256 bit machine, it would split and
do a signed unpack, but this resulted in one full vector and one empty
vector with a half-sized predicate. The effect was that only half the
elements were treated correctly.
The fix is to bisect the vector, sign extend, do the division, truncate
and then concat.
Fixes#59357.
Differential Revision: https://reviews.llvm.org/D139618
BasicAA currently has an optional dependency on the PhiValues
analysis. However, at least with our current pipeline setup, we
never actually make use of it. It's possible that this used to work
with the legacy pass manager, but I'm not sure of that either.
Given that this analysis has not actually been in use for a long
time, and nobody noticed or complained, I think we should drop
support for it and focus on one code path. It is worth noting that
analysis quality for the non-PhiValues case has significantly
improved in the meantime.
If we really wanted to make use of PhiValues, the right way would
probably be to pass it in via AAQI in places we want to use it,
rather than using an optional pass manager dependency (which are
an unpredictable PITA and should really only ever be used for
analyses that are only preserved and not used).
Differential Revision: https://reviews.llvm.org/D139719
This adds a fold which teaches the backend to fold
splat(bitcast(buildvector(x,..))) or
splat(bitcast(scalar_to_vector(x))) to a single splat.
This only handles lane 0 splats, which are only valid under LE, and
needs to be a little careful with the types it creates for the new
buildvector.
Differential Revision: https://reviews.llvm.org/D139611
An expression of the form `gep(base, select(pred, const1, const2))` can result
in a set of offsets instead of just one. PointerInfo can now track these sets
instead of conservatively modeling them as Unknown. In general, AAPointerInfo
now uses AAPotentialConstantValues to examine the operands of the GEP.
Reviewed By: jdoerfert
Differential Revision: https://reviews.llvm.org/D138646
The patch also added function expandVPBITREVERSE to expand ISD::VP_BITREVERSE nodes.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D139697
