Add ptrtoaddr tests with address spaces with unstable and external but
stable pointer representations.
Currently we incorrectly form ptrtoaddr for unstsable pointers. See
discussion in https://github.com/llvm/llvm-project/pull/178861 for more
details.
Now that getModRefInfo for calls handles read and write effects by
examining both calls, the clobbering query no longer needs to treat
reads as clobbers. Update the check to consider writes only, aligning
call handling with other instructions
Make it possible to use `s_alloc_vgpr` at the IR level. This is a huge
footgun and use for anything other than compiler internal purposes is
heavily discouraged. The calling code must make sure that it does not
allocate fewer VGPRs than necessary - the intrinsic is NOT a request to
the backend to limit the number of VGPRs it uses (in essence it's not so
different from what we do with the dynamic VGPR flags of the
`amdgcn.cs.chain` intrinsic, it just makes it possible to use this
functionality in other scenarios).
Currently vector splice intrinsics are costed through getShuffleCost
when the offset is fixed. When the offset is variable though we can't
use a shuffle mask so it currently returns invalid.
This implements the cost in RISCVTTIImpl::getIntrinsicInstrCost as the
cost of a slideup and a slidedown, which matches the codegen.
It also implements the type based cost whenever the offset argument
isn't available.
It may be possible to reduce the cost in future when one of the vector
operands is known to be poison, in which case we only generate a single
slideup or slidedown.
When `Zvabd` exists, `llvm.abs` is lowered to `vabs.v` so the cost
is 1.
Reviewers: mshockwave, topperc, lukel97, skachkov-sc, preames
Reviewed By: topperc
Pull Request: https://github.com/llvm/llvm-project/pull/180146
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
For some dependence tests, there are corner cases where `BTC <s 0`. This
patch adds test cases for those situations. Notably, in these tests,
dependencies are missed even though no overflow occurs.
This patch adds a case in getIntrinsicInstrCost and
getTypeBasedIntrinsicInstrCost in
llvm/include/llvm/CodeGen/BasicTTIImpl.h for Intrinsic::clmul. This
patch uses TLI->isOperationLegalOrCustom to check if the instruction is
cheap. If not cheap, it sums up the cost of the arithmetic operations
(AND, SHIFT, XOR) multiplied by the bit width.
Fixes#176354
Fixes#176208. Scaled back version of #176515 that only affects the RISCV backend.
Only modifies the cost for cases when DIV is a legal operation.
Updates the cost for both Scalar and Vector types.
Used `TTI::TCC_Expensive` as suggested by
https://github.com/llvm/llvm-project/issues/176208#issuecomment-3760902537.
---------
Co-authored-by: Luke Lau <luke_lau@icloud.com>
This commit introduces the VectorInstrContext (VIC) infrastructure to
improve cost estimates for insert/extracts based on the context
instruction in which the insert/extract is used.
This is similar to CastContextHint, and allows providing context on how
the insert/extract is going to be used before creating IR. This is
useful in the LoopVectorizer, where costs need to estimated before
creating IR.
The new hint currently only replaces an existing check in AArch64,
but new uses will be introduced in follow-ups, including
https://github.com/llvm/llvm-project/pull/177201.
PR: https://github.com/llvm/llvm-project/pull/175982
When inserting elements into <vscale x N x i64> scalable vectors, the X86 backend now returns InstructionCost::getInvalid() instead of panicking.
Fixes#176823
Following on from #170796, this PR implements the second part of
https://discourse.llvm.org/t/rfc-allow-non-constant-offsets-in-llvm-vector-splice/88974
by allowing non-constant offsets in the vector splice intrinsics.
Previously @llvm.vector.splice had a restriction enforced by the
verifier that the offset had to be known to be within the range of the
vector at compile time. Because we can't enforce this with non-constant
offsets, it's been relaxed so that offsets that would slide the vector
out of bounds return a poison value, similar to
insertelement/extractelement.
@llvm.vector.splice.left also previously only allowed offsets within the
range 0 <= Offset < N, but this has been relaxed to 0 <= Offset <= N so
that it's consistent with @llvm.vector.splice.right.
In lieu of the verifier checks that were removed, InstSimplify has been
taught to fold splices to poison when the offset is out of bounds.
The cost model isn't implemented in this PR, and just returns invalid
for any non-constant offsets for now. I think the correct way to cost
these non-constant offets isn't through getShuffleCost because they
can't handle variable masks, but instead just through
getIntrinsicInstCost.
Fixes#175909
We compute the cost of a gather/scatter by multiplying the cost of the
scalar element type memory op by the estimated number of elements. On
rv32 though a scalar i64 load costs 2, even if we have zve64x.
This causes the cost to diverge between a vector of f64 and vector of
i64, even though both are the same. This fixes it by just using
TTI::TCC_Basic as the scalar memory op cost. The element type is checked
to be legal at this point.
I think we have the same issue for the strided op cost, but we don't
have test coverage for it yet.
This adds costsize (and latency / code-lat) costs for AArch64's
getCastInstr. This mostly affects how much unrolling is performed, and
helps bring the codesize cost in line with the number of instructions we
generate.
On Apple Silicon cores, single-element LD1 have should have the same
performance characteristics as regular loads. Consider the insert for
insertelement (load) as free
In terms of additional code-gen changes, this enables unrolling in more
cases, due to the dynamic cost of vector loops with scalar loads +
insert being estimated more accurately.
PR: https://github.com/llvm/llvm-project/pull/175970
DA uses `DependenceInfo::isKnownPredicate` instead of
`ScalarEvolution::isKnownPredicate` in several places. The former is
intended to be a "wrapper" for the later. Specifically, it performs the
following processes:
- Replace `zext(X) cmp zext(Y)` with `X cmp Y`.
- Replace `X >=s Y` with `X - Y >=s 0`
- Replace `X <=s Y` with `X - Y <=s 0`
- Replace `X >s Y` with `X - Y >s 0`
- Replace `X <s Y` with `X - Y <s 0`
The first one can return an incorrect result when the most significant
bit of `X` and `Y` are different. Everything other than the first one
can be incorrect when `X - Y` overflows. Actually, when a `SCEVUnknown`
is involved (e.g., `%n <s %n + 1` will be `0 <s 1`), this function often
returns a result that ignore the possibility of overflow.
This patch removes `DependenceInfo::isKnownPredicate` and replace all
uses of it with `ScalarEvolution::isKnownPredicate`. There is a
degradation in some test cases, but this should be correct.
Resolve#169810
There's a divergence with the rv32 costs that I plan on fixing in
another patch, so this precommits the tests for them.
The zve32f RUN lines were split off into another file so the check prefixes
are easier to reason about.
The -riscv-v-vector-bits-max RUN lines were also removed to simplify the
check prefixes since I'm not sure if they were intentionally testing any
specific logic.
`dereferenceable(<n>)` with n being potentially zero can come up when
using an operand bundle with a variable size. Currently this implies
that the pointer is non-null, even though `[nullptr, nullptr)` is a
valid range in any programming language I'm aware of. This patch removes
this implication and updates the language reference to reflect that
`dereferenceable` with a zero argument is valid.
It's time to enable this option so we can take the next steps,
see also the discussion in #175173. I.e., this is required to see
if we can take advantage of the size of statically declared
arrays during delinearization.
The monotonicity definition states its domain as follows:
```
/// The property of monotonicity of a SCEV. To define the monotonicity, assume
/// a SCEV defined within N-nested loops. Let i_k denote the iteration number
/// of the k-th loop. Then we can regard the SCEV as an N-ary function:
///
/// F(i_1, i_2, ..., i_N)
///
/// The domain of i_k is the closed range [0, BTC_k], where BTC_k is the
/// backedge-taken count of the k-th loop
```
Current monotonicity check implementation doesn't match this definition
because:
- Just checking nowrap property of addrecs recursively is not sufficient
to ensure monotonicity over the entire domain. The nowrap property may
hold for certain paths but not for all possible iteration combinations
of nested loops.
- It doesn't consider cases where exact backedge-taken counts are
unknown.
Therefore we need to fix either the definition or the implementation.
This patch adds the test cases that demonstrate this mismatch.
When locating the join blocks of a divergent block, the algorithm relies
on pseudo-edges from the header of a reducible cycle to the cycle exits.
This was missed in the actual traversal, producing unnecessary joins
inside the reducible cycle. This caused an assert in the included test,
which expected that if a join existed in a reducible cycle for a
divergent branch outside the cycle, then it must be header.
This fixes the reverted commit from #174117
Some machines are able to make use of AUIPC + ADDI or LUI + ADDI fusion, make sure to consider that in the cost model for `RISCVTTIImpl::getConstantPoolLoadCost()`.
This reverts commit 371fad28244a16f64e707eb8e9889202fdacf85f.
The change only fixes the superficial assertion. The real problem is that bb.3
and bb.4 should not have been identified as joins of bb.5
Some of the MIR test hit a bug where it errors if there is a
raw global reference as the referenced value. Worked around some
of those by just keeping a no-op bitcast constant expression.
This PR implements the first change outlined in
https://discourse.llvm.org/t/rfc-allow-non-constant-offsets-in-llvm-vector-splice/88974?u=lukel
In order to allow non-immediate offsets in the llvm.vector.splice
intrinsic, we need to separate out the "shift left" and "shift right"
modes into two separate intrinsics, which were previously determined by
whether or not the offset is positive or negative.
The description in the LangRef has also been reworded in terms of
sliding elements left or right and extracting either the upper or lower
half as opposed to extracting from a certain index, which brings it
inline with the definition of `llvm.fshr.*`/`llvm.fshl.*`.
This patch teaches AutoUpgrade.cpp to upgrade the old intrinsics into
their new equivalent one based on their offset, so existing uses of
vector.splice should still work.
Uses of llvm.vector.splice in `llvm/test/CodeGen` haven't been replaced
in this PR to keep the diff small and kick the tyres on the AutoUpgrader
a bit. I planned to do this in a follow up NFC but can include it in
this PR if reviewers prefer.
Similarly the shuffle costing kind `SK_Splice` has just been kept the
same for now, to be split into `SK_SpliceLeft` and `SK_SpliceRight`
later.
ptrtoaddr can be handled the same as ptrtoint here. The pointer known
bits cover the full pointer width, and ptrtoaddr either passes those
through directly or truncates to the address size.
This reverts commit 2c376ffeca490a5732e4fd6e98e5351fcf6d692a because it
breaks assembler.
```
$ llvm-mc -triple=amdgcn -mcpu=gfx1250 -show-encoding <<< "v_wmma_i32_16x16x64_iu8 v[16:23], v[0:7], v[8:15], v[16:23] matrix_b_reuse"
v_wmma_i32_16x16x64_iu8 v[16:23], v[0:7], v[8:15], v[16:23] clamp ; encoding: [0x10,0x80,0x72,0xcc,0x00,0x11,0x42,0x1c]
```
We have a fundamental issue in the clamp support in VOP3P instructions,
which will need more changes.
