LLVM currently uses LDAR/STLR and variants for acquire/release
as well as seq_cst operations. This is fine as long as all code uses
this convention.
Normally LDAR/STLR act as one way barriers but when used in
combination they provide a sequentially consistent model. i.e.
when an LDAR appears after an STLR in program order the STLR
acts as a two way fence and the store will be observed before
the load.
The problem is that normal loads (unlike ldar), when they appear
after the STLR can be observed before STLR (if my understanding
is correct). Possibly providing weaker than expected guarantees if
they are used for ordered atomic operations.
Unfortunately in Microsoft Visual Studio STL seq_cst ld/st are
implemented using normal load/stores and explicit fences:
dmb ish + str + dmb ish
ldr + dmb ish
This patch uses fences for MSVC target whenever we write to the
memory in a sequentially consistent way so that we don't rely on
the assumptions that just using LDAR/STLR will give us sequentially
consistent ordering.
Differential Revision: https://reviews.llvm.org/D141748
Change-Id: I48f3500ff8ec89677c9f089ce58181bd139bc68a
This copies the atomic-ops file.
Committed without review.
Differential Revision: https://reviews.llvm.org/D141964
Change-Id: I7d7b05339d9ca23f88848c56f73a3e4d3e1abeba
Occupancy is expressed as waves per SIMD. This means that we need to
take into account the number of SIMDs per "CU" or, to be more precise,
the number of SIMDs over which a workgroup may be distributed.
getOccupancyWithLocalMemSize was wrong because it didn't take SIMDs
into account at all.
At the same time, we need to take into account that WGP mode offers
access to a larger total amount of LDS, since this can affect how
non-power-of-two LDS allocations are rounded. To make this work
consistently, we distinguish between (available) local memory size and
addressable local memory size (which is always limited by 64kB on
gfx10+, even with WGP mode).
This change results in a massive amount of test churn. A lot of it is
caused by the fact that the default work group size is 1024, which means
that (due to rounding effects) the default occupancy on older hardware
is 8 instead of 10, which affects scheduling via register pressure
estimates. I've adjusted most tests by just running the UTC tools, but
in some cases I manually changed the work group size to 32 or 64 to make
sure that work group size chunkiness has no effect.
Differential Revision: https://reviews.llvm.org/D139468
Unlike older ASICs GFX10+ have a lot of VGPRs. Therefore, it is possible
to achieve high occupancy even with all or almost all addressable VGPRs
used. Our scheduler was never tuned for this scenario. The VGPR Critical
Limit threshold always comes very high, even if maximum occupancy is
targeted. For example on gfx1100 it is set to 192 registers even with
the requested occupancy 16. As a result scheduler starts prioritizing
register pressure reduction very late and we easily end up spilling.
This patch makes VGPR critical limit similar to what we would have on
pre-gfx10 targets with much more limited VGPR budget while still trying
to maintain occupancy as it does now.
Pre-gfx10 ASICs shall not be affected as the limit shall be the same
as before, and on gfx10+ it shall only affect regions where we have
to spill.
Fixes: SWDEV-377300
Differential Revision: https://reviews.llvm.org/D141876
Add the following intrinsic:
SQCVT
SQCVTU
UQCVT
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D142035
Add the following intrinsic:
FCVT
BFCVT
FCVTZS
FCVTZU
SCVTF
UCVTF
This patch also adds SelectCVTIntrinsic to handle the cases when the
intrinsic returns multiple (two or four) outputs
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: kmclaughlin
Differential Revision: https://reviews.llvm.org/D142032
Also, move the XformToShuffleWithZero and combineCarryDiamond folds later after some of the more basic canonicalizations/combines (such as this) have had a chance to occur
Fixes the v8i1-masks.ll regression from D127115
Both sclamp and uclamp are part of the SVE2p1 feature so I've
renamed the tests accordingly:
sve2-intrinsics-sclamp.ll -> sve2p1-intrinsics-sclamp.ll
sve2-intrinsics-uclamp.ll -> sve2p1-intrinsics-uclamp.ll
Currently both calling conventions preserve registers that are used to
store a return value. This causes the returned value to be lost:
define i32 @bar() {
%1 = call preserve_mostcc i32 @foo()
ret i32 %1
}
define preserve_mostcc i32 @foo() {
ret i32 2
; preserve_mostcc will restore %rax,
; whatever it was before the call.
}
This contradicts the current documentation (preserve_allcc "behaves
identical to the `C` calling conventions on how arguments and return
values are passed") and also breaks [[clang::preserve_most]].
This change makes CSRs be preserved iff they are not used to store a
return value (e.g. %rax for scalars, {%rax:%rdx} for __int128, %xmm0
for double). For void functions no additional registers are
preserved, i.e. the behaviour is backward compatible with existing
code.
Differential Revision: https://reviews.llvm.org/D141020
This shows the failure to merge to and(ext(x),and(c1,ext(c2))) if the outer and has already been folded to a clear shuffle mask
Similar to the v8i1-masks.ll from regression D127115
Was previously buggy to assume that NOT'd Value was always an
instruction. If the NOT'd value is not an Instruction, we should just
return as its either a constant, in which can we will re-run the logic
after constant-folding, or its a type we can't evaluate anyways.
This is a follow up to: `D140939`
Reviewed By: pengfei, RKSimon
Differential Revision: https://reviews.llvm.org/D142339
Given a patch like D129506, using instructions not valid for the current
target feature set becomes an error. This means that emitting Arm
instructions in a Thumb target (or vice versa) becomes an error. When
running in Thumb mode only thumb thunks will be needed, and in Arm mode
only arm thunks are needed. This patch limits the emitted thunks to just
the ones valid for the current architecture.
Differential Revision: https://reviews.llvm.org/D129693
This makes most sense for isFNegFree targets, but shouldn't make
things worse without it. This avoids AMDGPU test regressions in a
future patch.
For some reason APFloat::compareAbsoluteValue is private, so compute
the neg of the constants.
Now, when llc encounters the case that contains a lot of
extract_vector_elt and a BUILD_VECTOR, it will replace these to
vector_shuffle to decrease the size of code, the actions are done in
createBuildVecShuffle in DAGCombiner.cpp, but now the code cannot handle
the case that the size of source vector reg is more than twice the dest
size.
Reviewed By: pengfei
Differential Revision: https://reviews.llvm.org/D139685
This is essentially a modernized copy of
select-fabs-fneg-extract.ll. Stop using kernels with loads and stores,
don't use fsub for fneg, and port the examples to half.
The intent is to lower the clang X form SVE builtins to these
intrinsics. The suffix _x is already in use to signify unpredicated
SVE intrinsics hence my choice to use _u to signify those intrinsics
where the result for inactive lanes is undefined.
Differential Revision: https://reviews.llvm.org/D141937
The baseline `allowsMemoryAccess()` is wrong for X86.
It assumes that aligned memory operations are always allowed,
but that is not true.
For example, We can not perform a 32-byte aligned non-temporal load
of a 32-byte vector, without AVX2 that is, yet `allowsMemoryAccess()`
will say it is allowed, so we may end up merging non-temporal loads,
only to split them up to legalize them, and here we go again.
NOTE: the test changes here are superfluous. The main effect is that without this change,
in D141777, we'd get stuck endlessly merging and splitting non-temporal stores.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D141776
Previous logic was buggy and erroneously asserted that I->operand(0) must
be the RMW instruction. This change fixes that and makes it so that the
RMW instruction can be used in operand 0 or 1.
Also update the tests to explicitly test RMW as operand 0/1 (no change
to codegen).
Reviewed By: pengfei
Differential Revision: https://reviews.llvm.org/D142166
Summary:
For any enumerator whose definition does not have an initializer,
the associated value is the value of the previous enumerator plus one.
In order to avoid the possibility that two unrelated enumerators to
have the same value, we should try to cluster the uninitialized enumerators
together.
Reviewers: arsenm
Differential Revision
https://reviews.llvm.org/D141643
Add the following intrinsic:
FCVTN
BFCVTN
NOTE: These intrinsics are still in development and are subject to future changes.
Reviewed By: david-arm
Differential Revision: https://reviews.llvm.org/D142025
This change adjusts the cost modeling used when the target does not have a schedule model with individual instruction latencies. After this change, we use the default latency information available from TargetSchedule. The default latency information essentially ends up treating most instructions as latency 1, with a few "expensive" ones getting a higher cost.
Previously, we unconditionally applied the first legal pattern - without any consideration of profitability. As a result, this change both prevents some patterns being applied, and changes which patterns are exercised. (i.e. previously the first pattern was applied, afterwards, maybe the second one is because the first wasn't profitable.)
The motivation here is two fold.
First, this brings the default behavior in line with the behavior when -mcpu or -mtune is specified. This improves test coverage, and generally makes it less likely we will have bad surprises when providing more information to the compiler.
Second, this enables some reassociation for ILP by default. Despite being unconditionally enabled, the prior code tended to "reassociate" repeatedly through an entire chain and simply moving the first operand to the end. The result was still a serial chain, just a different one. With this change, one of the intermediate transforms is unprofitable and we end up with a partially flattened tree.
Note that the resulting code diffs show significant room for improvement in the basic algorithm. I am intentionally excluding those from this patch.
For the test diffs, I don't seen any concerning regressions. I took a fairly close look at the RISCV ones, but only skimmed the x86 (particularly vector x86) changes.
Differential Revision: https://reviews.llvm.org/D141017
The original check was trying to avoid checking UndefMask itself,
and deduce it via simpler means, but checking `NonZeroMask`
does not, e.g., check `ZeroMask`.
Fixes https://github.com/llvm/llvm-project/issues/60168
This change switches both targets from using target specific CompilerBarrier nodes to the recently introduced generic MEMBARRIER instruction.
A couple things to call out.
First, this changes the assembly comment printed. I'm not sure this matters, but if it does, we can simply drop this patch. This is a minor clean up at best.
Second, the ordering operand on the target instruction appears to be unused. We could easily add ordering to the generic instruction, but since we don't seem to have a motivating case in tree, I simply dropped the ordering when selecting to the generic instruction.
Differential Revision: https://reviews.llvm.org/D141513
Follow-up for 4ece50737d5385fb80cfa23f5297d1111f8eed39 (D142027).
Assignment Tracking Analysis now always runs and is skipped internally if
assignment tracking is disabled. Update these tests to expect to see the
pass run.
Buildbot failure: https://lab.llvm.org/buildbot/#/builders/57/builds/24094
Follow-up for 4ece50737d5385fb80cfa23f5297d1111f8eed39 (D142027).
Assignment Tracking Analysis now always runs and is skipped internally if
assignment tracking is disabled. Update these tests to expect to see the
pass run.
Buildbot failure: https://lab.llvm.org/buildbot/#/builders/216/builds/16085
This leverages the new logging format in that we don't need to buffer
the training data, we can just write it out.
Differential Revision: https://reviews.llvm.org/D142168
A lit test used overaligned i8, apparently due to an old copy-paste
error, intending to specify i32 alignment.
Change the datalayout string to use naturally aligned i8.
Add a flag state (and a MIR key) to MachineFunctions indicating whether they
contain instruction referencing debug-info or not. Whether DBG_VALUEs or
DBG_INSTR_REFs are used needs to be determined by LiveDebugValues at least, and
using the current optimisation level as a proxy is proving unreliable.
Test updates are purely adding the flag to tests, in a couple of cases it
involves separating out VarLocBasedLDV/InstrRefBasedLDV tests into separate
files, as they can no longer share the same input.
Differential Revision: https://reviews.llvm.org/D141387
Adds intrinsics for the following SME2 instructions:
* BFVDOT (32-bit)
* FVDOT (32-bit)
* SVDOT (2-way) (32-bit)
* SVDOT (4-way) (32-bit and 64-bit)
* UVDOT (2-way) (32-bit)
* UVDOT (4-way) (32-bit and 64-bit)
* SUVDOT (32-bit)
* USVDOT (32-bit)
NOTE: These intrinsics are still in development and are subject to future changes.
Differential Revision: https://reviews.llvm.org/D142000
