An AnyOf reduction (aka vector.reduce.or) with a fixed-width vector is
canonicalized to a bitcast of the mask vector to an integer of the same
overall size, which is then compared against zero.
If the scalar result of the bitcast is smaller than the element size of
vectors being selected, we often end up with suboptimal codegen. This
fixes the main cases, removing scalarized code.
Back when `TargetTransformInfo::instCombineIntrinsic` was added in
https://reviews.llvm.org/D81728, several transforms common to both ARM
and AArch64 were kept in the non-target-specific `InstCombineCalls.cpp`
so they could be shared between the two targets.
I want to extend the transform of the `tbl` intrinsics into static
`shufflevector`s in a similar manner to
https://github.com/llvm/llvm-project/pull/169110 (right now it only
works with a 64-bit `tbl1`, but `shufflevector` should allow it to work
with up to 2 operands, and it can definitely work with 128-bit vectors).
I think separating out the transform into a TTI hook is a prerequisite.
~~I'm not happy about creating an entirely new module for this and
having to wire it up through CMake and everything, but I'm not sure
about the alternatives. If any maintainers can think of a cleaner way of
doing this, I'm very open to it.~~
I've moved the transforms into
`Transforms/Utils/ARMCommonInstCombineIntrinsic.cpp`, which is a lot
simpler.
Currently, on Darwin running with -fsanitize=memtag-stack generates
compact-unwind exception unwinding that does not untag MTE-tagged memory
on the way back up.
This patch forces dwarf unwinding on MTE-tagged frames.
rdar://162195539
This tweaks `findStateChangeInsertionPoint` to also avoid clobbering X0,
which should be possible in most cases (since X0's live ranges are
likely to be very short before register allocation).
This improves codegen in a few cases, as not all redundant copies
to/from X0 are eliminated.
Chain calls never return and don't need to preserve any SGPRs.
Therefore, we don't need to limit the registers used for callees to the
CCR_SGPR_64 register class - it's fine to use any available SGPRs.
Also introduce a new pseudo, SI_TCRETURN_CHAIN, which also has a plain
SGPR_64 operand. This is necessary because we won't be able to lower
SI_CS_CHAIN_TC to SI_TCRETURN anymore, since its operand accepts a wider
range of registers than the latter.
Currently, out-of-range branch fixups will be silently miscompiled. GNU
as will instead print an "operand out of range" error instead.
Check that the branch target fixups fit into the proper range and have
low zero bits. The implementation is inspired by SystemZ:
0ed8e66f88/llvm/lib/Target/SystemZ/MCTargetDesc/SystemZMCAsmBackend.cpp (L31)
(My actual interest here is not actually assembly usage, but LLVM
failing to use the correct branch kind for reasons I've not tracked down
yet. Currently this just silently truncates the branch target instead of
producing an error.)
This change enables the LoadStoreVectorizer to merge and vectorize
contiguous chains even when their scalar element types differ, as long
as the total bitwidth matches. To do so, we rebase offsets between
chains, normalize value types to a common integer type, and insert the
necessary casts around loads and stores. This uncovers more
vectorization opportunities and explains the expected codegen updates
across AMDGPU tests.
Key changes:
- Chain merging
- Build contiguous subchains and then merge adjacent ones when:
- They refer to the same underlying pointer object and address space.
- They are either all loads or all stores.
- A constant leader-to-leader delta exists.
- Rebasing one chain into the other's coordinate space does not overlap.
- All elements have equal total bit width.
- Rebase the second chain by the computed delta and append it to the
first.
- Type normalization and casting
- Normalize merged chains to a common integer type sized to the total
bits.
- For loads: create a new load of the normalized type, copy metadata,
and cast back to the original type for uses if needed.
- For stores: bitcast the value to the normalized type and store that.
- Insert zext/trunc for integer size changes; use bit-or-pointer casts
when sizes match.
- Cleanups
- Erase replaced instructions and DCE pointer operands when safe.
- New helpers: computeLeaderDelta, chainsOverlapAfterRebase,
rebaseChain, normalizeChainToType, and allElemsMatchTotalBits.
Impact:
- Increases vectorization opportunities across mixed-typed but
size-compatible access chains.
- Large set of expected AMDGPU codegen diffs due to more/changed
vectorization.
This PR resolves#97715.
If the high bits are assumed 0 for the cast, use zext. Previously
we would emit a build_vector and a bitcast with the high element
as 0. The zext is more easily optimized. I'm less convinced this is
good for globalisel, since you still need to have the inttoptr back
to the original pointer type.
The default value is 0, though I'm not sure if this is meaningful
in the real world. The real uses might always override the high
bit value with the attribute.
Now we are just inserting V_NOP instrtuctions, try to schedule
something into the shadow.
It is still somewhat imprecise, for example AdvanceCycle() will
use TII.getNumWaitStates() anyway, but in a scheduling mode
we are not required to be precise. We must be finally precise
in the hazard recognizer mode. Then EmittedInstrs buffer is also
limited to MaxLookAhead even though VALU only hazards may actually
never expire and require an endless buffer. But that's OK, we can
at least mitigate what the buffer can hold. The buffer is also
currently much bigger than any of VALU hazards may need.
That said the rest of the 'fix*' functions here can be changed
the same way, these which are using V_NOPs. This one is just the
worst because it may require up to 9 nops.
This is in preparation of handling these in scheduler. I do not expect
any changes to the produced code here, it is just an infrastructure.
Our current problem with the VALU pipeline hazards is that we only
insert V_NOP instructions in the hazard recognizer mode, but ignore
it during scheduling. This patch is meant to create a mechanism to
actually account for that during scheduling.
This patch introduces the necessary infrastructure to legalize vector
operations on vectors that are longer than what the SPIR-V target
supports. For instance, shaders only support vectors up to 4 elements.
The legalization is done by splitting the long vectors into smaller
vectors of a legal size.
Specifically, this patch does the following:
- Introduces `vectorElementCountIsGreaterThan` and
`vectorElementCountIsLessThanOrEqualTo` legality predicates.
- Adds legalization rules for `G_SHUFFLE_VECTOR`,
`G_EXTRACT_VECTOR_ELT`,
`G_BUILD_VECTOR`, `G_CONCAT_VECTORS`, `G_SPLAT_VECTOR`, and
`G_UNMERGE_VALUES`.
- Handles `G_BITCAST` of long vectors by converting them to
`@llvm.spv.bitcast` intrinsics which are then legalized.
- Updates `selectUnmergeValues` to handle extraction of both scalars
and vectors from a larger vector, using `OpCompositeExtract` and
`OpVectorShuffle` respectively.
Fixes https://github.com/llvm/llvm-project/pull/165444
The pass preserved LiveStacksAnalysis but failed to preserve
LiveIntervalsAnalysis, LiveRegMatrixAnalysis, VirtRegMapAnalysis, and
SlotIndexesAnalysis under NPM. This caused these analyses to be
invalidated and recomputed, leading to incorrect behavior in subsequent
passes like VirtRegRewriter.
Fix by explicitly preserving all required analyses in the NPM version,
matching the legacy pass manager behavior.
---------
Co-authored-by: vikhegde <vikram.hegde@amd.com>
New SFB subsets are being added with the scheduler class name as a
suffix, so now is the time to go back to the base extension and add IALU
to its name.
This also:
- Drops a hyphen from the other SFB features for mul and minmax, to more
closely match their scheduling classes.
- Updates the predicates on specific SFB pseudos so we get verifier
errors if we introduce the pseudos when we don't have the right
subtarget feature.
- Updates the SFB Documentation comment to make it no longer
SiFive-specific.
Similar to how getElementCount avoids the need to reason about fixed and
scalable ElementCounts separately, this patch adds getTypeSize to do the
same for TypeSize.
It also goes through and replaces some of the manual uses of getVScale
with getTypeSize/getElementCount where possible.
This patch extends the MachineSMEABIPass to support ZT0. This is done
with the addition of two new states:
- `ACTIVE_ZT0_SAVED`
* This is used when calling a function that shares ZA, but does not
share ZT0 (i.e., no ZT0 attributes)
* This state indicates ZT0 must be saved to the save slot, but ZA must
remain on, with no lazy save setup
- `LOCAL_COMMITTED`
* This is used for saving ZT0 in functions without ZA state
* This state indicates ZA is off and ZT0 has been saved
* This state is general enough to support ZA, but the required
transitions have not been implemented†
To aid with readability, the state transitions have been reworked to a
switch of `transitionFrom(<FromState>).to(<ToState>)`, rather than
nested ifs, which helps manage more transitions.
† This could be implemented to handle some cases of undefined behavior
better.
During operation legalization, element type should have been turn into
XLenVT which makes the SHL a no-op. We need to use exact vector element
type instead.
### Summary
This patch let RISCVTargetLowering::lowerSELECT to lower some
floating-point select operations through an integer zicond select when:
* Zicond is available, and
* FP values live in GPRs (Zfinx/Zdinx), and
* Select condition is an integer type.
In that scenario there is no extra cost for GPR <-> "FP GPR" moves, so
we can implement FP selects with a CZERO-based sequence instead of a
branch.
For example, for
```c
float foo(int cond, float x) {
return (cond != 0) ? x : 0.0f;
}
```
the current lowering produces:
```asm
foo:
mv a2, a0
li a0, 0
beqz a2, .LBB0_2
.LBB0_1:
mv a0, a1
.LBB0_2:
ret
```
With this patch, when targeting rv64ima_zicond_zfinx we instead get:
```asm
foo:
czero.nez a2, zero, a0
czero.eqz a0, a1, a0
or a0, a2, a0
ret
```
The existing branch-based lowering is preserved for:
* targets without Zicond
* targets where FP registers are separate (+f, +d without zfinx/zdinx)
### Testing
Adds llvm/test/CodeGen/RISCV/zicond-fp-select-zfinx.ll to cover:
* RV64 Zfinx/Zicond vs Zfinx without Zicond
* RV64 Zdinx/Zicond vs Zdinx without Zicond
* RV32 Zfinx/Zicond vs Zfinx without Zicond
Also adds baseline RV32F/RV64F/RV64D cases to ensure we still use
branches when FP registers are separate.
The tests check that:
* With Zicond + Zfinx/Zdinx, FP select lowers to a CZERO+OR sequence
with no conditional branches.
* Without Zicond (or without Zfinx/Zdinx), we still get branch-based
code and no czero.* instructions.
This patch is a rework of #160470 (which was reverted).
With getMemIntrinsicCost() now available, we can re‑land the change and
reduce vp_load_ff boilerplate.
In the Neoverse N3 Software Optimisation Guide, SVE non termporal gather
load, vector+scalar 64-bit element size and gather load, vector + imm,
64-bit element size both show throughput of 4/5. However, it currently
shows as 2/3. This patch adds a new resource group in order to show the
correct throughput.
Some ASIMD instructions in the Neoverse N3 scheduler model seem to have
been missed and have default definitions, which give them incorrect
latency and throughput. This patch fixes such instructions to match the
current N3 SWOG.
The test case generates a extract_subvector(index) leading into a mul.
Make sure we don't try and treat the scalable vector extract as a 128bit
vector in the mull combine.
Fixes#168912
A funnel shift with the first two operands is a rotate. AArch64 has
scalar instructions for ror so can handle a ROTR by a constant or
variant amount cheaply. A ROTL is a ROTR by the opposite amount, which
for constants is cheap and for variable shifts requires an extra neg.
This patch adds SVE support for the `masked.compressstore` intrinsic via
the existing `VECTOR_COMPRESS` lowering and compressing the store mask
via `VECREDUCE_ADD`.
Currently, only `nxv4[i32|f32]` and `nxv2[i64|f64]` are directly
supported, with other types promoted to these, where possible.
This is done in preparation for LV support of this intrinsic, which is
currently being worked on in #140723.
This commit improves the lowering of vectors of fp16 when truncating and
(previously) extending. Truncating has to be handled in a specific way
to avoid double rounding.
When SIMULATED_TRAP is at the end of a block with no successors,
insertSimulatedTrap incorrectly returns the original MBB despite adding
HaltLoopBB to the CFG.
EmitInstrWithCustomInserter detects CFG changes by comparing the
returned MBB with the original. When they match, it assumes no
modification occurred and skips MachineLoopInfo invalidation. This
causes stale loop information in subsequent passes, particularly when
using the NPM which relies on accurate invalidation signals.
Fix: Return HaltLoopBB to properly signal the CFG modification.
Most predicable Thumb instructions do not encode the predicate operand,
but rather take it from an enclosing IT block.
Add `bits<0> p` to the encoding of these instructions to make the
predicate operand decodable by the generated code.
The previous approach was to analyze an instruction after it has been
decoded and add missing predicate operand if necessary. The
post-decoding pass is still required to check predicate applicability
and advance IT block state, but it no longer modifies a decoded
instruction.
Some of the custom decoder methods have become redundant and can be
removed in the future, delegating the decoding task to TableGen-erated
decoder.
Pull Request: https://github.com/llvm/llvm-project/pull/156540
- Following #168029. This is a step toward a unified interface for
masked/gather-scatter/strided/expand-compress cost modeling.
- Replace the ad-hoc parameter list with a single attributes object.
API change:
```
- InstructionCost getExpandCompressMemoryOpCost(Opcode, DataTy,
- VariableMask, Alignment,
- CostKind, Inst);
+ InstructionCost getExpandCompressMemoryOpCost(MemIntrinsicCostAttributes,
+ CostKind);
```
Notes:
- NFCI intended: callers populate MemIntrinsicCostAttributes with same
information as before.
